Repository navigation
Permission denied errors when rebuilding (Python?) bundles with extensions that install binaries #556
Description
When trying to rebuild Python, hatchling, and Python-bundle-PyPI in #546, we ran into weird permission issues for both hatchling and Python-bundle-PyPI. It's not clear yet what's causing it, but it seems to happen for (Python) bundles that include extensions that not only install files to lib, but also to bin.. The removal step seems to work fine and successfully removes the existing installation, but in the build phase the extension will suddenly see the old bin directory again (with read-only permissions), and fail with errors like:
Successfully built hatchling
Installing collected packages: hatchling
ERROR: Could not install packages due to an OSError.
Consider using the `--user` option or check the permissions.
Traceback (most recent call last):
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 449, in run
installed = install_given_reqs(
^^^^^^^^^^^^^^^^^^^
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/req/__init__.py", line 72, in install_given_reqs
requirement.install(
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/req/req_install.py", line 800, in install
install_wheel(
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py", line 731, in install_wheel
_install_wheel(
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py", line 648, in _install_wheel
generated_console_scripts = maker.make_multiple(scripts_to_generate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py", line 436, in make_multiple
filenames.extend(self.make(specification, options))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_internal/operations/install/wheel.py", line 429, in make
return super().make(specification, options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py", line 425, in make
self._make_script(entry, filenames, options=options)
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py", line 325, in _make_script
self._write_script(scriptnames, shebang, script, filenames, ext)
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/scripts.py", line 293, in _write_script
self._fileop.write_binary_file(outname, script_bytes)
File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/site-packages/pip/_vendor/distlib/util.py", line 555, in write_binary_file
os.remove(path)
PermissionError: [Errno 13] Permission denied: '/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/hatchling/1.18.0-GCCcore-12.3.0/bin/hatchling'
(at easybuild/tools/run.py:682 in parse_cmd_output)
I've tried a lot of possible workarounds in both #546 and #555:
- add write permissions (
chmod -R u+w) instead of removing the existing installation dir - add write permissions and remove the existing installation dir
- also remove the parent dir (e.g.
hatchling) instead of only removing the installation dir of the particular version - only remove the contents of the installation dir and not the dir itself
- move the existing installation dir instead of removing it
- wipe the CVMFS cache between the removal and build steps
- use a newer fuse-overlayfs
- unsetting
EASYBUILD_READ_ONLY_INSTALLDIRbefore starting the build
None of them solved the issue, though. So, in the end, I opted for working around it by adding write permissions to the affected installation directories on the Stratum 0, and then the rebuilds completed successfully.
Activity
bedroge commentedon Jun 7, 2024
We're now seeing the same issue for our EESSI-extend module (see #578), which is a bundle.
I still don't have a clue why it only happens for some easyconfigs...
bedroge commentedon Oct 15, 2024
Also seeing similar issues for LAMMPS in #788. This one is not a bundle, but a cmakemake easyblock.
I've debugged this a bit interactively, but still can't find the issue. After the directory gets recreated in the build container, it looks empty at first, but when you do an
lson a file that previously existed, it suddenly shows up... Theupperdir looks like this after the directory gets removed:and like this when it's recreated:
Might also be related to containers/fuse-overlayfs#324?
bedroge commentedon Oct 15, 2024
I've also tried some workarounds here, for instance by explicitly removing every individual file in the installation directory (using
find ..... -exec rm ...), as suggested by @casparvl , but that didn't work either.As an alternative I was thinking of bind mounting a host directory to the installation prefix in the container, but that doesn't seem possible on top of fuse-mounted CVMFS repos. The only way I could make it work is by also bind mounting
/cvmfsitself from the host, and then bind mounting an empty host directory on top of it. Though it works, it would mean that we have to assume that/cvmfsis available on the host, which is probably not always true. But perhaps we could still consider implementing this, and only use it in case/cvmfs/software.eessi.iois available on the host. If not, we can fall back to using the fakeroot approach (and hope it works 😅 ). It will make the code in the build script quite complex, though.boegel commentedon Oct 15, 2024
@bedroge This smells a lot like a bug in
fuse-overlayfs, no?Which version are we using, when have we last tried to update it?
bedroge commentedon Oct 15, 2024
Forgot to mention that, but I also tried using a newer version of fuse-overlayfs (1.14), that didn't help either.
It could be a bug, though we also have a bit of a complex setup here with read-only dirs in a fuse-mounted CVMFS repo, where we delete stuff with fakeroot, and then reuse the
upperdir for a fuse-mounted writable overlay on top of it.bedroge commentedon Oct 15, 2024
And I really don't understand why we only see it sometimes. It worked fine for, for instance, GCC and OpenMPI rebuilds.
boegel commentedon Oct 15, 2024
That complicates making a small reproducer to be able to report this to fuse-overlayfs upstream, of course... :(
Any luck with
strace?bedroge commentedon Oct 18, 2024
Based on @casparvl's suggestion, I've been trying some other workarounds, and at some point it suddenly seemed to work when I removed all files and directories manually and individually. But when I tried to do the same thing with some smart
findcommands, I couldn't make it work anymore. Even worse, the manual procedure then suddenly didn't seem to work anymore either... After lots of attempt, I think I found out why: it seems to matter what you do in the second container session (i.e. the build step) in a very strange and unexpected way.So I tried this on the
EESSI-extend/2023.06-easybuildinstallation, which is known to fail for rebuilds.Removal step, with
--fakeroot:Then resume the container without
--fakerootand run:This is the issue that we often see in PRs, where the installation prefix seems empty, but if you know what to look for, files are suddenly still there.
Now I'm redoing the same steps in a clean container, do the removal in the same way, and just do one additional
lsin the second step:So, the issue only pops up when you first do an
lson the (empty) installation directory itself 🤯 😵💫bedroge commentedon Oct 18, 2024
And though the fix seems easy, it may not be trivial. The affected subdir seems to be different for each installation: sometimes it's
easybuild, sometimes it'sbin. We could make a dump of all subdirs of the installation prefix before actually removing them, and then do anlson each of them in the build script, but I really don't like that 😅 I've tried several more simple workarounds, but haven't found any yet. For instance, doing this in the removal container doesn't solve it. It looks like it's solved, but whenever you recreate the removed directory, you would have to pull the same trick again. 😞boegel commentedon Jun 12, 2025
Hitting this problem with
EESSI-extend(again) in EESSI/software-layer-scripts#2, though with a slightly different error message, so dumping it here to make this issue easier to find:boegel commentedon Jun 13, 2025
I've experimented a bit with bubblewrap (
bwrap) as a potential replacement forfuse-overlayfs, but I'm not getting very far.It looks like
bwrapcan't be used in a container environment (see also containers/bubblewrap#505).That's a shame, because of latest version of
bwrap(0.11.0) includes support for--overlay.I did bump into containers/fuse-overlayfs#428, where a small patch is proposed that could fix our problem.
Maybe we can build a patched version of
fuse-overlayfsin our build container?boegel commentedon Jun 14, 2025
Looks like I found a alternative to
fuse-overlayfswhich doesn't show the problems we see withfuse-overlayfs, withunionfs(see also EESSI/filesystem-layer#226):Manually tested with:
In this container, permissions are honored, but
rm -rfworks as expected:Also
chmod -R u+won a directory works fine.ocaisa commentedon Sep 19, 2025
This seems to have been solved by the switch to
unionfsbedroge commentedon Oct 23, 2026
Reopening this, as we're still seeing issues for rebuilds. Sometimes they fail with errors like:
but then the next attempt works fine. See e.g. #1842.
It's not clear to me why it's so random/inconsistent.
boegel commentedon Oct 23, 2026
Have we only seen this on Python package installations? Or are we just more likely to hit it there because those includes lots of tiny files?
bedroge commentedon Oct 23, 2026
Initially it was (almost?) always happening for bundles. I think we have recently (since the move to unionfs) seen this for other types of rebuilds as well.
In that same PR (#1842) there seems to be a persistent failure for only zen4. The only thing that's different for that one is that it does a sort of module-only installations, because that toolchain is not supported on zen4. But I don't see how that would matter w.r.t. file ownerships/permissions.
bedroge commentedon Oct 23, 2026
Similar failures in #1784 and #1675 for instance.
boegel commentedon Oct 23, 2026
We're still using
unionfs3.6 in our build container. There's a 3.8 version we could try.https://github.com/rpodgorny/unionfs-fuse/blob/master/NEWS mentions this for unionfs 3.7 release:
Seems potentially relevant, especially since we symlink
libtolib64(or vice versa), and the problems we're seeing are exactly forchmod?bedroge commentedon Oct 23, 2026
Thanks, definitely worth a try. Opened a PR here: EESSI/filesystem-layer#289.