BUG: Avoid re-extracting the ITK build cache for module dependencies - #307
BUG: Avoid re-extracting the ITK build cache for module dependencies#307axel-grc wants to merge 1 commit into
Conversation
When building a module that depends on another ITK module (e.g. RTK depends on ITKCudaCommon), the ITK build cache is downloaded and extracted once in the parent directory. build-module-deps.sh then symlinks the module directory to that cache (ITKPythonPackage -> ../ITKPythonPackage) and used to re-run the full download-and-extract flow inside the module directory. Re-extracting in the module directory was problematic for two reasons: - Redundant: the same cache was just extracted in the parent directory with the same arguments, and the module directory writes through the symlink into that same shared tree. - Recent GNU tar (openat2 jailification, e.g. Ubuntu tar 1.35+dfsg-4ubuntu0.2+) fails with 'Invalid cross-device link' (EXDEV) when extracting members under ITKPythonPackage/ because the symlink escapes the extraction root. The module dependency build also registers the module (e.g. CudaCommon) into the shared ITK build tree (ITK_DIR) it builds against, so the registration must land in the tree the parent build reuses afterwards. Building against a module-local copy instead of the shared tree breaks serial module dependencies. Fix: when the parent already holds the extracted cache, build the module directly against the shared tree with dockcross-manylinux-build-module-wheels.sh, without re-extracting it. When no cache is available (standalone use of build-module-deps.sh), keep the full download-and-extract-and-build path.
dzenanz
left a comment
dzenanz
left a comment
There was a problem hiding this comment.
Looks good on a glance. It would be good if someone else reviewed too. Does it make sense to apply this patch to main?
We need this fix for RTK next release which will be based on release 5.4, but I think we can apply this to main as well. |
|
Thanks @axel-grc. I have the feeling that the problem this PR addresses has been fixed, I guess by an update of the GitHub runner. See failing workflow with runner 2.336 and successful workflow with runner 2.336. Do you confirm? The problem is I don't see anything in the runner release log that explains it. |
Even with runner 2.337, the tar error is still there, so the fix is required. |
3 participants
When building a module that depends on another ITK module (e.g. RTK depends on ITKCudaCommon), the ITK build cache is downloaded and extracted once in the parent directory. build-module-deps.sh then symlinks the module directory to that cache (ITKPythonPackage -> ../ITKPythonPackage) and used to re-run the full download-and-extract flow inside the module directory.
Re-extracting in the module directory was problematic for two reasons:
Redundant: the same cache was just extracted in the parent directory with the same arguments, and the module directory writes through the symlink into that same shared tree.
Recent GNU tar (openat2 jailification, e.g. Ubuntu tar 1.35+dfsg-4ubuntu0.2+) fails with 'Invalid cross-device link' (EXDEV) when extracting members under ITKPythonPackage/ because the symlink escapes the extraction root.
The module dependency build also registers the module (e.g. CudaCommon) into the shared ITK build tree (ITK_DIR) it builds against, so the registration must land in the tree the parent build reuses afterwards. Building against a module-local copy instead of the shared tree breaks serial module dependencies.
Fix: when the parent already holds the extracted cache, build the module directly against the shared tree with
dockcross-manylinux-build-module-wheels.sh, without re-extracting it. When no cache is available (standalone use of build-module-deps.sh), keep the full download-and-extract-and-build path.