Fixes relative to RPATH handling

[dnicolodi]

Builds on top of #783 and replaces #724

Fixes #711
Fixes #813

[dnicolodi]

This should fix RPATH handling for good.

The only thing not included is automatic translation of the $ORIGIN anchor in install_rpath arguments to @loader_path on macOS. It would not be hard to implement, however, it would be a deviation from what Meson implements. I am not sure it is a good idea to implement it.

There is one case in which this may break projects that work now: when libraries or modules require setting an RPATH to dynamically link to a library installed in the Python install path (with something like install_dir: py.get_install_dir() / 'package') and that work now for how because meson-python does not remove build RPATH entries added by Meson and the source layout matches the install layout. This was the case for example for SciPy 1.15, see #724 (comment) and previous discussion. The comments there indicated that, for what SciPy is concerned, it should be fine to break this now. Note that SciPy 1.15 used install_rpath arguments that are implemented here, thus it would be fine, but it uses the $ORIGIN anchor on macOS too, and that does not work. See above. The behavior changes only with Meson 1.9.0 or later (older Meson versions did not export the required metadata), thus projects that pin the Meson version are fine.

@rgommers I think I added test cases for all scenarios we discussed. It would be great if you could test with packages that may be affected and that do not pin the meson-python version to any released version.

[dnicolodi]

There is a failure in the pixi test job:

FAILED tests/test_wheel.py::test_sharedlib_in_package_rpath - AssertionError: assert {'$ORIGIN', '.../default/lib'} == {'$ORIGIN'}
  
  Extra items in the left set:
  '/home/runner/work/meson-python/meson-python/.github/workflows/.pixi/envs/default/lib'
  
  Full diff:
    {
        '$ORIGIN',
  +     '/home/runner/work/meson-python/meson-python/.github/workflows/.pixi/envs/default/lib',
    }

I don't know where the extra RPATH entry comes from. I suspect that it is something that the build environment adds but I don't know why or where, nor what would be a good way to control for it in the test.

[dnicolodi]

I don't know where the extra RPATH entry comes from.

It comes from the LDFLAGS added to the Python configuration. Meson gets them from pkgconfig but it can be obtained from sysconfig too. I amended the tests to take it into consideration. It is not beautiful but it works.

[dnicolodi]

This PR should also update the docs on shared libraries, so we recommend the new preferred form. Which I think is @loader_path on macOS, and $ORIGIN elsewhere.

Done.

[rgommers]

Thanks for the updates!

It comes from the $LDFLAGS environment variable.

Yes indeed. For a very long time, conda-forge set CFLAGS/CXXFLAGS/LDFLAGS in its compiler activation scripts in order to both set conda-forge defaults (e.g., -O2) and achieve working relocatable environments. This meant including -Wl,-rpath=$PREFIX/lib into LDFLAGS, with $PREFIX the root of the active conda environment.

Setting those env vars led to a lot of unwanted side effects (e.g., the -O2 made it hard to get plain debug builds), so over the last year effort was put into "minimally activated compilers", which can be installed with compilers>2 or gcc>=15 (Clang version I can't remember off the top of my head). To keep relocation working while no longer population LDFLAGS et al., the needed compiler flags were moved directly into gcc/clang spec files. So with the most recent compilers, you still get -rpath $PREFIX/lib in the produced extension modules and shared libraries, but that rpath is no longer visible in compiler flags or environment variables.

I believe Homebrew uses spec files too. As do Linux distros in general, but there you typically don't notice because the system loader already has the right search paths, so you don't need extra RPATHs set in spec files. Nix/Spack use compiler wrapper scripts instead; all achieve the same end result of adding an RPATH when invoking a compiler.

In all those cases: you can get extra RPATH entries from a compiler invocation, and that may or may not be observable from either LDFLAGS or build.ninja contents.

I think the takeaway is that we can test:

• that the expected rpaths added by the test package are present
• that build/destdir directory rpaths are not present
  and that extra paths may be present (we cannot reliably know if they were real, tests have to assume they're valid).

[rgommers]

Flags from sysconfig never propagate, that's a setuptools-only behavior, so this might be misleading. For conda/pixi, it's (used to be) the environment variable LDFLAGS , which only happens to match sysconfig.get_config_var('LDFLAGS'), but that isn't guaranteed and will no longer be true if python gets rebuilt with "minimally activated compilers".

More importantly, we can't reliably know if extra RPATHs are valid or not, we only know when they're invalid (a build/destdir path or something we added ourselves in a test package).

[rgommers]

In our own CI we could assert an exact set, because we have control over that. So if we want something like that, like assert rpaths == set(expected, extra_from_conda_env_or_compiler), we can - but it should then be gated on something like MESONPY_CI_ONLY.

[dnicolodi]

Flags from sysconfig never propagate

I know. This is just an hack to get to what the pkgconfig contains hoping that it stays in sync with what is in sysconfig. I don't like this either, but I wanted to keep the tests as strict as possible and I was unable to come up with a nicer solution.

[rgommers]

I don't think keeping the test strict is essential, but if you do want that, then I think it's correct to read os.environ['LDFLAGS']. The current sysconfig.get_config_var('LDFLAGS') happens to match today, but I expect it to start diverging as soon as conda-forge's python is rebuilt with the minimally activated compilers.

[dnicolodi]

os.environ['LDFLAGS'] is the first thing I tried, but it is not set.

[rgommers]

Some of the most relevant docs/discussion/code:

• conda-forge minimally activated compilers tracking issue: conda-forge.github.io#2595 (rationale and design discussion for moving compiler policy out of activation-time environment variables.)
• The conda-gcc-specs README documents how conda-forge's GCC injects include/library paths and RPATH via GCC specs.
• And here is Homebrew doing the same -rpath-via-spec-file: Homebrew/install_steps/formula_actions.rb#L57

[dnicolodi]

• And here is Homebrew doing the same -rpath-via-spec-file: Homebrew/install_steps/formula_actions.rb#L57

Uhm. I wonder how the RPATH tests work without the sysconfig hack on Homebrew. Maybe there is something wrong with our Homebrew CI setup...

[dnicolodi]

I think the takeaway is that we can test:

that the expected rpaths added by the test package are present

that build/destdir directory rpaths are not present and that extra paths may be present (we cannot reliably know if they were real, tests have to assume they're valid).

Checking that the build paths are not present requires computing the build rpaths, which is not always straightforward. Furthermore, sometimes the build paths are the same as the expected rpaths, making the things even more complex.

I think we can assume that any rpath added by the build environment is an absolute path. Thus we can simply filter these out and keep testing that only the expected rpaths are there. What do you think?

[rgommers]

Maybe there is something wrong with our Homebrew CI setup...

That code I linked is Homebrew on Linux, in a function named run_configure_gcc_runtime. There's nothing wrong in CI which is macOS; we could test Homebrew on Linux as well, but I'm not sure how much it adds - I've never seen it used. If we're thinking about expanding, I'd personally be more interested in Nix. But I don't think either of them are urgent, our platform coverage is pretty good.

I think we can assume that any rpath added by the build environment is an absolute path. Thus we can simply filter these out and keep testing that only the expected rpaths are there. What do you think?

That seems like a good approach for the test suite.

I'd make one addition: before filtering out absolute paths, reject any that point into the source or build directory. We should always be able to know those: the build directory is either explicitly supplied by -Cbuild-dir or created inside the source tree. That catches absolute build path leaking in without having to do anything complicated. We can then ignore other absolute paths from the build environment and check the remaining relative paths against the expected ones.

Of course there may be a test that explicitly adds -Wl,-rpath=path-to-some-linked-lib; that test package is then expected to have an absolute path. I.e., the "filter out all abspaths" may need an exception for a specific test.

I'll note that I was just polishing a test package for path ordering and (lack of) duplicates, so we can't only compare sets in general. I hope to finish that later today.

[dnicolodi]

That code I linked is Homebrew on Linux

Right. I didn't pay attention.

I'd make one addition: before filtering out absolute paths, reject any that point into the source or build directory. We should always be able to know those: the build directory is either explicitly supplied by -Cbuild-dir or created inside the source tree. That catches absolute build path leaking in without having to do anything complicated. We can then ignore other absolute paths from the build environment and check the remaining relative paths against the expected ones.

This seems like a good approach. This can be extended to check for duplicates at the same time. I'll see how complex it is too do this.

Of course there may be a test that explicitly adds -Wl,-rpath=path-to-some-linked-lib; that test package is then expected to have an absolute path. I.e., the "filter out all abspaths" may need an exception for a specific test.

There are just two tests that strictly check all rpath entries. I don't think it is necessary to extend it to all tests. The other tests are builds of the same package which add some rpaths via env vars or command line flags.