meson-python doesn't support files in {prefix}/share

[noamraph]

Currently, meson-python doesn't support files under {prefix}/share. For some files (those that explicitly go to {datadir}, which is usually {prefix}/share) it will not crash, but those file will go to {prefix}, not {prefix}/share, in the python installation. For most files, it will just refuse to build the wheel.

This currently prevents wheel installation of Meld.

To reproduce:

git clone --depth=1 https://gitlab.gnome.org/GNOME/meld.git
python3 -m venv /tmp/venv
/tmp/venv/bin/pip install meld/
...
meson-python: error: Could not map installation path to an equivalent wheel directory: '{prefix}/share/help/cs/meld/command-line.page'

Note that if you replace meson-python>=0.5.0 with meson-python==0.5.0 in meld/pyproject.toml the installation seems to succeed, but the files that should be in {prefix}/share are under {prefix}, for example, you'll find /tmp/venv/icons and /tmp/venv/meld which should have been /tmp/venv/share/icons and /tmp/venv/share/meld.

[rgommers]

Thanks for the issue! Quick comment: gh-517 discusses the same problem.

[dnicolodi]

Currently, meson-python doesn't support files under {prefix}/share. For some files (those that explicitly go to {datadir}, which is usually {prefix}/share) it will not crash, but those file will go to {prefix}, not {prefix}/share, in the python installation.

I don't understand what you mean. {prefix} is a Meson thing. There is no equivalent location in the wheel specification. Thus I do not understand what you mean when you say that files installed in {datadir} are installed in {prefix}: {prefix} is not a thing in a wheel or in the Python package installation paths.

Files the Meson would install in {datadir} are places in the {package-name}-{package-version}.data/data/. On installation of the wheel, these are copied in the destination defined by the sysconfig installation paths. However, AFAIK, there is no portable way to access the files installed at that location once the wheel is installed. meson-python provides a mapping to install files in the wheel data folder only for backward compatibility, not because it is a good idea to use it.

We could map files installed in {prefix}/shared to the data folder or a subdirectory in it, but what do you plan to use them for?

[rgommers]

or some files (those that explicitly go to {datadir}, which is usually {prefix}/share) it will not crash, but those file will go to {prefix}, not {prefix}/share,

The way installers currently handle this is more flexible, and probably won't change. Look at pybind11-global for example, it knows how to install to <prefix>/share/ by including files under data/share/ in the .data directory in a wheel:

pip and uv handle those the same.

$ uv pip install pybind11-global --system -vvv
...
TRACE Installing data/headers to /home/rgommers/mambaforge/include/python3.10/pybind11-global dist_name=PackageName("pybind11-global")
TRACE Installing data/data to /home/rgommers/mambaforge dist_name=PackageName("pybind11-global")

(/home/rgommers/mambaforge is the "prefix" there; I could use my distro pip to show installing to /usr/ too, but I'd rather not do that:)).

The code required to do this for pybind11-global is still not pretty, but there's a proper interface for it now in scikit-build-core (wheel.install-dir = ...): pybind/pybind11#5598 (comment)

However, AFAIK, there is no portable way to access the files installed at that location once the wheel is installed

"portable" is a relative term. I suspect that for Meld only Linux is of interest, and the idea is not to publish wheels for many platforms including Windows to PyPI, but rather to use pip inside distro build workflows etc. Which could work probably. @noamraph please correct me if I'm wrong - can you please expand as much as possible on what wheels would be used for?

Also, if you could add what wheels give you that using meson setup/compile/install do not (e.g., metadata, entrypoints) that would be really helpful to motivate making a change here.

We could map files installed in {prefix}/shared to the data folder or a subdirectory in it

Right now we error out as soon as we see {prefix}, because it's not in the install map:

meson-python/mesonpy/__init__.py

Line 103 in 6ea60fd

_INSTALLATION_PATH_MAP = {

It's been a recurring pain point. We can't add {prefix} to that mapping by default, because a lot will go wrong. But we could add it as an opt-in perhaps, if that satisfies the needs. It should then map to .data/data/ as well I'd think.

[noamraph]

Thanks @rgommers and @dnicolodi for your replies!

What I'm trying to do is to let people install Meld by running uv tool install meld, or run it with pipx meld or uvx meld. Or create a venv somewhere, run venv/bin/pip install meld and then run venv/bin/meld.

(The package name "meld" is already taken. So it would be something like meldmerge. It doesn't really matter.)

Thanks to your replies, I realized that what I was trying to do was not ideal. With my current PR to meson-python plus my current PR to Meld, after running venv/bin/pip install meld you would get the data files under venv/share, and meld would find them using sys.prefix. However, I now realize that a better solution would be to have them installed under venv/lib/python/site-packages/meld/share. I'm working on this.

---

BTW, From what I see, I would probably open another PR to meson-python. I updated Meld's build.meson to have this:

is_wheel = get_option('is_wheel')
if is_wheel
  datadir = python3.get_install_dir(pure: true, subdir: meson.project_name()) / 'share'
  localedir = datadir / 'locale'
else
  datadir = get_option('datadir')
  localedir = get_option('localedir')
endif
i18n.gettext(meson.project_name(), preset: 'glib', data_dirs: itsdir, install_dir: localedir)

It causes the i18n files to be installed to the expected path (/usr/local/lib/python/site-packages/meld/share/locale), but meson-python fails, because it expects the path in intro-install_plan.json to start with {py_purelib}, but it doesn't. I think that a solution would be to make meson-python pass to meson arguments like -Dpython.purelibdir=/mesonpy-py_purelib, and then use the paths in intro-installed.json (which are resolved paths) and use the first part of the paths there (so paths starting with /mesonpy-py_purelib/ would go to purelib, paths starting with /mesonpy-bindir would go to scripts, etc).

[rgommers]

However, I now realize that a better solution would be to have them installed under venv/lib/python/site-packages/meld/share. I'm working on this.

That sounds right, given your goal - you really want all data files to be installed as part of the package inside its own tree for portability reasons. Adding a build option for that to Meld makes sense.

but meson-python fails, because it expects the path in intro-install_plan.json to start with {py_purelib}, but it doesn't.

It doesn't. If you're using install_data, custom_target, or similar, rather than a method on the python.find_installation() object, then you need to use this: install_dir: py.get_install_dir() / 'meld/share',

Don't try to manually work with purelib/platlib.

[dnicolodi]

There is a bug in Meson:

i18n.gettext('intltest', install_dir: py.get_install_dir() / 'foo/locale')

results in a an absolute path recorded in the install data metadata. I think it is the same as the issue already reported for the pkgconfig module.

[dnicolodi]

See mesonbuild/meson#15494

[noamraph]

Thanks @dnicolodi !
I tried your branch of meson, and I get another such error. It comes from this:

if is_wheel
  configure_file(
    input: '@0@.gschema.xml'.format(resource_base_id),
    output: 'gschemas.compiled',
    command: [compile_schemas, meson.current_source_dir(), '--targetdir', meson.current_build_dir()],
    install_dir: datadir / 'meld',
  )
endif

Which results in:

mesonpy.BuildError: Could not map installation path to an equivalent wheel directory: '/usr/local/lib/python3/dist-packages/meld/share/glib-2.0/schemas/org.gnome.Meld.gschema.xml'

Thanks!
Noam

[dnicolodi]

I cannot reproduce the configure_file() issue. There also is a test in the meson-python test suite that verifies that this construct works. What is the definition of datadir in the snippet above?

[noamraph]

@dnicolodi sorry, it's the same as before:

datadir = python3.get_install_dir(pure: true, subdir: meson.project_name()) / 'share'

[noamraph]

@dnicolodi if it helps, here's the commit: https://gitlab.gnome.org/GNOME/meld/-/commit/144c8f59612d2a79d018f83746ec240c2f97c144?merge_request_iid=144

[dnicolodi]

I've tried to reproduce the issue with this code

project('foo', version: '1')

py = import('python').find_installation()

datadir = py.get_install_dir(pure: true, subdir: meson.project_name()) / 'share'

resource_base_id = 'org.gnome.Meld'

compile_schemas = [py, files('compile.py')]

configure_file(
  input: '@0@.gschema.xml'.format(resource_base_id),
  output: 'gschemas.compiled',
  command: [compile_schemas, meson.current_source_dir(), '--targetdir', meson.current_build_dir()],
  install_dir: datadir / 'meld',
)

where compile.py is

import os, shutil, sys
shutil.copyfile(os.path.join(sys.argv[1], 'org.gnome.Meld.gschema.xml'), os.path.join(sys.argv[3], 'gschemas.compiled'))

and the content of org.gnome.Meld.gschema.xml does not matter.

It works for me. I don't believe that the issue depends on the exact command executed by configure_file().

[dnicolodi]

However, note that the error message you report is about a org.gnome.Meld.gschema.xml file that seems to be the input of this configure_file() target, not the output of it. The file that cannot be mapped seems to be installed by this code in data/meson.build:

install_data(
  '@0@.gschema.xml'.format(resource_base_id),
  install_dir: gio_schemasdir,
)

where gio_schemasdir is defined in meson.build as

gio_schemasdir = dependency('gio-2.0').get_variable(
  pkgconfig: 'schemasdir',
  pkgconfig_define: ['datadir', datadir],
  default_value: datadir / 'glib-2.0/schemas',
)

which substitutes the datadir variable in the pkgconfig definition file with the string value of the datadir variable. Most likely you don't want to use pkgconfig to derive this installation path.

However, I don't know if it is appropriate to install this file into the Python wheel. AFAIK the gsetting schema files need to be in known locations. Similarly, I don't think that installing man pages in the wheel is appropriate, as man is most definitely not looking for man pages outside a few well known locations.