Add Support for pip's --platform, --no-binary, & --only-binary options

[MLBZ521]

This feature allows the user to specify additional pip options to filter against when installing libraries.

Particularly, I ran into an issue where I was wanting to build a "Universal" relocatable Python.Framework, but pip will natively install the a matching architecture wheel if available, even if a universal build is available.

With the support for these three options, I also included the ability to pass multiple values for each option (which pip supports), but optparse does not, so a callback function was added in locallibs/__init__.py to support this.

Example usage would be:

./make_relocatable_python_framework.py \
  --os-version 11 \
  --python-version 3.11.6 \
  --pip-requirements requirements.txt \
  --pip-platform macosx_11_0_universal2 macosx_11_0_arm64 macosx_10_9_x86_64 \
  --only-binary :all: \
  --upgrade-pip

[gregneagle]

Can all that complexity not just go into the requirements file?

Have you found out what versions of pip support all of these options?

[MLBZ521]

From the research I've done, any conditionals that are applied in the requirements file are applied to the system that is current running pip. Where I'm trying to build a relocatable Python Framework that is system agnostic (at least, for macOS).

Example:

• If make_relocatable_python_framework.py is ran on an Intel Mac, pip will install x86_64 wheels
• If make_relocatable_python_framework.py is ran on an ARM Mac, pip will install arm64 wheels

This creates an issue if trying to deploy at single relocatable framework .pkg.

Obviously a work around would be to simply create two different relocatable frameworks; one Intel based and one ARM based, but if Universal wheel binaries exist, this does allow for a single "universal" relocatable framework.

[MLBZ521]

Regarding what versions of pip support these options:

From the pip change log:

• 20.3b1 (2020-10-31)

  • Allow multiple values for --abi and --platform. (#6121)

• 18.1 (2018-10-05)

  • Allows dist options (--abi, --python-version, --platform, --implementation) when installing with --target (#5355)

• 7.0.0 (2015-05-21)

  • --no-use-wheel and --use-wheel are deprecated in favour of new options --no-binary and --only-binary. The equivalent of --no-use-wheel is --no-binary=:all:. (#2699)

[gregneagle]

I feel like I'm missing something here, since the Munki pkg build script uses make_relocatable_python_framework.py to build a Universal framework, and pretty sure Autopkg does as well.

[gregneagle]

The requirements file the Munki pkg build script uses is here: https://github.com/munki/munki/blob/main/code/tools/py3_requirements.txt and uses --no-binary :all:

You can see how make_relocatable_python_framework.py is called here: https://github.com/munki/munki/blob/main/code/tools/build_python_framework.sh#L51-L58

[MLBZ521]

Ah, ok... So a couple things:

• I looked for this kind of support in pip -r last week, but completely missed it or....my Google Fu wasn't working last week: https://pip.pypa.io/en/stable/reference/requirements-file-format/

  • This helps quite a bit

• In Munki, everything is built from source and not using wheels, which I was trying the opposite approach

  • If I specify the --no-binary option for the packages that end up not being Universal in my relocatable framework, that almost gets me there; the one that fails, is due to missing a Rust compiler

I think that clears some things up. I guess it just depends on which approach you wanted to take?

[MLBZ521]

Well, I dunno. I've tried the --no-binary option in a requirements.txt similar to Munki, yet I still end up with a non-universal libraries.

Example:

Python.framework/Versions/3.11/bin/python3 -s -m pip install cryptography --no-cache-dir --no-binary :all:
[...]
Building wheel for cryptography (pyproject.toml) ... done
  Created wheel for cryptography: filename=cryptography-42.0.1-cp311-cp311-macosx_12_0_universal2.whl size=1245037 sha256=be72eee20e7eda79a20a4691360abac3861eedd3a9433eee5d1e24412686f7fa
[...]
Successfully built cryptography cffi pycparser
Installing collected packages: pycparser, cffi, cryptography
Successfully installed cffi-1.16.0 cryptography-42.0.1 pycparser-2.21

Even though cryptography built wheel is labeled universal2, it still fails python_universal_tester.sh:

bash python_universal_tester.sh 3.11
Using Python 3.11
      94 libraries (*.so and *.dylib) found in the framework; only       93 are universal!
The following libraries are not universal:
[...]

But if I do:

Python.framework/Versions/3.11/bin/python3 -s -m pip install -r requirements.txt --no-cache-dir --platform macosx_10_12_universal2 --only-binary=:all: --target=Python.framework/Versions/3.11/lib/python3.11/site-packages
Collecting cryptography (from -r requirements.txt (line 1))
  Downloading cryptography-42.0.1-cp39-abi3-macosx_10_12_universal2.whl.metadata (5.3 kB)
Collecting cffi (from -r requirements.txt (line 2))
  Downloading cffi-1.16.0.tar.gz (512 kB)
[...]
Collecting pycparser (from -r requirements.txt (line 4))
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
[...]
Downloading cryptography-42.0.1-cp39-abi3-macosx_10_12_universal2.whl (5.9 MB)
[...]
Building wheels for collected packages: cffi
[...]
  Created wheel for cffi: filename=cffi-1.16.0-cp311-cp311-macosx_10_9_universal2.whl size=264691 sha256=701968fd990507d331bce36a541a5236fcad6644310b0e4ee2e2643e9793feb7
[...]
Successfully built cffi
Installing collected packages: pycparser, cffi, cryptography
Successfully installed cffi-1.16.0 cryptography-42.0.1 pycparser-2.21

I get the desired results:

bash python_universal_tester.sh 3.11
Using Python 3.11
All files are universal!

I've tried different combinations of things, including a different version of cryptography. The only thing I can think of is if there's something wrong with cryptography's build process that isn't actually creating a universal library, but that's guess; I'm not familiar with this.