Skip to content

Avoid parsing sys.version when detecting the Python implementation - #624

Open
LorenzoMartini wants to merge 1 commit into
cloudpipe:masterfrom
LorenzoMartini:fix/python-implementation-name
Open

LorenzoMartini wants to merge 1 commit into
cloudpipe:masterfrom
LorenzoMartini:fix/python-implementation-name

Conversation

@LorenzoMartini

@LorenzoMartini LorenzoMartini commented Nov 9, 2026 •

Copy link
Copy Markdown

Importing cloudpickle can raise ValueError on Python builds that include distributor text in sys.version, such as 3.13.13 | packaged by example | (...) [...]. The import-time platform.python_implementation() call parses that string to determine the interpreter implementation.

Use sys.implementation.name instead. It provides the implementation identifier directly, independently of the version string's formatting, and is available on all supported Python versions. The comparisons use its lowercase identifiers ("pypy" and "cpython"). The same change is applied to implementation checks in both test modules, and the unused platform imports are removed.

There is a concrete example of this formatting variation in conda-forge's Python branding patch: it adds a packaged by note and also patches the platform parser to accept it. PEP 421 describes the motivation for exposing implementation information directly.

The regression test adds a generic packaged by note to sys.version in a fresh interpreter, then imports cloudpickle and round-trips a lambda. It fails with the original implementation and passes with this change.

This follows the request to contribute the fix upstream in apache/spark#60244, so Spark can receive it through a future cloudpickle release.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant