[LorenzoMartini]

Fixes https://issues.apache.org/jira/browse/SPARK-61253

What changes were proposed in this pull request?

Use sys.implementation.name == "pypy" to detect PyPy in the vendored cloudpickle module, and remove the unused platform import.

Why are the changes needed?

Some Python distributions ship a sys.version string containing an additional packaged by ... note. For example, a version banner can have this form:

3.12.0 | packaged by <example> | (main, Oct 2 2023, 12:00:00) [GCC 12.2.0]

Check example in python-feedstock

platform.python_implementation() calls a helper that parses the full version banner. If the parser does not recognize the additional text, it raises ValueError, causing PySpark to fail during import even though the interpreter itself is supported.

sys.implementation.name provides the interpreter's identifier directly, without parsing build metadata. Its value is lowercase ("cpython" or "pypy") and the API has been available since Python 3.3, before any Python version supported by this Spark branch. This preserves the existing PyPy-specific behavior while avoiding dependence on the formatting of sys.version.

See the Python documentation and PEP 421.

Does this PR introduce any user-facing change?

Yes. PySpark can import and use cloudpickle when a vendor-branded Python version string would otherwise cause implementation detection to raise ValueError. There are no public API changes.

How was this patch tested?

Added a regression test that starts a fresh Python process, inserts a generic packaged by example note into its real version banner, then imports cloudpickle and round-trips a lambda. A fresh process ensures a previously cached import cannot hide the failure. The test fails with the original implementation and passes with this change.

The following serialization and import tests pass on CPython 3.13.13 (12 tests):

SPARK_SKIP_CONNECT_COMPAT_TESTS=1 python/run-tests \
  --python-executables=python \
  --testnames 'pyspark.tests.test_serializers SerializationTestCase SerializersTest,pyspark.tests.test_import_spark' \
  --parallelism 1

The interpreter was selected through an activated virtual environment. Ruff lint and formatting checks and git diff --check pass. JVM-dependent tests and a PyPy runtime were not run.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex CLI 0.154.0

[dongjoon-hyun]

Thank you for making a PR. However, Apache Spark community recommends to avoid [MINOR] for the code change, @LorenzoMartini . Please file a valid JIRA ID and use it in your contributions.

[LorenzoMartini]

Hy @dongjoon-hyun , I have https://issues.apache.org/jira/browse/SPARK-61253, is that not enough? Thanks

[LorenzoMartini]

Ah sorry, realized I didn't have it in the PR title. Updated!

[dongjoon-hyun]

Yes, it should be in the title to be a commit title.

[dongjoon-hyun]

Thank you for the fix, @LorenzoMartini . However, python/pyspark/cloudpickle is a vendored copy of cloudpipe/cloudpickle (currently identical to v3.1.2), and the upstream master branch still has the same platform.python_implementation() code. Since Apache Spark upgrades this directory by replacing it with the upstream release as a whole (e.g., SPARK-54269), a Spark-only patch here would be lost silently at the next upgrade.

Could you contribute this fix to cloudpipe/cloudpickle first? We can bring it into Apache Spark via the next cloudpickle upgrade after it lands in an upstream release.

[LorenzoMartini]

Thank you! Tracking cloudpickle fix in cloudpipe/cloudpickle#624, will follow up once released

[dongjoon-hyun]

Thank you!