[SPARK-58037] [PYTHON] [TESTS] Add DataFrame API golden file test framework for PySpark#57122
Open
mihailoale-db wants to merge 2 commits into
Open
[SPARK-58037] [PYTHON] [TESTS] Add DataFrame API golden file test framework for PySpark#57122mihailoale-db wants to merge 2 commits into
mihailoale-db wants to merge 2 commits into
Conversation
mihailoale-db
force-pushed
the
df-golden-port
branch
from
072c343 to
4ee89a0
Compare
…work for PySpark This introduces a DataFrame API golden file test framework for PySpark, the Python/Spark Connect counterpart of the Scala SQLQueryTestSuite. A test is described by a `.test` file (the Reyden golden file grammar: `--! section` headers, `!-- end` delimiters) which doubles as the golden file: expected outputs (analyzed and optimized logical plans, output schema, pretty-printed result table, sha256 result hash, or expected error) are stored inline per case and rewritten in place on regeneration (`SPARK_GENERATE_GOLDEN_FILES=1`). Each case references a standalone Python script that builds the DataFrame under test (assigns `df`); cases run in file order against a fresh Spark Connect session per file (the connect counterpart of SQLQueryTestSuite's per-file `newSession()`), so state cannot leak across files. - `python/pyspark/testing/df_golden.py`: the framework (parse/serialize, output normalization, plan extraction, result formatting, execution engine, validation). - `python/pyspark/sql/tests/df_golden/test_df_golden.py`: auto-discovers `.test` files over Spark Connect (ReusedConnectTestCase); one test method per file, no code change to add a file. - `python/pyspark/sql/tests/df_golden/test_df_golden_framework.py`: pure-Python unit tests for the parse/serialize/validate/normalize/render machinery. - `python/pyspark/sql/tests/df_golden/group_by.test` + `scripts/group_by/`: first corpus, 39 GROUP BY / aggregate cases ported from the SQL golden tests. - `regenerate.sh`: wrapper that runs the module with SPARK_GENERATE_GOLDEN_FILES=1. - Registered both test modules in `dev/sparktestsupport/modules.py`; ship the `.test` files and scripts in the sdist via `python/MANIFEST.in`. This is test-only (new framework plus its first corpus; no production changes). Co-authored-by: Isaac
mihailoale-db
force-pushed
the
df-golden-port
branch
from
4ee89a0 to
3b5a84c
Compare
The DataFrame golden test scripts under df_golden/scripts are executed by the framework with `spark` injected into their namespace, so `spark` is undefined at lint time and ruff reports F821. Add a per-file-ignores entry scoped to that scripts directory. Co-authored-by: Isaac
mihailoale-db
force-pushed
the
df-golden-port
branch
from
1dd9d6d to
0ec265d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR introduces a DataFrame API golden file test framework for PySpark, the Python/Spark Connect counterpart of the Scala
SQLQueryTestSuite.Framework (
python/pyspark/testing/df_golden.py):.testfile using a golden-file grammar (--! sectionheaders,!-- enddelimiters), including a__file_metadata__header block..testfile doubles as the golden file: expected outputs are stored inline per case (analyzed and optimized logical plans, output schema, pretty-printed result table, sha256 result hash, or expected analysis/execution error) and rewritten in place on regenerati>df); cases run in file order against a shared session, so earlier cases can set up temp views for later ones. Each.testfile runs in its own fresh Spark Connect sessio>explain(mode="extended")and normalized for determinism (mirroringSQLQueryTestHelper.replaceNotIncludedMsg); golden files are generated with ANSI mode on, matching the SQL golden tests.PySparkException) can become expected error sections; Python bugs in case scripts fail the test instead of being captured into goldens. Error messages are stripped of JVM stacktraces, the== DataFrame ==query context block, and trailing plan >.testcontent fails loudly: unknown sections, header keys, and tags are rejected, as are cases that assert nothing and files with no cases.Test suite (
python/pyspark/sql/tests/df_golden/test_df_golden.py):ReusedConnectTestCaseto run over Spark Connect..testfiles and registers one test method per file; adding a new test file requires no code changes. Discovering zero files registers a failing test so a packaging problem cannot silently produce a green no-op suite.regenerate.shwrapsSPARK_GENERATE_GOLDEN_FILES=1 python/run-tests, with a--verifymode that re-runs the suite against the regenerated files.Framework unit tests (
python/pyspark/sql/tests/df_golden/test_df_golden_framework.py):Initial test coverage (
group_by.test, 39 cases):every/some/bool_and/bool_or), HAVING filters, window expressi>Why are the changes needed?
The SQL golden file suite (
SQLQueryTestSuite) only exercises the SQL parser path. DataFrame programs constructed via the API — especially over Spark Connect, which serializes the plan and tags nodes withPLAN_ID_TAG— go through a different resolution path that th>Does this PR introduce any user-facing change?
No. This is test-only (a new test framework plus its first test corpus); there are no production code changes.
How was this patch tested?
This PR is test-only.
pyspark.sql.tests.df_golden.test_df_goldenpasses, running all 39group_by.testcases over Spark Connect.group_by.testcases were generated end to end against this Spark build viaSPARK_GENERATE_GOLDEN_FILES=1and then verified by re-running the suite without regeneration (all golden comparisons pass): analyzed/optimized plans, output schema, result tables, >pyspark.sql.tests.df_golden.test_df_golden_framework(45 tests) covering.testparsing round-trip, unknown section/tag/header rejection, vacuous-case and empty-file detection, explain-section extraction, error message normalization, double >Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)