Run examples/*.py in CI, fix the broken example, and give silent ones output - #1736
Open
voidstackloop wants to merge 1 commit into
Open
Run examples/*.py in CI, fix the broken example, and give silent ones output#1736voidstackloop wants to merge 1 commit into
voidstackloop wants to merge 1 commit into
Conversation
… output Nothing in CI ran the top-level examples/*.py scripts, so they drifted out of sync with the API. examples/csv-read-options.py crashed because it reads data.csv and data.csv.gz that do not exist in the repository, and nine other examples end in asserts without printing anything, so a reader cannot tell a working example from a no-op. - Make csv-read-options.py self-contained: it now writes its own small CSV and gzipped CSV into a temporary directory. - Add a terminal call to the examples that printed nothing so each one shows its result. - Add a CI step, gated to the 3.12 abi3 entry, that runs every examples/*.py script against the built wheel with an explicit skip list for examples that need network/credentials, hand-downloaded data, generated TPC-H data, or the optional Ray dependency. Closes apache#1728
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1728.
Rationale for this change
Nothing in CI runs the top-level
examples/*.pyscripts, so they silently driftfrom the API and from each other. Two consequences are already in the tree:
examples/csv-read-options.pycrashes: it readsdata.csvanddata.csv.gz, neither of which exists in the repository.assertwith no output, so a reader cannot tell aworking example from a script that does nothing.
Adding a CI job is what keeps this from recurring.
What changes are included in this PR?
examples/csv-read-options.pynow writes its own small CSV and gzipped CSVinto a temporary directory, so it is self-contained and runs without external
data.
export.py,import.py,python-udaf.py,python-udf.py,query-pyarrow-data.py,sql-to-pandas.py,sql-using-python-udaf.py,sql-using-python-udf.py,substrait.py) nowprint their result, keeping their existing assertions.
.github/workflows/test.ymlgains a Run example scripts step (Python 3.12entry) that runs every
examples/*.pyagainst the built wheel. It installsthe extra runtime dependencies some examples use (
pandas,polars,matplotlib) and skips an explicit list of examples that cannot run in CI:sql-parquet-s3.py— needs network access and AWS credentialssql-parquet.py,dataframe-parquet.py,sql-to-pandas.py— need the NYCtaxi parquet file documented in
examples/README.mdpython-udf-comparisons.py— needs the TPC-H dataset generated later in theworkflow
ray_pickle_expr.py— needs the optional (heavy)raydependencyAre there any user-facing changes?
No API changes. The examples are documentation; they are now runnable, print
their results, and CI will catch it if they break again.
Verification
datafusion==54.0.0wheel in a freshvenv: all pass.
multiprocessing_pickle_expr.pyandray_pickle_expr.pywerenot run locally (Windows has no
forkserverand Ray is unavailable there);they run on the Linux CI.
ruff check/ruff format --check(pinned 0.15.1),codespellandactionlint(v1.7.12) pass on the changed files.