Skip to content

Commit a06e53b

Browse files
timsaucerclaude
andcommitted
Wire the distributed example into CI, and say which example is which
Three maturin builds and three test invocations for #1719, plus the documentation change that keeping three example trees requires. The plan had been to retire `datafusion-ffi-query-planner-example` and fold it into the new engine. That is now off, on evidence from building the engine: roughly fifteen of its forty-seven tests cover planner *layering*, and `dfx_engine`'s planner structurally cannot delegate to a `fallback`. Delegating hands physical planning back to the host, which returns opaque `ForeignExecutionPlan` nodes the engine can neither serialize nor split — a stage-splitting planner has to plan for itself. So the new example has nothing for those tests to nest, and deleting the crate would delete real coverage of the most subtle part of #1679's contract. Three trees then, with distinct jobs, which the guide now states up front rather than leaving a reader to infer: `examples/distributed` is the worked example and the place to start; `datafusion-ffi-example` is the capsule-protocol test bed, one of every hook exercised hard; and `datafusion-ffi-query-planner-example` is the planner-composition test bed. The guide's "three roles in a query" section described only the latter two. Two stale claims fixed while in there. `examples/README.md` linked three `sql-on-*.py` files that do not exist. The planner example's README said its planner "owns no serializable types of its own and deliberately uses only built-in physical nodes", which stopped being true when `DistributedExec` was added — and the sentence mattered, because owning a node is exactly why that library ships its codec and planner as one bundle. The `actionlint` pre-commit hook needs Docker and could not run here; the workflow files are otherwise lint-clean and parse as YAML. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 8a52de0 commit a06e53b

5 files changed

Lines changed: 81 additions & 18 deletions

File tree

.github/workflows/build.yml

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,38 @@ jobs:
206206
args: --out dist
207207
rustup-components: rust-std
208208

209+
# The three libraries of the distributed example. Built in dependency
210+
# order for readability only; they are independent cdylibs.
211+
- name: Build distributed example storage library
212+
if: matrix.python-tag == 'abi3'
213+
uses: PyO3/maturin-action@v1
214+
with:
215+
target: x86_64-unknown-linux-gnu
216+
manylinux: "2_28"
217+
working-directory: examples/distributed/storage-library
218+
args: --out dist
219+
rustup-components: rust-std
220+
221+
- name: Build distributed example UDF library
222+
if: matrix.python-tag == 'abi3'
223+
uses: PyO3/maturin-action@v1
224+
with:
225+
target: x86_64-unknown-linux-gnu
226+
manylinux: "2_28"
227+
working-directory: examples/distributed/udf-library
228+
args: --out dist
229+
rustup-components: rust-std
230+
231+
- name: Build distributed example engine library
232+
if: matrix.python-tag == 'abi3'
233+
uses: PyO3/maturin-action@v1
234+
with:
235+
target: x86_64-unknown-linux-gnu
236+
manylinux: "2_28"
237+
working-directory: examples/distributed/engine-library
238+
args: --out dist
239+
rustup-components: rust-std
240+
209241
- name: Archive wheels
210242
uses: actions/upload-artifact@v7
211243
with:
@@ -220,6 +252,9 @@ jobs:
220252
path: |
221253
examples/datafusion-ffi-example/dist/*
222254
examples/datafusion-ffi-query-planner-example/dist/*
255+
examples/distributed/storage-library/dist/*
256+
examples/distributed/udf-library/dist/*
257+
examples/distributed/engine-library/dist/*
223258
224259
# ============================================
225260
# Build - Linux ARM64

.github/workflows/test.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,15 @@ jobs:
127127
uv run --no-project pytest python/tests/_test*.py
128128
cd ../datafusion-ffi-query-planner-example
129129
uv run --no-project pytest python/tests/_test*.py
130+
# The distributed example. Its tests spawn worker processes with
131+
# `sys.executable`, so they need the same interpreter the wheels
132+
# were installed into -- which `uv run` gives them.
133+
cd ../distributed/storage-library
134+
uv run --no-project pytest python/tests/_test*.py
135+
cd ../udf-library
136+
uv run --no-project pytest python/tests/_test*.py
137+
cd ../engine-library
138+
uv run --no-project pytest python/tests/_test*.py
130139
131140
- name: Run tpchgen-cli to create 1 Gb dataset
132141
if: matrix.wheel-tag == 'abi3'

docs/source/extension-guide/index.md

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -59,11 +59,27 @@ this section only makes sense once they are distinct in your head:
5959
the codecs that serialize them.
6060
- **A planner library** — owns a query planner and the configuration it needs.
6161

62-
The worked examples in this repository use two separate crates,
63-
[`datafusion-ffi-example`] and [`datafusion-ffi-query-planner-example`], so
64-
each role has a distinct shared-library identity. A real library may play more
65-
than one role; keeping them separate in the examples is what makes the
66-
boundaries observable.
62+
A real library may play more than one role. Keeping them in separate crates is
63+
what makes the boundaries observable, because each one is then a distinct
64+
shared library and the FFI conversions are real rather than same-library
65+
downcasts.
66+
67+
The examples in this repository are three trees, and it is worth knowing which
68+
one answers your question:
69+
70+
- [`examples/distributed`] is the **worked example**, and the place to start.
71+
Three libraries — functions, tables, and an engine — cooperate on one query
72+
whose leaf stage runs in separate worker processes. It is also the only
73+
example whose plans genuinely leave the process, so it is where the codecs
74+
encode durable metadata rather than tokens.
75+
- [`datafusion-ffi-example`] is the **capsule-protocol test bed**: one of every
76+
hook, exercised hard. Read it to see the shape of a getter, not to see a
77+
library designed well.
78+
- [`datafusion-ffi-query-planner-example`] is the **planner-composition test
79+
bed**: what happens when more than one library contributes a query planner,
80+
and how `fallback` nests them. The distributed example cannot cover this —
81+
a planner that rewrites the plan into stages has to plan for itself, so it
82+
has no use for a fallback.
6783

6884
The session owns the codecs used for the exchange and supplies them to the
6985
foreign planner. That is what lets the planner decode provider-owned objects,
@@ -134,3 +150,4 @@ checklist
134150

135151
[`datafusion-ffi-example`]: https://github.com/apache/datafusion-python/tree/main/examples/datafusion-ffi-example
136152
[`datafusion-ffi-query-planner-example`]: https://github.com/apache/datafusion-python/tree/main/examples/datafusion-ffi-query-planner-example
153+
[`examples/distributed`]: https://github.com/apache/datafusion-python/tree/main/examples/distributed

examples/README.md

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -51,23 +51,25 @@ Here is a direct link to the file used in the examples:
5151

5252
### Rust FFI Extensions
5353

54-
- [Table providers, functions, and codecs](./datafusion-ffi-example/)
55-
- [Independent query planner and planner configuration](./datafusion-ffi-query-planner-example/)
54+
Start with the worked example; the other two are focused test beds that
55+
exercise one part of the protocol hard rather than reading as a tutorial.
5656

57-
These two crates form a three-library interoperability example with
58-
`datafusion-python`. They are separate shared libraries so the tests exercise real FFI
59-
type and codec boundaries rather than same-library Rust downcasts.
57+
- [**Three libraries in one distributed query**](./distributed/) — a UDF
58+
library, a table provider with its own scan node, and a toy engine that
59+
splits the plan and runs each partition in a separate process. Read this one
60+
first.
61+
- [Capsule protocol conformance](./datafusion-ffi-example/) — table providers,
62+
catalogs, functions, config, and codecs, one of each.
63+
- [Query planner composition](./datafusion-ffi-query-planner-example/) — what
64+
happens when more than one library contributes a planner, and how they nest.
65+
66+
Each is a separate shared library, so the tests exercise real FFI type and
67+
codec boundaries rather than same-library Rust downcasts.
6068

6169
### Substrait Support
6270

6371
- [Serialize query plans using Substrait](./substrait.py)
6472

65-
### Executing SQL against DataFrame Libraries (Experimental)
66-
67-
- [Executing SQL on Polars](./sql-on-polars.py)
68-
- [Executing SQL on Pandas](./sql-on-pandas.py)
69-
- [Executing SQL on cuDF](./sql-on-cudf.py)
70-
7173
## TPC-H Examples
7274

7375
Within the subdirectory `tpch` there are 22 examples that reproduce queries in

examples/datafusion-ffi-query-planner-example/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,8 @@ ctx.register_udf(provider_udf)
7676
ctx.set_query_planner(MyQueryPlanner())
7777
```
7878

79-
`MyPlannerConfig` is transferred through the foreign session. `MyQueryPlanner` reads `ffi_query_planner.max_rows`, creates the plan with `DefaultPhysicalPlanner`, and adds a built-in `GlobalLimitExec`. The test changes the setting with `SET` and verifies the new row limit.
79+
`MyPlannerConfig` is transferred through the foreign session. `MyQueryPlanner` reads `ffi_query_planner.max_rows`, creates the plan with `DefaultPhysicalPlanner`, adds a built-in `GlobalLimitExec`, and wraps the result in a `DistributedExec` of its own. The test changes the setting with `SET` and verifies the new row limit.
8080

81-
The provider's codec chain is attached to the planner when it is installed and is also used to decode the returned physical plan in `datafusion-python`. Extension codecs compose: each `with_logical_extension_codec` / `with_physical_extension_codec` call appends to the session's codec chain, and each payload records which codec wrote it, so several libraries can install codecs on the same session and the order between them does not affect decoding. This planner owns no serializable types of its own and deliberately uses only built-in physical nodes. Install the codecs before the planner where possible; installing a codec afterwards rebuilds the planner against the new chain, but planner-last order is easier to audit. That rebuild is one level deep — a planner constructed with `fallback=` keeps the codecs its fallback was imported with — so codecs-first is a requirement rather than a preference once planners are layered. See [Rebinding a planner's codecs is one level deep](https://datafusion.apache.org/python/extension-guide/query-planners.html#install-codecs-before-a-layered-planner).
81+
The provider's codec chain is attached to the planner when it is installed and is also used to decode the returned physical plan in `datafusion-python`. Extension codecs compose: each `with_logical_extension_codec` / `with_physical_extension_codec` call appends to the session's codec chain, and each payload records which codec wrote it, so several libraries can install codecs on the same session and the order between them does not affect decoding. This planner does own a node of its own — `DistributedExec`, which nothing else in the process can serialize — and that is why it ships its codec and its planner as one bundle. Install the codecs before the planner where possible; installing a codec afterwards rebuilds the planner against the new chain, but planner-last order is easier to audit. That rebuild is one level deep — a planner constructed with `fallback=` keeps the codecs its fallback was imported with — so codecs-first is a requirement rather than a preference once planners are layered. See [Rebinding a planner's codecs is one level deep](https://datafusion.apache.org/python/extension-guide/query-planners.html#install-codecs-before-a-layered-planner).
8282

8383
For the limits behind that choice — how the codec chain dispatches, which node kinds survive the boundary, and what a derived context shares with the context it came from — see the [Extension Guide](https://datafusion.apache.org/python/extension-guide/index.html).

0 commit comments

Comments
 (0)