You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Wire the distributed example into CI, and say which example is which
Three maturin builds and three test invocations for #1719, plus the
documentation change that keeping three example trees requires.
The plan had been to retire `datafusion-ffi-query-planner-example` and fold it
into the new engine. That is now off, on evidence from building the engine:
roughly fifteen of its forty-seven tests cover planner *layering*, and
`dfx_engine`'s planner structurally cannot delegate to a `fallback`.
Delegating hands physical planning back to the host, which returns opaque
`ForeignExecutionPlan` nodes the engine can neither serialize nor split — a
stage-splitting planner has to plan for itself. So the new example has nothing
for those tests to nest, and deleting the crate would delete real coverage of
the most subtle part of #1679's contract.
Three trees then, with distinct jobs, which the guide now states up front
rather than leaving a reader to infer: `examples/distributed` is the worked
example and the place to start; `datafusion-ffi-example` is the
capsule-protocol test bed, one of every hook exercised hard; and
`datafusion-ffi-query-planner-example` is the planner-composition test bed.
The guide's "three roles in a query" section described only the latter two.
Two stale claims fixed while in there. `examples/README.md` linked three
`sql-on-*.py` files that do not exist. The planner example's README said its
planner "owns no serializable types of its own and deliberately uses only
built-in physical nodes", which stopped being true when `DistributedExec` was
added — and the sentence mattered, because owning a node is exactly why that
library ships its codec and planner as one bundle.
The `actionlint` pre-commit hook needs Docker and could not run here; the
workflow files are otherwise lint-clean and parse as YAML.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: examples/datafusion-ffi-query-planner-example/README.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -76,8 +76,8 @@ ctx.register_udf(provider_udf)
76
76
ctx.set_query_planner(MyQueryPlanner())
77
77
```
78
78
79
-
`MyPlannerConfig` is transferred through the foreign session. `MyQueryPlanner` reads `ffi_query_planner.max_rows`, creates the plan with `DefaultPhysicalPlanner`, and adds a built-in `GlobalLimitExec`. The test changes the setting with `SET` and verifies the new row limit.
79
+
`MyPlannerConfig` is transferred through the foreign session. `MyQueryPlanner` reads `ffi_query_planner.max_rows`, creates the plan with `DefaultPhysicalPlanner`, adds a built-in `GlobalLimitExec`, and wraps the result in a `DistributedExec` of its own. The test changes the setting with `SET` and verifies the new row limit.
80
80
81
-
The provider's codec chain is attached to the planner when it is installed and is also used to decode the returned physical plan in `datafusion-python`. Extension codecs compose: each `with_logical_extension_codec` / `with_physical_extension_codec` call appends to the session's codec chain, and each payload records which codec wrote it, so several libraries can install codecs on the same session and the order between them does not affect decoding. This planner owns no serializable types of its own and deliberately uses only built-in physical nodes. Install the codecs before the planner where possible; installing a codec afterwards rebuilds the planner against the new chain, but planner-last order is easier to audit. That rebuild is one level deep — a planner constructed with `fallback=` keeps the codecs its fallback was imported with — so codecs-first is a requirement rather than a preference once planners are layered. See [Rebinding a planner's codecs is one level deep](https://datafusion.apache.org/python/extension-guide/query-planners.html#install-codecs-before-a-layered-planner).
81
+
The provider's codec chain is attached to the planner when it is installed and is also used to decode the returned physical plan in `datafusion-python`. Extension codecs compose: each `with_logical_extension_codec` / `with_physical_extension_codec` call appends to the session's codec chain, and each payload records which codec wrote it, so several libraries can install codecs on the same session and the order between them does not affect decoding. This planner does own a node of its own — `DistributedExec`, which nothing else in the process can serialize — and that is why it ships its codec and its planner as one bundle. Install the codecs before the planner where possible; installing a codec afterwards rebuilds the planner against the new chain, but planner-last order is easier to audit. That rebuild is one level deep — a planner constructed with `fallback=` keeps the codecs its fallback was imported with — so codecs-first is a requirement rather than a preference once planners are layered. See [Rebinding a planner's codecs is one level deep](https://datafusion.apache.org/python/extension-guide/query-planners.html#install-codecs-before-a-layered-planner).
82
82
83
83
For the limits behind that choice — how the codec chain dispatches, which node kinds survive the boundary, and what a derived context shares with the context it came from — see the [Extension Guide](https://datafusion.apache.org/python/extension-guide/index.html).
0 commit comments