perf(gfql): route engine=auto to native polars for polars-frame graphs

[lmeyerov]

Summary

resolve_engine(AUTO) maps polars frames to Engine.PANDAS — that branch predates Engine.POLARS — so g.gfql(query) on a polars-frame graph silently bridges every query onto the generic pandas path and hands pandas frames back.

This routes engine='auto' (the default) to the native polars engine when both bound frames are polars, with an honest-NotImplementedError fallback to the legacy AUTO path. The fallback is permitted here because the user pinned no engine; the no-silent-fallback contract still applies to an explicit engine=.

This is a user-visible default-behaviour change, not a bench tweak. Scope is deliberately gfql-only (one guard in gfql()): changing resolve_engine itself would ripple into plotting/util paths that only handle pandas/cudf.

The measurements in the original body (a 2k-node microbench, "~3x / ~13x") have been replaced. Everything below is measured on dgx-spark under the exclusive perf lock, position-balanced over 8 slots, against origin/master 233b64c8.

What actually changes, at the surface

	before	after
frame type out of AUTO on a polars graph	pandas	polars (frames in = frames out)
nullable integer columns	upcast to float64 (30 → 30.0)	stay Int64 with nulls
nullable boolean columns	object	Boolean
shapes the native engine declines	ran on the generic path	still do, via the fallback (+0.23 ms)
queries carrying a policy	generic path	unchanged — deliberately not routed (see below)
explicit engine='pandas' / 'polars' / cudf	—	unchanged
graphs whose node and edge frames are not BOTH polars	—	unchanged

Perf: both directions, per-slot, with an engagement receipt

Matched graph-benchmark q1-q9, run through g.gfql(<cypher>) with no engine argument on polars-materialized frames — the surface this PR moves, and the one the committed lanes cannot see because they all pin engine=. Arms: A = 233b64c8, B = this branch. One session, slot order A B B A B A A B, --gpus all, perf lock held. 20k: 7 runs / 3 warmup; 100k: 5 / 2.

frames is the engagement receipt: the module of the returned _nodes frame, recorded per query. pandas → polars means the routing engaged on that cell. A cell that reported pandas on arm B would mean the routing did not engage there and its number would mean nothing.

20k (20,030 nodes / 260,107 edges)

q	A per-slot (ms)	B per-slot (ms)	A med	B med	Δ	verdict	frames
q1	40.17 40.52 41.31 41.80	12.77 14.24 14.49 14.49	41.31	14.49	−64.9%	FASTER	pandas→polars
q2	69.04 70.03 70.30 70.94	17.29 17.51 17.77 19.49	70.30	17.77	−74.7%	FASTER	pandas→polars
q3	22.77 22.84 23.95 24.02	7.02 7.69 7.85 10.92	23.95	7.85	−67.2%	FASTER	pandas→polars
q4	20.03 20.89 22.14 22.45	6.14 6.34 6.80 7.48	22.14	6.80	−69.3%	FASTER	pandas→polars
q5	101.10 102.67 102.81 103.57	7.37 8.10 8.61 11.86	102.81	8.61	−91.6%	FASTER	pandas→polars
q6	104.08 105.78 106.47 107.52	10.12 10.24 10.26 11.99	106.47	10.26	−90.4%	FASTER	pandas→polars
q7	39.14 39.21 39.57 40.19	10.19 10.54 10.60 11.78	39.57	10.60	−73.2%	FASTER	pandas→polars
q8	19.04 20.02 20.26 20.72	2.02 2.15 2.27 2.34	20.26	2.27	−88.8%	FASTER	pandas→polars
q9	34.23 37.68 38.48 39.03	14.71 15.27 15.72 16.25	38.48	15.72	−59.1%	FASTER	pandas→polars
nie	389.67 395.77 418.66 424.57	371.45 379.98 390.61 421.73	418.66	390.61	−6.7%	TIE	pandas→pandas

100k (107,434 nodes / 2,775,195 edges)

q	A per-slot (ms)	B per-slot (ms)	A med	B med	Δ	verdict	frames
q1	289.26 352.80 514.37 592.93	43.82 45.35 45.38 47.04	514.37	45.38	−91.2%	FASTER	pandas→polars
q2	434.52 483.38 536.71 604.67	53.75 54.94 57.25 57.51	536.71	57.25	−89.3%	FASTER	pandas→polars
q3	161.99 162.61 177.01 202.89	15.84 17.51 17.61 17.87	177.01	17.61	−90.1%	FASTER	pandas→polars
q4	154.23 154.73 165.14 167.18	12.01 12.17 13.04 13.63	165.14	13.04	−92.1%	FASTER	pandas→polars
q5	590.02 602.33 657.98 696.46	17.28 17.50 17.71 17.81	657.98	17.71	−97.3%	FASTER	pandas→polars
q6	609.75 614.65 654.05 655.09	23.51 23.94 28.03 28.59	654.05	28.03	−95.7%	FASTER	pandas→polars
q7	281.48 281.70 299.70 308.91	16.12 17.95 20.50 20.92	299.70	20.50	−93.2%	FASTER	pandas→polars
q8	161.20 168.27 172.52 174.41	5.17 5.33 6.44 6.68	172.52	6.44	−96.3%	FASTER	pandas→polars
q9	313.50 315.80 318.38 330.94	64.55 64.99 66.71 70.44	318.38	66.71	−79.0%	FASTER	pandas→polars
nie	5022 5101 5176 6662	5046 5102 5366 5450	5175.8	5365.6	+3.7%	TIE	pandas→pandas

Overlapping per-slot ranges are scored TIE, never a win. Every scored cell's per-slot ranges are disjoint by a wide margin, and canonical row values are identical on every cell of both scales.

What the untouched control arms say about the noise floor

Both scales also ran engine='pandas' and engine='polars' — paths this PR does not touch, so their true delta is zero and they measure the harness.

• 20k pandas: a uniform +1.1% to +6.5% tilt toward B on all nine, i.e. residual drift that favours A. The polars-auto wins are therefore conservative.
• 20k polars: nine TIEs and one spurious FASTER (q1, −11.9%).
• 100k pandas: flat (−1.8% to +4.2%), all TIE.
• 100k polars: eight TIEs and one spurious SLOWER (q5, +3.5%) with disjoint slot ranges.

Read that as the design's resolution floor: this 8-slot layout produces spurious non-overlapping verdicts up to about ±12% at 20k and ±5% at 100k. A claim in that band would not be safe from this harness. −59% to −97% is an order of magnitude outside it.

Explicitly hunting for what gets SLOWER

Nothing measurably regresses on the served shapes, at any of the four scales probed. The two structural ways this change can cost time were measured directly rather than assumed:

1. The declined-shape fallback tax. A shape the native engine declines pays the declined attempt and then the whole generic path. Isolated (engine='polars' timed until it raises):

graph	declined attempt	full routed call	tax
50 nodes	0.226 ms	60.1 ms	0.38%
1k nodes	0.223 ms	110.4 ms	0.20%
20k nodes	0.226 ms	952.4 ms	0.02%

A flat ~0.23 ms, independent of graph size — the decline is raised at planning, before any data is touched. On the lane's nie cell it is invisible (TIE at both scales).

2. Small-graph fixed cost. The native path could carry a larger constant than the bridge. It does not: at 50 / 1k / 20k nodes across four cheap shapes, arm B is faster or equal on every served cell (e.g. 50-node cheap_scan 12.02 → 1.16 ms; 50-node cheap_seed 21.66 → 1.17 ms).

VALUES, not just latency

A 45-shape matrix (node scans, typed/untyped filters, nulls, toLower, arithmetic, string concat, seeded 1-hop and 2-hop, undirected, five aggregate families, group-by, DISTINCT, ORDER BY/SKIP/LIMIT, multi-alias joins, OPTIONAL MATCH, variable-length, shortestPath, WITH, edge-property filters, whole-entity returns, empty results, and three native chain-API shapes) was run under AUTO on both builds, comparing row counts, canonical values, column dtypes, column order and row order.

Row values: 0 mismatches across all 45 shapes once integer/float representation is normalised, and 0 status changes. 44/45 routed natively; shortestPath took the fallback.

The differences that are real, and which the reviewer should rule on:

1. Frame type — pandas → polars everywhere. The declared contract change.
2. Nullable integers — 30.0 → 30. The old default upcast to float64 because pandas had no nullable int; polars keeps Int64. Values are equal numerically; the printed representation and the dtype both change. Same for nullable booleans (object → Boolean).
3. Column ORDER on the returned _edges — an aliased edge column moves from first to last (['r','s','d',…] → ['s','d',…,'r']) in 5 of 45 shapes. Column sets are identical.
4. A leaked internal column disappears — a -[*1..2]-> result carried __gfql_output_edge_hop__ in _edges under the old default and does not under the new one.

Two divergences that are NOT introduced here but that this PR makes the DEFAULT

These already exist between engine='polars' and engine='pandas' today. They are surfaced because a default-routing change is what promotes them from opt-in to universal. I am not proposing to decide these unilaterally.

query	old default	new default
RETURN sum(a.<string col>)	'abac' (pandas concatenates strings)	raises polars.exceptions.InvalidOperationError
RETURN avg(a.<string col>)	raises GFQLTypeError [type-mismatch]	silently returns null

min/max over strings agree on both engines.

Neither old behaviour is openCypher-correct (both should be type errors), but the directions differ in how much they matter:

• sum() — a working query becomes a hard failure, and the exception is a raw polars type leaking through the GFQL surface rather than a GFQLTypeError. That is an error-hygiene defect independent of the routing question.
• avg() — a query that used to raise a typed error now silently returns null. This is the direction that worries me: it is the silent-wrong-answer class, arriving by default.

My recommendation is that the avg()-on-string case be fixed in the polars engine (decline or raise GFQLTypeError) before this lands, and the sum() case at least be wrapped so a GFQLTypeError is raised instead of a leaked polars exception. Both are small and separable from this PR. Found by a 107-query differential sweep of AUTO-vs-AUTO; the other 99 were identical.

5. One extra OTel span per routed query. The guard recurses through the module-level gfql name, which is the @otel_traced("gfql.run")-decorated wrapper, so an AUTO-routed polars query now enters it twice (measured: 2 entries per user call on the routed path, 1 on the pandas path and 1 on explicit engine='polars'). Each user call therefore emits a nested duplicate gfql.run span and evaluates _gfql_otel_attrs twice. Results are unaffected. I have not changed this: the clean fix is to split the body into an undecorated _gfql_impl and have the guard call that, which is a mechanical but sizeable move on a file several other in-flight PRs touch, and a nested span arguably documents the routing decision. Flagging it as a reviewer call rather than deciding it here. (gfql.__wrapped__ would be a one-token fix but is untypeable without a banned cast.)

The policy hooks: a defect found, and fixed here

The original guard routed policy-carrying queries too. Measured, that was not safe:

• The native executor does not go through chain_impl, so it never emits postload / postchain. A policy that DENIES on postload blocked the query before and did not block it after. A governance hook that silently stops firing is worse than a slow query, and it would have applied to every user who never pinned an engine.
• The NIE fallback re-ran the query, firing preload / precompile / postcompile twice for one user call — a policy that counts or rate-limits would have double-counted.

policy is None is now part of the guard. The full hook trace and every deny/allow outcome across all six hooks is byte-identical to 233b64c8 on polars-frame graphs, pandas-frame graphs, and the declining shape. Explicit engine='polars' is unchanged and still carries the underlying gap — that gap is a separate, pre-existing bug worth its own issue; this PR only refuses to make it the default.

Regression lock-in

graphistry/pyg-bench#110 adds auto-engine-default-route to the gfql-perf-lockin lane.

It is a structural receipt, not a ladder, and that is the point: a growth ratio measures the fraction of a query proportional to the scaled dimension, and routing moves a constant, so a ladder would police the wrong direction and go green on a build that had silently reverted. The probe runs the named seeded pattern through the bare g.gfql(query) spelling on polars frames and asserts 2 = native polars frames out and zero pl.DataFrame.to_pandas calls; 1 = pandas frames out of a polars-frame graph (regression); 0 = invalid. Its positive and negative controls (engine='polars' must report 2, engine='pandas' must report 1) are part of the observation, so a probe that cannot discriminate cannot go green.

Validated both ways through the committed runner: 233b64c8 → 1 on every rung, verdict regression, exit 1; the same tree with this PR → 2 on every rung, exit 0. rows_returned {nodes: 33, edges: 32} on every rung of both.

Merge order: pyg-bench#110 is deliberately held until this lands. Against current pygraphistry master the probe correctly reports a regression — which is the proof that the gate works, but merging it first would leave gfql-perf-lockin red for a change that has not shipped, and a lane that is red for a legitimate reason is a lane people learn to ignore.

Tests

Positive: AUTO result frame-equal to explicit polars; edges-only graph (self._nodes is None is inside the guard's condition); both spellings of AUTO (enum and string); shortestPath NIEs when pinned and answers via the fallback.

Negative — for a routing change, what must not be routed matters as much: pandas frames untouched; mixed polars/pandas frames untouched; explicit engine='pandas' untouched; a denying postload policy still blocks; the compile/load hooks fire exactly once on a declining shape.

Verification

bin/lint.sh clean · bin/typecheck.sh clean (325 files) · graphistry/tests/compute/ — 123 failed on this branch and 123 failed on 233b64c8 in the same env, identical failure set (local umap/dask/env failures, unrelated and pre-existing); passes 6806 → 6815, the +9 being the tests added here.

🤖 Generated with Claude Code

https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

[lmeyerov]

Too early to enable by default

[lmeyerov]

Branch updated to land-ready state (owner's hold stands — not merging, staying draft).

Rebase state: merged current origin/master (6ab6d64) into the branch as merge commit 98f8e20 (repo convention: merge, not rebase). Only CHANGELOG.md conflicted; all code auto-merged. The changelog entry was rewritten to describe behavior only — AUTO on all-polars-frame graphs routes to the native polars engine with legacy fallback on decline; frames out follow the serving engine (polars for served shapes, pandas for declined ones); explicit engines and policy-bearing calls unchanged — and the stale perf numbers (q1–q9 timings, the 0.23 ms decline cost) were removed per the docs convention that perf numbers live in performance.rst/pyg-bench, not scattered changelog entries.

GPU scoping: AUTO selects CPU polars only. Engine.POLARS_GPU remains explicit opt-in — the guard routes to Engine.POLARS.value and nothing else, matching master's doctrine that AUTO never selects polars-gpu. cuDF frames never enter the guard at all (is_polars_df is a polars-module check), so the legacy AUTO→CUDF resolution is byte-for-byte unchanged — now pinned by TestAutoEngineCudfUntouched (importorskip-gated; it actually ran and passed on a local RTX 3080 Ti + cudf 25.10 box, not just skipped).

New cliff-regression receipt (commit 9e0dba3):

• test_auto_engine_gfql_serves_polars_index_1767_cliff — polars frames + explicit gfql_index_all(engine='polars') + g.gfql(...) with no engine argument now serves path=index, engine=polars per index_trace(). Before this PR, resolve_engine(AUTO) mapped polars frames to pandas, engine-mismatching the resident polars index into a silent scan (fix(gfql): index AUTO preserves resident polars frames (kills the polars-hop O(E) tax) #1767).
• test_auto_engine_hop_residual_still_scans_1767 — the scope boundary as executable documentation: direct g.hop() with no engine still resolves AUTO to pandas and declines with decision_code=engine_mismatch.
• TestAutoEngineLazyFrames — LazyFrame edges+nodes with engine unset route native and return eager polars equal to explicit engine='polars'.
• TestAutoEnginePandasOracleParity — AUTO-on-polars-frames answers equal the pandas-engine oracle in value for filter+traverse, aggregate, and WHERE-bearing shapes.

Scope boundary: this PR routes gfql() AUTO only. Direct hop(), chain(), and index-build AUTO resolution remain follow-ups (#1676/#1715 territory); mixed-frame graphs and policy-bearing calls stay on the generic path by design.

Local validation (CPU): conformance suite 150 passed / 0 failed; test_lowering.py 1403 passed / 27 skipped; index suite CPU lane 123/123 passed; ruff clean on changed files; scoped mypy adds zero new errors; type-hygiene ratchet OK. The cudf/polars-gpu-parameterized index failures on the local box (65) are byte-identical to clean origin/master (CuPy missing libnvrtc.so.12 — box env, pre-existing).

DGX-deferred validation: (1) cudf + polars-gpu index/conformance lanes on a working RAPIDS stack; (2) re-measurement of the routing win and the decline-fallback cost under the dgx-spark perf lock in pyg-bench (numbers deliberately removed from the changelog until then); (3) polars-version-drift check in-container (image 1.35.x vs local 1.42).

🤖 Generated with Claude Code

https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

[lmeyerov]

1. check: before + after, presumably auto with cudf data -> cudf out
2. new, as both cudf and polars-gpu can work for cudf->cudf, wrt engine: if polars-gpu is avialable, prefer it over cudf for auto

[lmeyerov]

Owner-directed policy addition (2026-08-02): AUTO on cuDF-frame graphs prefers polars-gpu when genuinely usable

Pushed as fc6decd1 (merge of current origin/master, incl. cache registry #1836 + transformer hoist #1837 — clean merge, no conflicts) and 90d69b97 (the policy addition). This supersedes the body's "AUTO never selects polars-gpu, and cuDF-frame graphs are untouched" line for the cuDF arm only, by owner decision.

What changed

• Guard order (graphistry/compute/gfql_unified.py): all-polars frames → engine='polars' (unchanged, first) → all-cuDF frames AND probe-True → the new polars-gpu route → otherwise unchanged. _is_cudf_frame uses the same cudf.core.dataframe module test as resolve_engine's legacy arm, so the two arms agree on what a cuDF-frame graph is (dask_cudf and cudf.pandas proxies excluded).
• The probe (graphistry/compute/gfql/lazy/__init__.py::polars_gpu_available): a REAL probe, not a package listing — polars imports, cudf + cudf_polars installed, and a tiny lazy plan actually collects on the same pl.GPUEngine(raise_on_fail=True) the real collects use. A box with RAPIDS wheels but a broken driver/toolchain (the missing-libnvrtc class: imports succeed, kernels fail) probes False and stays on the legacy path. lru_cache(maxsize=1) process singleton, registered EXEMPT in the GFQL cache registry with the written reason; the cache coverage lock is green.
• cudf in → cudf out on the served path: inputs cross cuDF→Arrow→polars inside chain dispatch's existing _coerce_input_formats; results cross back polars→Arrow→cuDF in _route_result_frames_to_cudf. df_to_engine(polars_frame, Engine.CUDF) (graphistry/Engine.py) gained the Arrow interchange — the mirror of the existing cuDF→Arrow→polars path — so neither direction takes the lossy pandas detour (nullable Int64/Boolean survive).
• Decline/fallback: identical shape to the polars arm — any NotImplementedError (honest engine decline, GPU-collect failure via lazy._gpu_raise, or a cudf-out conversion failure, which the boundary wraps as NIE) falls back to the legacy CUDF path with identical values. Explicit engine= always wins; policy= present bypasses (the native executor's postload/postchain hook gap must not become the default — same reasoning already in this PR).

Test tiers

• CPU-runnable anywhere: probe graceful-False legs (cudf missing / cudf_polars missing / GPU collect raising), memoization + registry-exemption pins; guard-order, explicit-engine, policy-bypass, mixed-frame, edges-only pins via the _polars_gpu_probe / _auto_cudf_polars_gpu_route seams.
• cudf-runnable (this box: RTX 3080 Ti, cudf 25.10, broken GPU kernels — probe genuinely False): legacy-path cudf-in/cudf-out TYPE pins (before AND after semantics — probe pinned False so they hold on GPU lanes too), NIE-fallback value parity, and the REAL route body on CPU via the _AUTO_CUDF_ROUTE_ENGINE='polars' seam — guard → Arrow coercion → native engine → Arrow cudf-out — including nullable-int survival across both Arrow boundaries and a real engine decline (shortestPath) landing back on legacy.
• DGX-DEFERRED (TestAutoEngineCudfPolarsGpuExecutionDGX, skipif probe False, same convention as test_engine_polars_gpu.py; not run — GPU box is under a locked benchmark):
  1. test_auto_on_cudf_routes_polars_gpu_and_returns_cudf — real engagement receipt + cudf frame types out
  2. test_gpu_route_values_match_legacy_cudf_path — value parity routed vs legacy
  3. test_explicit_polars_gpu_still_returns_polars — explicit engine keeps polars frames (no cudf-out boundary)

Verification (this box)

set -o pipefail on every pytest pipeline. Conformance suite 168 passed / 3 skipped (the DGX tier); cache-coverage lock 9 passed; index suite CPU lane: 213 passed, the 65 cudf-lane failures byte-identical to the pre-change baseline (local broken-kernel env, pre-existing — verified by diffing failure sets on a HEAD worktree); ruff clean; mypy clean (328 files); bin/ci_type_hygiene_guard.py OK (no growth).

🤖 Generated with Claude Code

https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

[lmeyerov]

DGX execution results (2026-08-02, official 26.02 container, --gpus all, safe_run): 25 passed, 0 skipped, 4.31s — including all three previously-deferred execution tests. The availability probe returned TRUE in-container (cudf-polars present; the real polars-gpu collect ran), so these are genuine GPU-engagement receipts: routed-vs-legacy value parity holds, cudf-in→cudf-out types hold, explicit polars-gpu contract unchanged. In-container polars is 1.35.2 (known image drift; the sanctioned measurement environment). The DGX-deferred list from the implementation comment is now fully validated. CI is rerunning on cb64a41 (coverage-floor test fix). Landing remains the owner's call.

🤖 Generated with Claude Code

https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

[lmeyerov]

Superseded by the successor PR from the identical commit — this PR object stopped receiving Actions events (dispatch runs green on the SHA but never attached here). All evidence and authorization carry over.