perf(gfql): route engine=auto to native polars for polars-frame graphs (supersedes #1743) - #1849
Merged
Merged
Conversation
resolve_engine(AUTO) maps polars frames to PANDAS (it predates Engine.POLARS), so g.gfql(query) on a polars-frame graph silently bridged to the generic pandas path: ~3-13x slower on cypher point queries and pandas frames out. Route AUTO to the native polars engine; an honest NotImplementedError (unsupported shape) falls back to the legacy AUTO path — allowed because the user did not pin an engine. Frames in = frames out: AUTO results on polars graphs are now polars. Repro: 2k-node polars graph, seeded 1-hop cypher — AUTO 25.0ms/pandas out before, 9.3ms/polars out after (engine='polars' = 8.8ms); polars-NIE shapes (shortestPath) still answer via the pandas fallback. Fixes the q5 finding in plans/gfql-benchmark-numbers (inferred-engine 13x penalty). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…c records The dual-path guards checked hasattr(out, 'to_dict') first, but polars DataFrame HAS to_dict (without orient) — so the polars branch was unreachable and these tests only ever ran because auto silently returned pandas. With auto routing polars-frame graphs natively they now receive polars frames; probe to_dicts first. Assertions (record values) unchanged and still pass on both engines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
The routing change is only as safe as the set of inputs it refuses, and one refusal is load-bearing rather than conservative. The native polars executor does not go through `chain_impl`, so it never emits the `postload` / `postchain` policy hooks that path emits. Measured on a polars-frame graph: a policy that DENIES on `postload` blocks the query under the generic path and does NOT block it under the native one. Routing a policy-carrying query there would therefore have silently stopped enforcing a denying postload policy for every user who never pinned an engine -- a governance hook that stops firing is worse than a slow query. The NIE fallback compounded it: re-running the query fired `preload` / `precompile` / `postcompile` TWICE for one user call, so a policy that counts or rate-limits would have double-counted. `policy is None` in the guard restores both, exactly: the full hook trace and every deny/allow outcome across all six hooks is now byte-identical to the pre-change build on polars-frame graphs, pandas-frame graphs, and the declining shape. Explicit `engine='polars'` is unchanged and still carries the pre-existing hook gap; this only refuses to make that gap the default. Tests: the positive half gains the edges-only graph (`self._nodes is None` is inside the guard's condition, so it must actually work) and both spellings of AUTO. A new negative class pins each refusal -- pandas frames, mixed polars/pandas frames, explicit `engine='pandas'`, a denying postload policy still blocking, and the compile/load hooks firing exactly once on a shape the native engine declines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
Resolve CHANGELOG.md: rewrite the #1743 entry to describe behavior only (AUTO on all-polars-frame graphs routes to the native polars engine with legacy fallback on decline; explicit engines and policy-bearing calls unchanged; AUTO never selects polars-gpu; cuDF untouched) and drop stale perf numbers per the docs benchmark-numbers convention. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
…oracle parity - test_auto_engine_gfql_serves_polars_index_1767_cliff: polars frames + explicit gfql_index_all(engine='polars') + g.gfql() with NO engine now serves path=index on engine=polars per index_trace() -- the #1767 engine-mismatch cliff can no longer regress silently. - test_auto_engine_hop_residual_still_scans_1767: the documented scope boundary as executable documentation -- direct g.hop() with no engine still resolves AUTO to pandas and declines with decision_code engine_mismatch (hop/chain/index-build AUTO are follow-ups). - TestAutoEngineLazyFrames: LazyFrame edges+nodes with engine unset route native and return EAGER polars equal to explicit engine='polars'. - TestAutoEngineCudfUntouched: AUTO on cudf frames never enters the polars guard; legacy CUDF resolution unchanged (importorskip-gated). - TestAutoEnginePandasOracleParity: AUTO-on-polars-frames answers equal the pandas-engine oracle in value for a filter+traverse, an aggregate, and a WHERE-bearing shape (normalized frame comparison). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
…ly usable Owner-directed policy addition to #1743 (2026-08-02), superseding the "AUTO never selects polars-gpu" doctrine for the cuDF arm only: when every bound frame is cuDF AND a once-per-process probe confirms the cudf-polars GPU stack actually works (polars imports, cudf + cudf_polars installed, and a REAL GPU collect succeeds - a box with wheels installed but a broken driver/toolchain probes False), engine='auto' runs the query on the lazy polars engine's GPU target instead of the legacy CUDF path. cudf frames IN still mean cudf frames OUT: inputs cross cuDF->Arrow->polars in chain dispatch's _coerce_input_formats; results cross back polars->Arrow->cuDF in _route_result_frames_to_cudf. df_to_engine gained the polars->cuDF Arrow interchange (mirror of the existing cuDF->polars one) so neither direction takes the lossy pandas detour. Decline shape mirrors the polars arm: any NotImplementedError (engine decline, GPU-collect failure via lazy._gpu_raise, or the cudf-out boundary) falls back to the legacy CUDF path with identical values. Explicit engine= always wins; policy= bypasses (same postload/postchain hook-gap reason); guard order is all-polars -> all-cudf -> unchanged. The probe (lazy.polars_gpu_available) is an lru_cache(maxsize=1) process singleton registered EXEMPT in the GFQL cache registry per DEVELOP.md, with the coverage lock green. Tests, three tiers: - CPU-runnable anywhere: probe graceful-False legs + memoization pins; guard-order / explicit-engine / policy / mixed-frame pins through the _polars_gpu_probe and _auto_cudf_polars_gpu_route seams. - cudf-runnable: cudf-in/cudf-out TYPE pins on the legacy path (before and after semantics), NIE fallback value parity, and the REAL route body on CPU via the _AUTO_CUDF_ROUTE_ENGINE='polars' seam (guard -> Arrow coercion -> native engine -> Arrow cudf-out), including nullable int survival across both Arrow boundaries. - DGX-deferred (skipif probe False): real polars-gpu engagement receipt, value parity vs legacy, explicit polars-gpu still returns polars. Verification (this box, RTX 3080 Ti with broken GPU kernels - probe is genuinely False here): conformance 168 passed / 3 skipped (DGX tier); cache-coverage lock 9 passed; index suite CPU lane green with the 65 cudf-lane failures byte-identical to the pre-change baseline (local env, pre-existing); ruff clean; mypy clean (328 files); type-hygiene guard OK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
The test-polars py3.12 coverage cell tripped the lazy/__init__.py per-file
floor (89.91% < 90.00%): the new polars_gpu_available() probe added 7
statements whose legs never executed on a CPU lane. Root causes, each now
pinned CPU-runnably through the existing find_spec/_engine_for seams:
- test_probe_false_when_cudf_polars_missing silently re-tested the
cudf-missing leg on cudf-less boxes (real find_spec('cudf') is None there,
short-circuiting before the leg the test names); cudf is now stubbed
present so the cudf_polars check is genuinely reached.
- New test_probe_false_when_spec_lookup_raises: the import-block except
(broken packaging metadata) probes False, never raises.
- New test_probe_false_when_engine_builder_returns_none: the typing-honesty
eng-is-None guard declines instead of collecting on None.
- New test_probe_true_when_collect_succeeds: the success leg's REAL collect
and value check run on CPU by seaming _engine_for to 'in-memory' -- only
the GPU-ness of the engine object is stubbed, the collect is genuine.
Local gfql-polars audit: lazy/__init__.py 96.33% (105/109) vs floor 90.00%;
full lane 4122 passed + 68 lowering, audit exit 0. No source changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
… BY (cudf AUTO route regressions) DGX full-container run on this branch showed 6 test_row_pipeline_ops regressions vs master (A/B-confirmed): with the cudf-polars GPU probe genuinely True, the new AUTO cudf->polars-gpu route now serves the row-pipeline "when_available" cudf tests, and two polars row kernels were not value-equivalent to the legacy row pipeline: - where_rows_polars reached `pl.col(c) == GT(...)` for filter_dict PREDICATE values (gt/lt/...) and leaked a raw polars TypeError -- not NotImplementedError, so the route's decline-and-fall-back contract broke and the query errored instead of re-serving on the legacy cuDF path (test_row_pipeline_cudf_where_unwind_group_by). - order_by_polars plain-sorted keys the legacy pipeline orders with LIST semantics: real List columns (element-wise Cypher list-orderability incl. nullable-bool / empty-nested cases) and stringified-list text (legacy parses "[...]" columns, or REJECTS mixed ones); the lexicographic polars sort silently returned DIFFERENT VALUES (order_by host-bridge + stringified-list + 3 nested-map parity failures). Fix shape is decline-not-serve (always value-safe; owner's route design untouched -- explicit engines and policy still bypass, NIE still falls back to legacy identically): - where_rows_polars: ASTPredicate values defer (NIE); any other value pl.lit can't lower defers instead of leaking TypeError. (Native predicate lowering exists in predicates.filter_expr_by_dict_polars; wiring it in is a future upgrade, deliberately out of this fix.) - order_by_polars: new _order_keys_hold_list_like_values sniff -- schema-only check for nested/object key dtypes, plus a _GFQL_LIST_TEXT_RE scan of string-typed keys (same regex the legacy detection uses); any hit defers (NIE). Deliberately slightly broader than legacy engagement (list-text + nulls corner): the decline re-serves those on the legacy path with identical values. Tests: kernel-level decline pins (polars-only, CI-runnable) + two route-level CPU-seam tests reproducing the DGX shapes end-to-end (cudf in -> route -> decline -> legacy cudf out, legacy values). Local verification (CPU): test_row_pipeline_ops 268/268 under the forced route seam (was 6 failed) and 268/268 without; polars suites + conformance file green; full tests/compute/gfql fail-set byte-identical to branch head (92 pre-existing local-env failures, 0 flips, +4 new passing); bin/lint.sh exit 0; scoped mypy adds no new errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
…rop) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
# Conflicts: # CHANGELOG.md # graphistry/tests/compute/gfql/index/test_index.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Identical content to #1743 at its final head (whose PR object stopped receiving Actions events entirely — dispatch runs completed green but never attach to PR checks; pushes and reopen spawned nothing). Evidence carried over: owner-directed design (polars routing + cudf→polars-gpu preference with probe gating and cudf-out preservation); DGX correctness gate PASSED (444/444 in the official container incl. the real-GPU execution classes, after the decline-not-serve fix for two polars row-kernel gaps); full dispatched matrix on this exact SHA: 72 success / 2 skipped. Cell receipts and the #1767-cliff regression pin included. Owner authorized merge on the correctness gate.
🤖 Generated with Claude Code
https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr