perf(gfql): index AUTO preserves polars frames (rework of #1767, stacked on #1743)

[lmeyerov]

DO NOT MERGE BEFORE #1743

Stacked on #1743 (fix/gfql-auto-engine-polars-native) and contains its commits until it lands (plus a sync merge of current master to pick up #1841's usable/reason columns). Draft against master so CI runs; once #1743 merges, this rebases/retargets to a clean index-only diff. Do not land this first — the whole point of the rework is the ordering.

What this is: the retracted #1767, made safe by sequencing

The retracted #1767 made gfql_index_all()/create_index() under engine='auto' preserve resident polars frames (index in place, no pandas coerce-and-replace). It was retracted because query-side AUTO still routed polars-frame graphs to pandas, so the AUTO-built polars index met an AUTO(pandas) query and every hop declined to the scan floor — a measured 125–534x DEFAULT-path regression (250K/1M/4M edges, dgx-spark, receipts in the #1767 thread).

#1743 removes the mismatch at its root: g.gfql(...) with no engine on an all-polars-frame graph now routes to the native polars engine. With that underneath, the index-side preservation inverts from the cliff into the win:

	index built (AUTO)	frames	AUTO gfql resolves	index serves?
master today	pandas	polars coerced to pandas copies	pandas	yes (but frames were swapped)
old #1767 alone	polars	preserved	pandas	no — 125–534x cliff
#1743 + this	polars	preserved	polars	yes

The inversion pin

test_inversion_auto_index_auto_gfql_serves_polars_index: gfql_index_all() + g.gfql(<index-served query>) with no engine argument anywhere — the exact two-default-spellings scenario that regressed — must show path=index, engine=polars in index_trace(), answer in polars frames, and value-match BOTH explicit-engine spellings (polars/polars and pandas/pandas oracles).

The #1841 flip

show_indexes() under AUTO now reports the routed truth: the "polars index + AUTO query" combination that #1841 was built to call out as usable=False flips to usable=True (test_polars_index_auto_query_usable_the_1841_flip, plus the all-default spelling in test_auto_built_index_reports_usable_under_auto). Explicit mismatched-engine previews (show_indexes(engine='pandas') against a polars index) keep #1838's decline wording, pinned per kind.

Change surface (index-only; no routing — that is #1743's job, underneath)

• graphistry/compute/gfql/index/api.py: new resolve_index_engine() — AUTO + both frames eager polars → Engine.POLARS; wired into create_index, _is_resident_index_valid, show_indexes. Explicit engines untouched.
• graphistry/compute/gfql/index/wire.py: the DDL CREATE INDEX NODE_PROP reuse check uses the same resolution (otherwise IF-valid reuse would mismatch the build it just did).

Ported from old #1767: the _resolve_index_engine mechanism (now also covering show_indexes/wire, which the old diff predated — #1841/#1838 didn't exist then), the frame-identity preservation contract, and the whole negative-pin test suite (LazyFrame, mixed-frame, nodes-only, edges-only, pandas-stays-pandas, cudf-stays-cudf, explicit-engine-still-coerces, index-service spy with pandas-oracle parity).
Dropped from old #1767: its 276x standalone-perf framing (the thread showed the explicit-engine fast path was already reachable on master; the honest claim here is default-path consistency), and its CHANGELOG entry (rewritten as behavior-only with the landing-order dependency stated).

Gate deliberately narrower than #1743's query gate, self-consistent because create_index coerces the frames it indexes (later AUTO queries route with the post-build frames): LazyFrame graphs keep the legacy pandas build (an index cannot gather rows from a lazy plan); edges-only graphs too — materialize_nodes() does not yet produce polars nodes from polars edges, a pre-existing gap also reachable via explicit engine='polars' on such a graph (probed empirically; guessing polars there would crash the build).

Known interaction left to #1743's review: on cudf-frame graphs, #1743's polars-gpu arm can route an AUTO query away from a resident cudf index (GPU lanes only); index resolution here deliberately leaves cuDF source-native and takes no position on that arm.

Tests (CPU lane, this box: cudf/gpu params env-excluded — no GPU libs)

• index suite graphistry/tests/compute/gfql/index/: 229 passed (83 deselected cudf/gpu)
• test_index.py alone: 142 passed; new/changed: inversion pin, fix(gfql): show_indexes reports engine usability, not just fingerprint validity #1841 flip x2, DDL SHOW GFQL INDEXES flip, per-kind reason under explicit mismatch, + the ported TestIndexAutoPreservesPolarsFrames (11 tests)
• polars cypher conformance + cache coverage lock: 177 passed, 3 skipped
• bin/lint.sh clean · scoped mypy (api.py, wire.py) clean · type-hygiene guard OK (no growth)
• Pre-existing env failures only: 2 cudf tests fail on this GPU-less box with libnvrtc OSError (master's own test_cudf_index_pandas_query_not_usable fails identically here); they are GPU-lane tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr