perf(gfql): per-type column-stat facts — make the dense kernel reachable on typed graphs

[lmeyerov]

Follow-on to #1854, which landed as no-harm / no-help on the multi-type GraphBench board. That was structural, not a tuning miss, and this fixes the cause.

Why whole-frame facts cannot work on a typed graph

• A typed pattern's domain is a strict subset of the node frame, so no dense-interval hint is derivable from a fact describing the whole frame.
• The edge endpoint interval spans every label, so containment in one label's interval can never be proved.

So on the board, _facts_prove_bounds returned False every time and the O(E) endpoint scan always ran.

---

1. Does this replace whole-frame stats, or build on them?

Purely additive. Whole-frame facts are still built and still used. Nothing about #1854's behavior changes.

Facts gain two extra key components, so the registry key goes from (role, column) to (role, column, type_column, type_value). The whole-frame fact is simply the (…, None, None) entry — same fact, same build, same consult, now just one point in a larger keyspace.

At build time both are produced:

whole-frame : ('nodes','id',None,None)  ('edges','s',None,None)  ('edges','d',None,None)
partitions  : ('nodes','id','kind','P')  ('nodes','id','kind','C')  ('edges','s','rel','F')  …

At consult time they are used for different query shapes, with whole-frame as the fallback:

query shape	what resolves
untyped (MATCH (a)-[]->(b))	whole-frame fact — exactly as on master
typed ((a:Person), {kind:'P'})	partition fact; falls back to whole-frame if none exists
anything else	no fact → the O(E) scan, as before

Verified on a mixed graph: the untyped query still derives its whole-frame hint (0, 5) while the four partition facts sit alongside unused. A miss at any step costs the scan, never an answer.

2. How the type-tag field is chosen

Three sources, in precedence order. None of them guess.

a) Explicit, by name — wins for its role.
gfql_index_col_stats(node_type_column='kind', edge_type_column='rel'). Asked for by name, so an unusable request raises rather than skipping silently.

b) A declared schema — used automatically, because it is a declaration.
bind(schema=GraphSchema(...)) already names the types: NodeType.labels maps to GFQL's label__<Label> convention and EdgeType.name to the relationship type. Derived candidates are restricted to columns the frame actually carries and skip when absent or unusable — a schema is a contract for the whole graph, and a frame legitimately carries only part of it.

c) Nothing else. No schema bound and no param ⇒ nothing is inferred and no extra work is done, pinned by an explicit negative test. In particular there is no column-name sniffing (type/category/kind) and no cardinality probing: that would make performance depend on whether someone happened to name a column kind, cost a distinct-count per candidate on every index_all(), and be hard to retract once shipped. If we want it later, node_type_column='auto' is the opt-in shape.

Whichever source supplies it, the consult side keys off what the query itself says — the single scalar equality a typed pattern lowers to — so a fact built on a column no query names is wasted build time, never a wrong answer.

Two conventions are both supported, because both occur:

• (a:Person) → the boolean {'label__Person': True}
• -[:FOLLOWS]-> → the string {'type': 'FOLLOWS'}
• explicit property maps → {'kind': 'P'}

Known conflict, handled defensively: schema.py declares edge types as label__<Name> booleans (EdgeType.columns, line 338) while the Cypher lowering emits type == '<Name>' (lowering.py:3048). Node labels agree; edges do not. Until that is reconciled at the source I offer both edge candidates and keep whichever the frame carries. Worth fixing, but not by picking a winner inside a perf PR.

3. Observed benefit

GB10 reference (pyg-bench #181 — quiet gate, perf lock, board image, build 3e162e427). Board-shaped graph: Person ids dense in [0, 400k), City beyond; 2.4M FOLLOWS (P→P) + 400k LIVES_IN (P→C); the board's q8 two-hop count restricted to Person/FOLLOWS.

rep	whole-frame	per-type	speedup
1	14.49 ms (proved=False, hint=None)	11.99 ms (proved=True, hint=(0, 399999))	1.21x
2	14.34 ms (proved=False, hint=None)	12.11 ms (proved=True, hint=(0, 399999))	1.18x

count=14396131 in every arm; the micro asserts value parity. A dev-box run read 1.38x and is superseded — the GB10 numbers are the reference.

Scope of that claim, stated plainly: this is a motivation lock on one shape, not a board result. The multi-type board lane is owed once this lands, and it needs pyg-bench #182's --index all+types to engage at all (gfql_index_all() passes no type columns, so a lane without it would measure exactly zero — a harness gap, not a null result).

Correctness notes

The dense kernel skips the domain semi-join when the partition's ids provably fill [min,max]. That check is n_unique == max - min + 1 — a claim about the id set — so duplicate rows cannot fake density, and overlapping labels do not shrink a label's id set. Multi-label (a:Person:Employee) lowers to two keys and the one-equality rule refuses it, so an intersection can never ride a single-label fact. Fuzzed over 120 randomized multi-label / duplicate-row graphs (40 with the hint engaged): zero divergences from the no-facts oracle.

Two bugs surfaced during that audit and are fixed here: the gate rejected bools, so idiomatic Cypher labels could never engage (every fixture used property maps, so nothing caught it); and list-valued type columns diverged across engines (pandas raised unhashable type: 'list', polars silently built unusable list-keyed facts) — both now decline by explicit precondition.

Pins

Three-arm gate (no facts / whole-frame / per-type — hint and proof appear only in the last, answer identical in all three) across pandas, polars and cuDF; label-syntax engagement per engine; the partition-key admission matrix; extra-predicate hint refusal; list-column decline plus its positive twin; schema-declared build and engagement; the no-schema zero-change twin; absent-label skip; unusable-request raise.

Verified on real GPU (dgx-spark GB10 via dgx-guard/safe_run.sh, so --gpus all is in force): 184 passed, 0 failed, 0 skipped on cudf 26.02.01 / cupy 13.6.0. Zero skips is the load-bearing part — the cuDF arms genuinely ran rather than opting out, which a local run cannot do (no libnvrtc, so every cuDF case fails identically on master).

Two things in this PR are cuDF-only code paths that a dev box physically cannot execute, so they were exercised directly on device rather than inferred:

to_arrow arm  -> ['P', 'C', 'P']          # _column_to_pylist's cudf branch, device memory
int arm       -> [3, 1, 2]
kind int/str/bool/list -> 'i' / 'O' / 'b' / 'O'   # _dtype_kind over cudf exotic dtypes
list type col   -> DECLINED                # end to end on device
scalar type col -> [('C', 2, 3), ('P', 0, 1)]     # identical to pandas/polars

That last pair matters: the _dtype_kind helper exists precisely because pandas' is_integer_dtype raises TypeError on cudf ListDtype, and this confirms the replacement declines cleanly on real device data while scalar keys still build identically across all three engines.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi

[lmeyerov]

Correction to the PR body: the board cannot engage this as written

I claimed above that a board-visible win is "plausible but unmeasured." That understates the problem, and I want it on the record before review.

Per-type facts are opt-in by column name only. gfql_index_all() calls gfql_index_col_stats(g, engine=engine) with no type columns, so it builds whole-frame facts and nothing else. The GraphBench harness indexes via --index all → gfql_index_all(). So a board lane on this build would measure exactly zero effect — not "a small effect we failed to detect," but a code path that is never reached.

The board data is otherwise a perfect fit: its frames carry node_type (Person/City/...) and rel (FOLLOWS/LIVES_IN/...), which is precisely the shape the partition facts key on.

This does not affect the correctness of the change or the receipted micro (pyg-bench #181 requests the type columns explicitly, which is why it engages). It affects only the claim about the board.

Two follow-ups, kept separate on purpose:

1. Measurable now, no library change: teach the benchmark harness to request the type columns explicitly, so the board hypothesis becomes answerable. That is a pyg-bench change and I will make it there.
2. Owner call, deliberately NOT done here: should gfql_index_all() discover type columns and build per-type facts automatically? There is no type-column binding on Plottable to key off, so any discovery would be a cardinality/dtype heuristic — i.e. a change to default indexing behavior with a scan cost attached. Given the standing hold on default-routing policy changes (perf(gfql): route engine=auto to native polars for polars-frame graphs #1743), I am not making that call inside a perf PR. If the answer is yes, it wants its own PR and its own receipts.

So: merge this for the mechanism and the micro, and treat "does the board move" as open until (1) lands and a lane runs.

[lmeyerov]

Membership hazards audited, two engine-divergence bugs fixed

Reviewer raised that "some optimization may short circuit based on membership." Audited; here is what I found and what I changed.

The short-circuit is sound, and here is why

The dense kernel skips the domain semi-join when _dense_interval_from_fact proves the partition's ids fill [min, max]. That check is n_unique == max - min + 1 — a statement about the id set, so duplicate rows inflate row count but cannot fake density. Density gives the biconditional id ∈ [lo,hi] ⟺ id ∈ partition, which is exactly what licenses treating the semi-join as the identity.

Overlapping labels do not weaken it: a node being both Person and Employee does not remove it from the Person id set, and facts are keyed per column, so two label columns are never combined.

Verified empirically: 120 randomized trials with multi-label nodes and duplicate node rows (40 with the hint engaged) — zero divergences from the no-facts oracle. Plus hand cases for overlapping labels, duplicate rows in one label, and the same id in both label groups.

Multi-label patterns cannot ride a single-label fact

MATCH (a:Person:Employee) lowers to {'label__Person': True, 'label__Employee': True} — two keys — and the one-equality rule returns None, so it falls back to the scan. Same for (a:Person {age:30}) → {'age': 30, 'label__Person': True}. The intersection case is refused by construction, not by luck.

Two real bugs found and fixed

1. The boolean label form was silently excluded. (a:Person) lowers to the BOOLEAN {'label__Person': True}; the gate rejected bools, so the node-side hint could never resolve for idiomatic Cypher labels. The whole typed path was confined to explicit property maps ({kind: 'P'}) — which is what every fixture and the board happened to use, so nothing caught it. Fixed, with end-to-end label-syntax pins per engine.

2. List-valued type columns diverged across engines. resolve_filter_column rewrites an absent label__X into a membership test on a labels LIST column, so a list-valued type column is reachable in practice. pandas raised unhashable type: 'list' from the groupby; polars silently built facts keyed on lists. Neither is acceptable — a query never yields a list-valued partition key, so those facts could never be consulted. Both engines now decline by explicit precondition, pinned both ways.

Still open, for a follow-up — the two conventions disagree

schema.py declares edge types as label__<Name> booleans (EdgeType.columns, line 338), but the Cypher lowering emits {'type': '<Name>'} — a string column literally named type (lowering.py:3048). Node labels agree (label__<L> on both sides); edges do not. Facts built from a declared schema would therefore key on label__FOLLOWS while queries ask for type == 'FOLLOWS' — a permanent miss (no wrong answer, just no benefit). Worth reconciling before wiring schema-driven fact building, which is why I have not wired it in this PR.

[lmeyerov]

Schema integration: type partitions now come from the declared schema

Reviewer asked to research the existing node/edge type surfaces and integrate rather than bolt on. Done — and it resolves the discovery question without any column-name guessing.

What exists on master

• No node_type / edge_type parameter on bind().
• bind(schema=GraphSchema(...)) → _gfql_schema (PlotterBase.py:1643) is settable. NodeType.labels explicitly "maps to GFQL's existing label__<Label> convention" and EdgeType.name / EdgeTopology.relationship_type name relationship types. Its only consumer today is opt-in Arrow-upload validation.
• The g.schema getter PRs (feat(gfql): infer typed schemas from graph data #1636 / feat(gfql): add Plottable schema accessor #1637 / feat(schema): add pretty print helpers #1639) are still open and untouched since 2026-05-25, so there is no read accessor — only getattr(g, "_gfql_schema", None).
• The Cypher lowering hardcodes the convention: (a:Person) → {'label__Person': True} (lowering.py:3036), -[:FOLLOWS]-> → {'type': 'FOLLOWS'} (lowering.py:3048).

What this PR now does

gfql_index_col_stats reads a bound schema and builds the partitions it declares — label__<Label> per node label, and the edge relationship types — restricted to columns the frame actually carries.

This is a declaration, not a heuristic: the user said these are the types. It deliberately does not sniff column names by cardinality, which would make performance depend on whether someone happened to name a column kind, cost a distinct-count probe per candidate on every index_all(), and be hard to retract once shipped.

Contract, kept consistent with what was already there:

• Explicit node_type_column= / edge_type_column= still raise when unusable (asked for by name) and win for their role.
• Schema-derived candidates skip when absent or unusable — a schema is a contract for the whole graph and a given frame legitimately carries only part of it.
• No schema bound → nothing inferred, no extra build. Callers who never opted in see zero change; pinned by an explicit negative test.

One unresolved conflict, handled defensively

schema.py declares edge types as label__<Name> booleans (EdgeType.columns, line 338) while the lowering emits type == '<Name>'. Node labels agree across both; edges do not.

Until that is reconciled I offer both edge candidates and keep whichever the frame carries. A fact on a column no query names costs build time and never correctness, so covering both is the safe direction — but the conflict is worth fixing at the source, and I would rather flag it than pick a winner inside a perf PR.

Remaining question for you

Should gfql_index_all() go further and infer type columns when no schema is declared? I did not do it: with no type-column binding to key off, that is necessarily a cardinality/dtype heuristic — a default-indexing behavior change with an involuntary scan cost — which is the same class of change held back in #1743. If you want it, node_type_column='auto' as explicit opt-in is the shape I would suggest.

[lmeyerov]

Real-GPU verification on GB10 (not just CI)

Earlier I said the cuDF arms "exercise in CI's GPU lane." That was an assumption; it is now measured. Run on dgx-spark (NVIDIA GB10) through dgx-guard/safe_run.sh, so --gpus all, the RMM cap, the host watchdog and a hard timeout were all in force — a run without --gpus all fabricates failures rather than skipping, which is worse than not running it at all.

Environment, printed by the run itself rather than assumed:

cudf 26.02.01 cupy 13.6.0
cupy compute: 6

That second line is the NVRTC JIT probe that fails on my dev box, so this is genuinely the path local runs cannot reach.

Result: 183 passed, 0 failed, 0 skipped across test_lowering.py + the index suite for the typed-facts selection. Zero skips is the load-bearing part — it means the cuDF arms executed rather than silently opting out.

Confirmed by name, every parametrization:

test_t6_per_type_facts_prove_typed_bounds_whole_frame_cannot[none-False-None-cudf]        PASSED
test_t6_per_type_facts_prove_typed_bounds_whole_frame_cannot[whole_frame-False-None-cudf] PASSED
test_t6_per_type_facts_prove_typed_bounds_whole_frame_cannot[per_type-True-expect_hint2-cudf] PASSED
test_t6_per_type_facts_engage_for_cypher_label_syntax[cudf]                               PASSED
test_t6_per_type_declines_list_valued_type_column[pandas|polars]                          PASSED
test_t6_per_type_scalar_type_column_still_builds[pandas|polars]                           PASSED
test_t6_declared_schema_builds_partition_facts_and_engages                                PASSED
test_t6_no_schema_builds_no_partition_facts                                               PASSED
test_t6_partition_key_admits_the_boolean_label_form                                       PASSED
test_t6_per_type_extra_predicate_refuses_the_hint                                         PASSED

So the boolean-label__X fix, the list-column decline, and the schema-derived build are all verified on real GPU hardware, not inferred from a green matrix.

Build 6fba477, image graphistry/test-rapids-official:26.02-gfql-polars. Runner kept at ~/gbaudit/gputest.sh on the box.

[lmeyerov]

extern ? do we ahve such constants somewher ealready?