Skip to content

perf(gfql): minimal-join fused plan for the two-star grouped count (q5/q6/q7) - #1846

Open
lmeyerov wants to merge 8 commits into
masterfrom
perf/gfql-q7-two-star-minimal-join
Open

perf(gfql): minimal-join fused plan for the two-star grouped count (q5/q6/q7)#1846
lmeyerov wants to merge 8 commits into
masterfrom
perf/gfql-q7-two-star-minimal-join

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What

A minimal-join fused plan for the two-star grouped count (graph-benchmark q5–q7 class: MATCH (p)-[r1]->(i), (p)-[r2]->(c) WHERE ... RETURN keys, count(p)). The fused polars lane carried two semi-joins and an eager three-op tail that are provably redundant, together ~30% of the lane: profiled one-shot on the 20k graph-benchmark q7 (local CPU, diagnosis-only), 3.5 ms of the 4.2 ms call was the fused helper, of which the single collect was 2.6 ms and the eager group_by/sort tail another 0.4 ms.

Why it is safe

Three algebraic removals, one plan:

  • LEFT-arm shared-domain semi-join: a per-src count followed by an INNER join on the same key commutes with a src-restriction before the group_by — out-of-domain groups are dropped by the join, per-key counts are untouched. The one consumer that needs the RESTRICTED counts (the empty-match all-left-counts==1 probe behind the openCypher n=0 row) rebuilds them WITH the semi-join on that branch only — boundary semantics byte-identical, pinned by a test whose out-of-domain person holds a count of 2 (fails against the unrestricted counts).
  • RIGHT-arm second-leaf semi-join, whenever the group-property lookup join runs: the lookup is keyed UNIQUE on the same ids, so the INNER join keeps exactly the semi's row multiset while attaching the group columns. Without group properties the semi stays.
  • The grouped tail (group_by/sort/head/select) rides the SAME single collect when group keys exist — same ops, same maintain_order, same null placement, inside the one plan instead of three eager ops each paying engine dispatch. LIMIT 0 keeps the eager tail (the contract there is the 0-row WITH-columns frame), pinned.

Tests: lane pin on the exact q7 anatomy (range filter + pushdown + one-sided toLower residual + two group keys + ORDER BY count DESC LIMIT 1) vs both twins; the boundary-probe discriminator; the grouped LIMIT 0 column contract; a one-collect structural lock-in for the grouped tail. Values byte-identical vs the pandas oracle and the forced-decline eager twin.

Measured cells (attributed: q5/q6/q7, both scales)

Measurements come from the COMBINED candidate build 938f22851e68918799da2bee49c88db2e2c103ee (master f875724ce + this branch + perf/gfql-q8-class-fix + perf/gfql-singlehop-subsumed-semijoin + perf/gfql-q8-bincount-r2), with per-fix attribution by disjoint cell sets: this PR owns q5/q6/q7, the single-hop PR owns q1/q3/q4, the q8 branches own q8.

Comparator quotes, baseline boards (results/graphbench-board-{20k,100k-v2}-20260802) vs candidate lanes (results/graphbench-board-{20k,100k}-cand-20260803, graphistry/pyg-bench#170):

Cell Baseline (master pin) Candidate (this build)
q7 @ 20k polars 5.68 ms — TIE polars 3.47 ms — WIN 1.49x
q7 @ 100k polars 9.88 ms — TIE polars 6.69 ms — WIN 1.46x
q5 @ 100k polars 13.68 ms — TIE polars 11.35 ms — WIN 1.18x (WEAK: slot ranges overlap)
q5 @ 20k polars 4.50 ms — WIN 1.20x (WEAK: slot ranges overlap) polars 4.05 ms — WIN 1.29x (no overlap)
q6 @ 20k polars 5.73 ms — WIN 1.55x polars 4.58 ms — WIN 1.96x
q6 @ 100k polars 14.84 ms — WIN 1.62x polars 11.10 ms — WIN 1.89x

Untouched cells did not move: cells outside this PR's attribution set changed only where attributed to the sibling branches above; the two unattributed cells (q2, q9) kept their WIN verdicts at both scales (q9 within noise; q2's improvement rides the fused single-hop lane of the semi-join subsumption branch).

Receipts

  • Lanes: graphistry/pyg-bench#170 — results/graphbench-board-20k-cand-20260803/ and results/graphbench-board-100k-cand-20260803/ (compare.txt, SOURCE_COMMITS, lock-receipt, 1 Hz load receipts, SHA256SUMS).
  • Build SHA in both lanes' SOURCE_COMMITS: pygraphistry 938f22851e68918799da2bee49c88db2e2c103ee, pyg-bench fae1e975.
  • VALIDITY: 20k "VALID: no external process in any over-limit capture (0 receipted benchmark-own spike seconds)"; 100k "VALID: … (106 receipted benchmark-own spike seconds)".
  • Protocol: position-balanced K G I I G K K G I I G K slots, RUNS=51/WARMUP=5, per-slot medians, tie band 10%, weak-flag on overlapping slot ranges; verdicts published only where GFQL row sets match Kuzu.

Do not merge without owner review.

🤖 Generated with Claude Code

https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr

lmeyerov and others added 2 commits August 2, 2026 16:39
The fused polars two-star lane (graph-benchmark q5-q7 class: MATCH
(p)-[r1]->(i), (p)-[r2]->(c) WHERE ... RETURN keys, count(p)) carried two
semi-joins and an eager three-op tail that are provably redundant, and
together were ~30% of the lane. Profiled one-shot on the 20k
graph-benchmark q7 (local CPU, diagnosis-only): 3.5ms of the 4.2ms call
was the fused helper, of which the single collect was 2.6ms and the eager
group_by/sort tail another 0.4ms.

Three algebraic removals, one plan:

- LEFT arm shared-domain semi-join: a per-src count followed by an INNER
  join on the same key commutes with a src-restriction before the
  group_by -- out-of-domain groups are dropped by the join, per-key counts
  are untouched. The one consumer that needs the RESTRICTED counts (the
  empty-match all-left-counts==1 probe behind the openCypher n=0 row)
  rebuilds them WITH the semi-join on that branch only, so boundary
  semantics are byte-identical -- and pinned by a test whose out-of-domain
  person holds a count of 2 (fails against the unrestricted counts).
- RIGHT arm second-leaf semi-join, whenever the group-property lookup
  join runs: the lookup is keyed UNIQUE on the same ids, so the INNER
  join keeps exactly the semi's row multiset while attaching the group
  columns. Without group properties the semi stays.
- The grouped tail (group_by/sort/head/select) rides the SAME single
  collect when group keys exist -- same ops, same maintain_order, same
  null placement, inside the one plan instead of three eager ops that
  each pay their own engine dispatch. LIMIT 0 -- where head(0) empties a
  live match and the contract is the 0-row WITH-columns frame, not the
  0x0 boundary frame -- keeps the eager tail, pinned.

Local diagnosis-only effect (cold binding, warm compile, values
byte-identical vs pandas oracle and the forced-decline eager twin):
q7 20k 3.96ms -> 3.19ms and 100k 10.01ms -> 7.22ms; the class moves with
it (q5 11.88 -> 8.14, q6 14.65 -> 9.20 at 100k). Local same-HW Kuzu
anchors for q7: 5.65ms at 20k, 13.95ms at 100k. The authoritative numbers
must come from the locked DGX rerun of the receipted matched q1-q9 lane.

Tests: lane pin on the exact q7 anatomy (range filter + pushdown +
one-sided toLower residual + two group keys + ORDER BY count DESC LIMIT 1)
vs both twins; the boundary-probe discriminator; the grouped LIMIT 0
column contract; and a one-collect structural lock-in for the grouped
tail. Suites at master-identical fail-sets (gfql subtree 93 pre-existing
env fails on both).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
lmeyerov and others added 6 commits August 3, 2026 04:37
Master-merge composition put gfql_fast_paths.py one cast over the per-file
hygiene ratchet; same declaration conversion the combined candidate used.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
Owner review on the PR: cross-database verdicts, millisecond tables, and lane
paths belong in pyg-bench results and the docs board, not the public repo's
changelog or code comments. The entry now describes the plan change and its
value-identity contract, with pointers to the receipted lanes; code comments
keep the mechanism rationale without inline measurements.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
…nimal-join fused plan

Every #1846 algebraic removal now has a SERVED-side and a KEPT-side test:

- (b) right-arm second-leaf semi: plan-shape pin that it IS emitted without
  group properties (kept) and NOT emitted when the unique-keyed lookup join
  runs (served), observed by instrumenting polars inside the fused helper.
- (a) left-arm shared-domain semi: populated-match out-of-domain multiplicity
  ungrouped (fixture self-check that restricted vs unrestricted counts differ;
  value stays correct via the final inner join) plus a structural pin that the
  n=0 boundary probe REBUILDS the semi (two src-side semis, hot plan + probe).
- (c) grouped tail: no eager group_by when the tail rides the single collect
  (served) vs exactly one eager group_by on grouped LIMIT 0 (kept); ungrouped
  count+LIMIT shapes never reach the fused lane (unchanged behavior pin).
- differential: 8 seeded graphs x (group props yes/no) x (empty/populated) x
  (LIMIT 0/None/positive), fused vs forced-decline eager twin (full polars
  frame+schema equality) vs pandas oracle (column order + records), with a
  deterministic core keeping every seed's populated combos non-vacuous and the
  dropped shared-domain semi observable.

Residual+lowering suites: 1474 passed / 166 skipped (baseline 1458/166, zero
failures both sides). ruff clean; scoped mypy adds no errors in the test file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant