perf(gfql): minimal-join fused plan for the two-star grouped count (q5/q6/q7) - #1846
Open
lmeyerov wants to merge 8 commits into
Open
perf(gfql): minimal-join fused plan for the two-star grouped count (q5/q6/q7)#1846lmeyerov wants to merge 8 commits into
lmeyerov wants to merge 8 commits into
Conversation
The fused polars two-star lane (graph-benchmark q5-q7 class: MATCH (p)-[r1]->(i), (p)-[r2]->(c) WHERE ... RETURN keys, count(p)) carried two semi-joins and an eager three-op tail that are provably redundant, and together were ~30% of the lane. Profiled one-shot on the 20k graph-benchmark q7 (local CPU, diagnosis-only): 3.5ms of the 4.2ms call was the fused helper, of which the single collect was 2.6ms and the eager group_by/sort tail another 0.4ms. Three algebraic removals, one plan: - LEFT arm shared-domain semi-join: a per-src count followed by an INNER join on the same key commutes with a src-restriction before the group_by -- out-of-domain groups are dropped by the join, per-key counts are untouched. The one consumer that needs the RESTRICTED counts (the empty-match all-left-counts==1 probe behind the openCypher n=0 row) rebuilds them WITH the semi-join on that branch only, so boundary semantics are byte-identical -- and pinned by a test whose out-of-domain person holds a count of 2 (fails against the unrestricted counts). - RIGHT arm second-leaf semi-join, whenever the group-property lookup join runs: the lookup is keyed UNIQUE on the same ids, so the INNER join keeps exactly the semi's row multiset while attaching the group columns. Without group properties the semi stays. - The grouped tail (group_by/sort/head/select) rides the SAME single collect when group keys exist -- same ops, same maintain_order, same null placement, inside the one plan instead of three eager ops that each pay their own engine dispatch. LIMIT 0 -- where head(0) empties a live match and the contract is the 0-row WITH-columns frame, not the 0x0 boundary frame -- keeps the eager tail, pinned. Local diagnosis-only effect (cold binding, warm compile, values byte-identical vs pandas oracle and the forced-decline eager twin): q7 20k 3.96ms -> 3.19ms and 100k 10.01ms -> 7.22ms; the class moves with it (q5 11.88 -> 8.14, q6 14.65 -> 9.20 at 100k). Local same-HW Kuzu anchors for q7: 5.65ms at 20k, 13.95ms at 100k. The authoritative numbers must come from the locked DGX rerun of the receipted matched q1-q9 lane. Tests: lane pin on the exact q7 anatomy (range filter + pushdown + one-sided toLower residual + two group keys + ORDER BY count DESC LIMIT 1) vs both twins; the boundary-probe discriminator; the grouped LIMIT 0 column contract; and a one-collect structural lock-in for the grouped tail. Suites at master-identical fail-sets (gfql subtree 93 pre-existing env fails on both). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
…ormance Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
Master-merge composition put gfql_fast_paths.py one cast over the per-file hygiene ratchet; same declaration conversion the combined candidate used. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
Owner review on the PR: cross-database verdicts, millisecond tables, and lane paths belong in pyg-bench results and the docs board, not the public repo's changelog or code comments. The entry now describes the plan change and its value-identity contract, with pointers to the receipted lanes; code comments keep the mechanism rationale without inline measurements. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
…nimal-join fused plan Every #1846 algebraic removal now has a SERVED-side and a KEPT-side test: - (b) right-arm second-leaf semi: plan-shape pin that it IS emitted without group properties (kept) and NOT emitted when the unique-keyed lookup join runs (served), observed by instrumenting polars inside the fused helper. - (a) left-arm shared-domain semi: populated-match out-of-domain multiplicity ungrouped (fixture self-check that restricted vs unrestricted counts differ; value stays correct via the final inner join) plus a structural pin that the n=0 boundary probe REBUILDS the semi (two src-side semis, hot plan + probe). - (c) grouped tail: no eager group_by when the tail rides the single collect (served) vs exactly one eager group_by on grouped LIMIT 0 (kept); ungrouped count+LIMIT shapes never reach the fused lane (unchanged behavior pin). - differential: 8 seeded graphs x (group props yes/no) x (empty/populated) x (LIMIT 0/None/positive), fused vs forced-decline eager twin (full polars frame+schema equality) vs pandas oracle (column order + records), with a deterministic core keeping every seed's populated combos non-vacuous and the dropped shared-domain semi observable. Residual+lowering suites: 1474 passed / 166 skipped (baseline 1458/166, zero failures both sides). ruff clean; scoped mypy adds no errors in the test file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A minimal-join fused plan for the two-star grouped count (graph-benchmark q5–q7 class:
MATCH (p)-[r1]->(i), (p)-[r2]->(c) WHERE ... RETURN keys, count(p)). The fused polars lane carried two semi-joins and an eager three-op tail that are provably redundant, together ~30% of the lane: profiled one-shot on the 20k graph-benchmark q7 (local CPU, diagnosis-only), 3.5 ms of the 4.2 ms call was the fused helper, of which the single collect was 2.6 ms and the eager group_by/sort tail another 0.4 ms.Why it is safe
Three algebraic removals, one plan:
Tests: lane pin on the exact q7 anatomy (range filter + pushdown + one-sided toLower residual + two group keys + ORDER BY count DESC LIMIT 1) vs both twins; the boundary-probe discriminator; the grouped LIMIT 0 column contract; a one-collect structural lock-in for the grouped tail. Values byte-identical vs the pandas oracle and the forced-decline eager twin.
Measured cells (attributed: q5/q6/q7, both scales)
Measurements come from the COMBINED candidate build
938f22851e68918799da2bee49c88db2e2c103ee(masterf875724ce+ this branch +perf/gfql-q8-class-fix+perf/gfql-singlehop-subsumed-semijoin+perf/gfql-q8-bincount-r2), with per-fix attribution by disjoint cell sets: this PR owns q5/q6/q7, the single-hop PR owns q1/q3/q4, the q8 branches own q8.Comparator quotes, baseline boards (
results/graphbench-board-{20k,100k-v2}-20260802) vs candidate lanes (results/graphbench-board-{20k,100k}-cand-20260803, graphistry/pyg-bench#170):TIEWIN 1.49xTIEWIN 1.46xTIEWIN 1.18x (WEAK: slot ranges overlap)WIN 1.20x (WEAK: slot ranges overlap)WIN 1.29x(no overlap)WIN 1.55xWIN 1.96xWIN 1.62xWIN 1.89xUntouched cells did not move: cells outside this PR's attribution set changed only where attributed to the sibling branches above; the two unattributed cells (q2, q9) kept their WIN verdicts at both scales (q9 within noise; q2's improvement rides the fused single-hop lane of the semi-join subsumption branch).
Receipts
results/graphbench-board-20k-cand-20260803/andresults/graphbench-board-100k-cand-20260803/(compare.txt, SOURCE_COMMITS, lock-receipt, 1 Hz load receipts, SHA256SUMS).938f22851e68918799da2bee49c88db2e2c103ee, pyg-benchfae1e975.Do not merge without owner review.
🤖 Generated with Claude Code
https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr