[prrao87]

Why

Grouping by a node (WITH forum, collect(friend) AS friends) after a hop that fans out carries every column of the node through the grouping, once per input row. #101 fixed this for in-memory node tables: it groups by the node's id and adds the other columns after the aggregate with a NodeLookup. Lance node tables in lazy mode kept the plain grouping, because turning the lookup on read the columns from disk a second time and made IC1 11% slower. This PR enables the lookup for Lance node tables only where it replaces a read instead of adding one, which resolves #99.

What changed

• Lance node tables get the lookup when a hop fetched the columns. If a CSR hop in the grouping's input fetches the grouped node's columns, and nothing between the hop and the grouping reads them (a projection passing them through doesn't count; a WHERE or DISTINCT does), the query groups by the id and looks the columns up afterwards. Pruning then removes the columns from the hop, so they are read once per node instead of once per edge. Columns read by a scan, as in IC1, or used before the grouping, as in IC10, keep the plain grouping.
• The lookup reads a Lance table once when it covers a large share of it. Past 1/8 of the table's rows, it reads its columns in one sequential pass and gathers from memory, with the same threshold the hop's fused target lookup uses. Below that it fetches by row id.
• Lookups are narrowed to what the projection above them uses. Column pruning worked from the set of names used anywhere in the plan, so a lookup whose columns were projected away was kept when the same node's columns were used elsewhere. In IC5, the lookup under the OPTIONAL MATCH join's left side fetched title for 77K forums and threw it away; in memory-mode IC14 eight of nine lookups were like this. Both modes benefit.
• User impact: none beyond speed. Results are unchanged and explain() shows NodeLookupExec in more lazy plans. The CSR guide's "Grouping by a node" section and the internals pages describe when Lance tables get the lookup.

IC5 in lazy mode, before and after (the hasMember hop under the grouping):

main:   CsrExpandExec: key=friend__id, edges=1611869, take=forum[forum__id*, forum__title, forum__creationdate], ...
branch: NodeLookupExec: key=forum__id, take=forum[forum__title, forum__creationdate]
          AggregateExec: ... gby=[forum__id@0 as forum__id], aggr=[array_agg(friend__id) ...]
            ...
              CsrExpandExec: key=friend__id, edges=1611869, take=forum[forum__id*], ...

Variants tried first

Variant	Lazy IC5	Lazy IC7	Lazy IC10	Lazy ldbc query5
Lookup whenever a hop fetches the columns (5 alternating processes)	-4.8%	-6.6%	+4.4%
Plus: skip columns read before the grouping (this PR, full run)	-8.8%	-6.3%	+1.5%
Hop's read-once check moved into a shared helper				+2.6%

The first version also looked up IC10's friend and city, whose columns a WHERE and a WITH DISTINCT read before the grouping, so the hops kept fetching them and the lookups read them again. IC10's +4.4% held in 5 alternating processes per build (ranges 29.0 to 29.8 ms on main, 30.2 to 32.0 ms on the branch). Moving the hop's per-chunk read-once check into a function shared with the lookup cost lazy ldbc query5 2.6% against a control build identical but for that change (+0.5%), so the hop keeps its inline check and only the lookup uses the shared one.

Validation

• cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace (705 passed), make test (1718 passed), make lint, zensical build --strict and typos pass locally. docs/check_snippets.py passes for the pages this PR changes; cypher.md:596 and sql.md:122 fail identically on main. CI is pending.
• New tests: the lookup over 20K groups from Lance with deleted rows (read-once path) and over about 200 groups (row-id path); Lance groupings that keep the plain plan when the node comes from a scan, or when a WHERE or DISTINCT reads its columns first; and a lookup projected away under an OPTIONAL MATCH join being dropped. Each compares against the join path.
• Every query's physical plan in both suites and modes was diffed against main. Plans changed only in lazy IC1, IC5, IC7 and IC12 and memory IC1, IC3, IC4, IC5 and IC14 (plus a reordered AND in memory IC3's filter, which varies run to run).

Benchmarks

Apple M5 (10 cores), 24 GB, macOS 26.6.2. Release wheels built from main at 8fa7a80 and from this branch, loaded with uv run --with per CONTRIBUTING.md; main and branch alternated twice per mode. Benchmark repositories: graph-benchmark-ldbc 3afadb9, graph-benchmark-ldbc-complex fc7b44c. Every result check passed. The 1-minute load average stayed between 2.1 and 4.0 during the runs with only the desktop app open.

Suite	Mode	Geomean branch/main	Median main run gap
graph-benchmark-ldbc-complex	lazy	0.988	0.5%
graph-benchmark-ldbc-complex	memory	1.000	1.6%
graph-benchmark-ldbc	lazy	1.001	0.6%
graph-benchmark-ldbc	memory	1.017	2.5%

In lazy mode: IC5 111.0 to 101.2 ms (-8.8%) and IC7 14.4 to 13.5 ms (-6.3%), both beyond the run gaps and seen in all four rounds of this work. IC1, the query that blocked #101 in lazy mode, gets one small lookup and is unchanged (-0.1%).

graph-benchmark-ldbc in memory mode is 1 to 2% slower on nearly every query, in this run and the three before it. None of its plans changed, the code those plans run is the same as on main, and the same queries in lazy mode, which plan the same way, show no shift (1.001). So this is most likely build-to-build variance, but I haven't confirmed that with a second build of main.

Every query, both suites and modes (mean ms)

graph-benchmark-ldbc: memory

Query	main (ms)	branch (ms)	change	main run gap	branch run gap
query1	0.62	0.62	+1.2%	1.0%	0.2%
query2	0.54	0.55	+3.1%	3.8%	1.2%
query3	0.63	0.64	+2.3%	3.0%	0.4%
query4	0.50	0.51	+2.3%	2.9%	2.0%
query5	1.05	1.07	+1.7%	0.8%	2.5%
query6	0.43	0.44	+2.5%	3.0%	1.6%
query7	1.02	1.05	+2.6%	2.5%	0.6%
query8	0.63	0.64	+1.6%	2.5%	0.1%
query9	0.80	0.82	+1.5%	2.9%	1.5%
query10	0.71	0.73	+2.5%	2.4%	0.2%
query11	1.50	1.53	+2.2%	2.8%	0.5%
query12	0.75	0.76	+1.5%	2.6%	0.4%
query13	0.79	0.81	+2.2%	4.3%	0.2%
query14	0.52	0.53	+1.4%	3.7%	1.0%
query15	0.55	0.56	+1.8%	3.2%	0.0%
query16	0.66	0.68	+2.2%	4.1%	1.5%
query17	0.90	0.92	+2.0%	0.3%	0.2%
query18	0.69	0.71	+2.5%	3.9%	1.1%
query19	1.01	1.03	+1.4%	0.7%	1.5%
query20	3.47	3.45	-0.5%	0.7%	0.6%
query21	0.77	0.77	+0.1%	2.6%	0.4%
query22	0.71	0.75	+6.4%	1.3%	9.5%
query23	0.73	0.74	+1.8%	1.4%	0.3%
query24	0.69	0.70	+0.8%	1.8%	0.8%
query25	0.79	0.80	+1.2%	1.3%	1.0%
query26	0.73	0.74	+0.3%	1.9%	0.6%
query27	0.86	0.86	+0.3%	1.7%	0.8%
query28	1.39	1.40	+1.1%	1.1%	1.0%
query29	0.69	0.69	+0.3%	1.0%	1.0%
query30	41.57	41.78	+0.5%	0.1%	0.4%

Geomean branch/main 1.017; median main run gap 2.5%

graph-benchmark-ldbc: lazy

Query	main (ms)	branch (ms)	change	main run gap	branch run gap
query1	1.13	1.14	+1.0%	0.8%	2.3%
query2	1.41	1.39	-1.3%	2.0%	0.9%
query3	0.79	0.79	-0.1%	0.6%	2.1%
query4	0.78	0.78	+0.4%	0.4%	0.7%
query5	3.51	3.53	+0.5%	0.4%	2.2%
query6	0.73	0.73	+1.1%	0.1%	0.8%
query7	1.08	1.09	+0.5%	0.2%	2.2%
query8	1.72	1.74	+1.6%	0.6%	2.0%
query9	1.33	1.33	+0.0%	0.6%	0.5%
query10	0.92	0.93	+0.3%	0.4%	0.6%
query11	1.96	1.96	-0.3%	0.7%	1.2%
query12	6.64	6.77	+1.9%	4.1%	0.2%
query13	1.04	1.04	+0.1%	1.1%	0.1%
query14	0.68	0.68	-0.0%	0.8%	0.4%
query15	0.76	0.76	-0.4%	1.2%	0.4%
query16	1.10	1.10	-0.2%	0.9%	0.4%
query17	1.38	1.38	-0.5%	0.2%	0.5%
query18	1.13	1.13	-0.2%	0.3%	0.3%
query19	1.50	1.50	+0.2%	0.0%	0.2%
query20	23.68	23.50	-0.8%	0.3%	0.2%
query21	0.84	0.83	-0.3%	0.4%	0.4%
query22	0.92	0.92	+0.2%	0.7%	0.1%
query23	1.23	1.22	-0.2%	0.4%	0.1%
query24	1.05	1.05	-0.1%	0.1%	0.3%
query25	1.18	1.19	+0.4%	0.6%	0.7%
query26	1.06	1.06	+0.4%	0.4%	0.1%
query27	1.30	1.29	-0.6%	1.0%	0.3%
query28	2.01	2.02	+0.3%	0.8%	0.7%
query29	1.02	1.03	+0.2%	0.4%	0.5%
query30	41.44	41.42	-0.1%	0.9%	1.2%

Geomean branch/main 1.001; median main run gap 0.6%

graph-benchmark-ldbc-complex: memory

Query	main (ms)	branch (ms)	change	main run gap	branch run gap
query1 (plan changed)	22.95	23.28	+1.5%	1.1%	0.6%
query2	2.45	2.44	-0.4%	3.1%	1.0%
query3 (plan changed)	25.76	25.83	+0.3%	3.1%	1.5%
query4 (plan changed)	5.75	5.76	+0.1%	0.3%	1.0%
query5 (plan changed)	74.62	75.95	+1.8%	1.7%	0.1%
query6	21.47	21.74	+1.3%	2.1%	1.1%
query7	9.60	9.61	+0.1%	0.4%	0.7%
query8	2.27	2.24	-1.1%	0.8%	0.8%
query9	38.09	37.99	-0.3%	4.8%	3.1%
query10	20.38	20.20	-0.9%	0.5%	1.2%
query11	2.98	2.97	-0.3%	0.2%	0.6%
query12	17.43	18.02	+3.4%	6.4%	16.4%
query13	1.11	1.08	-2.5%	1.6%	2.4%
query14 (plan changed)	48.74	47.19	-3.2%	1.5%	1.9%

Geomean branch/main 1.000; median main run gap 1.6%

graph-benchmark-ldbc-complex: lazy

Query	main (ms)	branch (ms)	change	main run gap	branch run gap
query1 (plan changed)	32.03	32.00	-0.1%	0.1%	0.0%
query2	25.40	25.52	+0.4%	0.5%	1.5%
query3	165.02	164.81	-0.1%	1.6%	0.8%
query4	12.90	12.75	-1.1%	0.5%	1.7%
query5 (plan changed)	110.99	101.22	-8.8%	0.1%	0.8%
query6	22.96	22.94	-0.1%	0.5%	1.5%
query7 (plan changed)	14.43	13.52	-6.3%	1.3%	0.2%
query8	3.29	3.27	-0.6%	2.0%	0.8%
query9	311.50	306.65	-1.6%	2.1%	2.6%
query10	28.57	29.01	+1.5%	0.0%	0.0%
query11	5.32	5.33	+0.1%	0.1%	0.2%
query12 (plan changed)	20.02	19.81	-1.0%	0.8%	0.7%
query13	1.42	1.44	+0.8%	0.3%	1.6%
query14	50.09	50.50	+0.8%	0.9%	0.1%

Geomean branch/main 0.988; median main run gap 0.5%

🤖 Generated with Claude Code