chore(bench): one command compares a one-file sync before and after a change

.claude/skills/codegraph-build-validation/SKILL.md

@@ -87,10 +87,13 @@ Base effort quotes on these rather than on how heavy a step sounds. Measured on
 | Worker upstream reconcile with conflict resolution and full verification | 15–28 min |
 | Upstream reconcile to a reviewed PR: worker, review, review fixes | 38–52 min |
 | Stacked landing: stack suite, squash-merge each PR, tree check | ~5 min |
+| One-file sync comparison on n8n, `scripts/benchmarks/runtime/sync-ab.mjs` (two sessions, 22 runs) | 5 min; 3 min 30 s with a stored noise floor; `--proxies` 1 min 30 s |
 | Pre-deploy gate: build, kernel, suite and source probes must all pass before a build is promoted (run by a maintainer-local helper that is not part of this repository) | 2.5 min |
 
 `eval:precision` per corpus is not recorded yet; add it the next time one is timed.
 
+For a claim that a change moves a one-file sync's memory or time, run `scripts/benchmarks/runtime/sync-ab.mjs` rather than writing a driver: it restores a saved index per run, handles the synthesis skip-list fingerprint, stops when the two builds do different work, and writes the PR table. Screen with `--proxies` first; usage is in `scripts/benchmarks/runtime/README.md`.
+
 ### Windows-gated tests
 
 Behavior that differs by platform (path resolution, drive letters, `SENSITIVE_PATHS`, `%APPDATA%` config dirs, CRLF) must be gated, not assumed. Use `it.runIf(process.platform === 'win32')(...)` for Windows-only assertions and `it.runIf(process.platform !== 'win32')(...)` for POSIX-only ones — e.g. `/etc` is sensitive on POSIX but resolves to `C:\etc` (non-existent) on Windows, so an ungated `/etc` assertion fails on Windows. Validate the Windows side for real (see below); don't merge a Windows-gated test you haven't seen run.

scripts/benchmarks/runtime/README.md

@@ -10,6 +10,7 @@ The arms can differ in build, in runtime, or in both.
 | `run-index-sync.sh <list>` | Full index, then two one-file syncs: wall time, peak RSS, CPU, database size, node and edge counts | `index-sync.tsv`, `logs/` |
 | `run-mcp.sh <list>` | MCP server: start to first explore answer, warm explore median and p90, 8 explores at once, watcher sync, idle memory and CPU | `mcp.jsonl` |
 | `run-cli.sh <corpus> <query>` | CLI start: `--version`, `status`, one `explore` (hyperfine, 20 runs) | `cli-*.json`, `cli.txt` |
+| `sync-ab.mjs --corpus … --arm base=… --arm cand=…` | One change on one corpus: a one-file sync before and after, with same-build controls, until the result qualifies or stays inside run-to-run variation (see below) | `runs.jsonl`, `summary.md`, `summary.json`, `chart.json`, `logs/` in `--out` |
 | `summarize.py $BENCH_OUT` | Medians per corpus and arm from `index-sync.tsv`, and from `mcp.jsonl` when present | stdout |
 | `render-charts.mjs $BENCH_OUT --date D --upstream SHA --fork SHA --node V --bun V` | The README chart (`--cards`), the benchmark doc's heatmap (`--heatmap`) and the current run's result tables as Markdown (`--tables`; the benchmark doc's comparison with an earlier run is written by hand), from the accepted attempts in `$BENCH_OUT` (directories `index-<corpus>-aN`, `mcp-<corpus>-aN` and `cli-gin-aN` whose `status` file reads `accepted`) | SVG and Markdown files |
 
@@ -57,6 +58,103 @@ tracked, because each run restores it with `git checkout`.
 
 `queries/` holds the explore questions used for gin and n8n.
 
+## One-file sync comparison
+
+`sync-ab.mjs` answers "does this change move a one-file sync's peak memory, wall
+time or CPU time on this corpus?" in one command, for a pull request's numbers.
+It needs a saved index of the corpus (a `.codegraph` directory made with
+`codegraph init` and copied aside) and two built checkouts or two runtimes.
+
+```bash
+node scripts/benchmarks/runtime/sync-ab.mjs \
+  --corpus ~/cg-scratch/n8n --file packages/workflow/src/utils.ts \
+  --append '\nexport const benchProbeMarker = 1;\n' --index ~/cg-scratch/n8n-index \
+  --arm base=$(fnm exec --using codegraph which node)@$HOME/worktrees/base \
+  --arm cand=$(fnm exec --using codegraph which node)@$PWD \
+  --out ~/cg-scratch/ab-$(date +%F-%H%M) --save-noise ~/cg-scratch/noise.json
+```
+
+Each timed run restores the saved index into `--cgdir` (default `.cg-ab`) and
+checks it byte for byte. It appends `--append` to `--file`, runs `sync` under
+`/usr/bin/time -v`, and restores the file. Then it hashes the edges and
+near-duplicate tables. The order of the two builds alternates within each
+session. A session has `--pairs` before/after pairs (default 3) and
+`--controls` same-build pairs (default 2), whose largest deviation is the
+noise floor. A metric qualifies when every session's median after/before ratio
+lies outside both the `--band` (default 0.03) and that floor, in the same
+direction.
+
+It stops early in three cases:
+
+- **The two builds did different work.** After the first pair it compares the
+  graph hashes and the counts in the sync's `[phase-timing]` and
+  `[synth-timing]` lines (references resolved, files resynthesized, pairs
+  re-scored), and checks whether either build rebuilt the synthesis skip list.
+  When the graph differs, it exits with status 3. When only the work differs,
+  it also exits with status 3, unless you pass `--allow-work-diff` for a change
+  meant to do less work.
+- **The answer is already clear.** After the first session it stops when every
+  pair lies outside the band and outside twice the control spread on `--metric`
+  (default `peak`).
+- **The noise floor is already known.** `--noise FILE` reuses the floor that
+  `--save-noise` stored for the same host, kernel, runtime version, corpus
+  revision, edit and `--env`, and then runs no controls. A kernel or runtime
+  upgrade changes the key, so it is measured again.
+
+The saved index keeps its synthesis skip list under a fingerprint of the
+resolution code that wrote it. A build whose resolution code differs would
+ignore the list and rescan every file. That is a real cost, the first sync
+after an upgrade, but it is not the change under test. So each build first
+runs once untimed, and when that run rebuilt the list, the build gets a copy of
+the index labelled with its own fingerprint. The run prints a note when this
+happens. `--no-relabel` measures the rescan instead.
+
+`--proxies` runs each build once under `perf stat -e instructions:u`, and on
+Node once with `alloc-total.cjs`, which totals the bytes allocated. These
+counts vary much less than peak memory or wall time, so they screen a change
+in four runs before the full comparison. The summary reports the after/before
+ratio of each.
+
+What the proxies predicted for four landed changes, against the results their
+pull requests measured (2026-10-18; a repeat of two of them moved each ratio by
+0.003 at most):
+
+| Change | Runtime | Measured result | Instructions | Bytes allocated |
+|---|---|---|---:|---:|
+| #493 (Python import line) | Node | sync 29% faster | 0.676 | 1.000 |
+| #493 | Bun | sync 35% faster | 0.658 | n/a |
+| #494 (bases memo) | Node | sync about 3% faster | 0.957 | 0.999 |
+| #494 | Bun | sync time no clear change | 0.959 | n/a |
+| #528 (near-duplicates) | Node | peak memory no clear change | 0.998 | 0.947 |
+| #528 | Bun | peak memory 12% lower | 1.036 | n/a |
+| #529 (import-only lookup) | Node | peak memory 4.7% lower, wall no clear change | 0.952 | 0.930 |
+| #529 | Bun | peak memory 5.7% lower | 0.971 | n/a |
+
+The instruction count tracked large time changes closely (#493). Changes of
+about 5% in instructions did not reliably show up in wall time, whose
+same-build variation on n8n was up to 7%. Bytes allocated did not predict peak
+memory: #528 allocated 5% less on Node with no change in peak, and its 12% drop
+on Bun had no proxy at all. Use `--proxies` to tell whether a change does less
+work, and run the full comparison for any memory claim.
+
+`summary.md` holds the result table and the per-session ratios for a pull
+request body. `chart.json` holds the qualified rows in the input format of the
+before/after chart script. Every timed run is a line in `runs.jsonl`.
+Admission waits until no `capped-*` scope runs, at least `--min-avail-gib`
+(default 8) is free and the 1-minute load is under `--max-load` (default 4).
+`BENCH_WRAP` prefixes each run, for example a memory cap; unlike the other
+runners, it defaults to nothing.
+
+Measured on n8n 7cc0450a, comparing #528 (5ccc8336) with #529 (the import-only
+lookup), on Node 24.22.0:
+
+| Command | Runs | Wall-clock | Result |
+|---|---:|---|---|
+| Default, two sessions | 22 | 5 min | Peak 961 to 915.5 MiB, 4.7% lower, qualified (controls 2.6%) |
+| `--noise` with the floor stored by the run above | 14 | 3 min 30 s | 953 to 910.5 MiB, 4.5% lower, qualified |
+| `--proxies` | 6 | 1 min 30 s | Instructions 0.952, bytes allocated 0.930 (1,692 to 1,574 MiB) |
+| `--no-relabel` | 4 | 1 min 30 s | Stopped after the first pair: the after build rebuilt the skip list |
+
 ## After a Bun upgrade
 
 Keep the previous Bun executable and run both against the same build, with

scripts/benchmarks/runtime/alloc-total.cjs

@@ -0,0 +1,26 @@
+// Preload for sync-ab.mjs --proxies (Node only): total bytes allocated over the whole
+// command, from V8's sampling heap profiler with collected objects kept. Writes
+// {"allocatedMiB": N} to ALLOC_TOTAL_OUT at exit.
+const fs = require('node:fs');
+const inspector = require('node:inspector');
+const out = process.env.ALLOC_TOTAL_OUT;
+const session = new inspector.Session();
+session.connect();
+session.post('HeapProfiler.startSampling', {
+  samplingInterval: 4096,
+  includeObjectsCollectedByMajorGC: true,
+  includeObjectsCollectedByMinorGC: true,
+});
+process.on('exit', () => {
+  // Same-thread inspector sessions answer synchronously, so this runs before exit.
+  session.post('HeapProfiler.stopSampling', (error, result) => {
+    if (error) return fs.writeFileSync(out, JSON.stringify({ error: String(error) }));
+    let total = 0;
+    const walk = (node) => {
+      total += node.selfSize;
+      for (const child of node.children) walk(child);
+    };
+    walk(result.profile.head);
+    fs.writeFileSync(out, JSON.stringify({ allocatedMiB: Math.round((total / 2 ** 20) * 10) / 10 }));
+  });
+});