Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 23 additions & 7 deletions scripts/query-benchmark.ts
Original file line number Diff line number Diff line change
Expand Up @@ -109,13 +109,18 @@ console.log = (...args) => console.error(...args);

const RUNS = 5;

// First 2-3 native fnDeps calls per process pay a cold-start cost (rusqlite
// statement-cache warmup, OS page cache for the DB file, NAPI-side static
// init from tree-sitter's transitive crates linked into the .node binary).
// On Linux x86_64 CI, that pulled median(5) into cold-start territory once
// tree-sitter 0.25 grew the binary's init footprint (#1076), even though
// steady-state per-call latency is unchanged. Discard the first WARMUP_RUNS
// before timing so the metric reflects warm-call latency, not cold-start.
// The first 2-3 native calls into a given query path per process pay a
// cold-start cost (rusqlite statement-cache warmup, OS page cache for the DB
// file, NAPI-side static init from tree-sitter's transitive crates linked
// into the .node binary). On Linux x86_64 CI, that pulled median(5) into
// cold-start territory once tree-sitter 0.25 grew the binary's init footprint
// (#1076), even though steady-state per-call latency is unchanged. Discard
// the first WARMUP_RUNS before timing so the metric reflects warm-call
// latency, not cold-start.
//
// This applies per query path, not just once per process: each of fnDeps,
// fnImpact and diffImpact prepares its own statements and touches its own DB
// pages, so warming one does not warm the next (#2590).
const WARMUP_RUNS = 3;

async function benchDepths(fn, name, depths) {
Expand Down Expand Up @@ -166,6 +171,17 @@ async function benchDiffImpact(targets: HubTargets) {
fs.writeFileSync(hubFile, original + '\n// benchmark-probe\n');
execFileSync('git', ['add', hubFile], { cwd: root, stdio: 'pipe' });

// Warm the diffImpact path before timing, exactly as benchDepths does for
// fnDeps/fnImpact. diffImpactData is a distinct query path from those --
// its own prepared statements, its own DB pages, and a `git` subprocess to
// read the staged diff -- so the warmup those calls already paid does not
// carry over. Without this, diffImpact was the only query metric measured
Comment on lines +175 to +178

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Prepared-statement warmup claim is inaccurate

The new explanation says these calls warm diffImpact's prepared statements, but every invocation opens and closes a separate database connection and the statement cache is connection-scoped. Only process-wide initialization and OS page-cache state persist, so this wording can mislead future benchmark investigations about which cold-start costs remain in measured runs.

Fix in Claude Code

// cold: the same defect #2584 fixed for benchmark.ts's Full build, and
// that #2436 traced part of the gate's non-determinism to (#2590).
for (let i = 0; i < WARMUP_RUNS; i++) {
diffImpactData(dbPath, { staged: true, depth: 3, noTests: true });
}

let lastResult = null;
const latencyMs = round1(
await timeMedian(() => {
Expand Down
Loading