follow-up: Pre-publish benchmark gate's diffImpact-latency check flakes on unrelated PRs (mirrors #2436's Full build / 1-file rebuild noise)

[carlos-alm]

Summary

tests/benchmarks/regression-guard.test.ts's query-benchmark suite (diffImpact latency, baseline consistently 10.1ms per the recorded dev vs 3.17.0 comparison) has flaked repeatedly across multiple, completely unrelated PRs in a single fixer-batch session on 2026-08-19:

PR	Change	diffImpact result
#2581 (issue #2526)	.claude/hooks/guard-git.sh bash hook	10.1 → 23.2ms (round 1), also flaked separately on No-op rebuild
#2582 (issue #2530)	src/domain/analysis/exports.ts query logic	10.1 → 18/22.0/21.8ms across 3 separate rounds
#2585 (issue #2558)	.claude/hooks/normalize-ifs.mjs bash hook	10.1 → 21.8ms (round 1), also flaked separately on No-op rebuild
#2586 (issue #2560)	src/extractors/javascript.ts / crates/codegraph-core/src/extractors/javascript.rs — one new match arm per engine	10.1 → 21.0ms, then 10.1 → 22.2ms (two consecutive failures on the SAME metric)

The baseline (10.1) is byte-identical across every single occurrence — strongly suggesting it's a static recorded value from benchmark history (not re-measured each run), while the "dev" measurement is a live, per-CI-run query-latency sample landing consistently in the 18-23ms range regardless of what the PR actually changes (a bash hook script never invoked during buildGraph, TS query-layer code, and now Rust/TS extractor code — none of which share any plausible causal path to diffImpact's query-time cost).

Why this needs its own follow-up

This is the same underlying class of problem #2436/#2550 already documented for the Full build/1-file rebuild metrics (cross-invocation CI-runner variance that an in-process median can't smooth out) — but for a different metric (diffImpact latency, a query-benchmark metric, not a build-benchmark one) that isn't yet covered by any existing mitigation.

Suggested fix

Mirror the existing KNOWN_REGRESSIONS stopgap already used for 3.16.0:Full build / 3.16.0:1-file rebuild (see tests/benchmarks/regression-guard.test.ts) with an equivalent entry for the current version's diffImpact latency query metric, until the deeper cross-invocation noise problem tracked in #2550 is actually resolved.

Related

• Pre-publish benchmark gate fails non-deterministically on unrelated PRs #2436 (original benchmark-gate non-determinism investigation)
• follow-up: benchmark gate noise is cross-invocation (CI runner variance), not within-process — needs job-level replication or dedicated runner (#2436) #2550 (cross-invocation noise needs job-level replication or a dedicated runner — root cause, still open, needs infra I can't provision)