Skip to content

fix(bench): give benchmark.ts's Full build metric a warmup-then-median too - #2584

Merged
carlos-alm merged 1 commit into
mainfrom
fix/issue-2549
Aug 19, 2026
Merged

fix(bench): give benchmark.ts's Full build metric a warmup-then-median too#2584
carlos-alm merged 1 commit into
mainfrom
fix/issue-2549

Conversation

@carlos-alm

Copy link
Copy Markdown
Contributor

Summary

Closes #2549

Test plan

  • scripts/ is excluded from both tsconfig.json and the biome lint scope (src/+tests/ only), and no existing unit test imports or executes scripts/benchmark.ts directly (confirmed: tests/unit/hub-selection.test.ts's reference to scripts/benchmark.ts is a synthetic fixture filename string, unrelated to the real file's content) — matching PR fix(bench): give Full build the same warmup as the other incremental metrics #2551's own precedent for this exact class of fix (no test file changes there either, since verifying requires an actual full build's timing behavior, impractical for the fast suite).
  • Ran node --experimental-strip-types --import ./scripts/ts-resolve-loader.js scripts/benchmark.ts directly against this repo — completed successfully for both engines (wasm and native), producing valid buildTimeMs, perFile.buildTimeMs, and phases output for each.
  • npx vitest run tests/unit/hub-selection.test.ts — 6/6 pass, confirming the unrelated fixture reference is unaffected.
  • npm run lint clean.

…n too

scripts/benchmark.ts's buildTimeMs (feeding BUILD-BENCHMARKS.md) measured a
single cold buildGraph call with no warmup and no median at all -- a worse
instance of the exact class of bug #2551 already fixed in the sibling
incremental-benchmark.ts's own fullBuildMs (feeding INCREMENTAL-BENCHMARKS.md).
Full build is the first buildGraph call in this worker process's lifetime,
so it's the metric most exposed to one-time NAPI/rusqlite/OS-page-cache
cold-start cost.

Mirrors incremental-benchmark.ts's methodology exactly: 2 warmup runs
(deleting the DB before each, matching a real full build), then a median of
5 timed runs via timeMedianWithValue, which also preserves buildResult for
the phases breakdown further down. Verified by running the script directly
against this repo for both engines -- valid buildTimeMs/perFile/phases
output for each.

docs check acknowledged
@greptile-apps

greptile-apps Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR changes the full-build benchmark from one cold sample to two warmups followed by the median of five timed runs, while retaining the build result associated with the median sample for phase reporting.

  • Adds a dedicated five-run full-build sample count.
  • Deletes the benchmark database before every warmup and timed full build.
  • Uses timeMedianWithValue so timing and phase data come from the same selected run.

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness, security, or quality issues identified.

The timing helper preserves the result from the median-duration invocation, and the repeated database-reset methodology matches the repository’s existing sibling full-build benchmark.

Important Files Changed

Filename Overview
scripts/benchmark.ts Replaces the single cold full-build measurement with the established warmup-and-median methodology without introducing an actionable defect.

Reviews (1): Last reviewed commit: "fix(bench): give benchmark.ts's Full bui..." | Re-trigger Greptile

@carlos-alm
carlos-alm merged commit f054ad0 into main Aug 19, 2026
56 of 60 checks passed
@carlos-alm
carlos-alm deleted the fix/issue-2549 branch August 19, 2026 05:10
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

follow-up: scripts/benchmark.ts's build-time metric has no median/warmup at all (#2436 investigation)

1 participant