fix(vitest-plugin): call the bench setup and teardown hooks in analysis mode

[adriencaccia]

Stacked on #75.

A benchmark can declare setup and teardown through its options, and they never ran under CodSpeed's default instrumentation. The analysis runner drives each benchmark itself rather than handing it to tinybench, and it only ever read the bench function, so the hooks were silently dropped. Walltime mode was unaffected, since it delegates to Vitest's default execution where tinybench invokes them.

The runner now builds the same Bench and Task Vitest builds for a benchmark and calls the hooks around the warmup and the measured run, matching tinybench's cycle semantics and keeping them outside the instrumentation window.

Hook calls observed for one benchmark declaring both hooks, before and after:

mode	before	after
no CodSpeed (plain Vitest)	4	4
walltime	4	4
simulation	0	4
memory	0	4

Every mode now produces the identical sequence plain Vitest does — setup warmup, teardown warmup, setup run, teardown run, each receiving a task named after the benchmark. Checked against the packed package on vitest 4.1.11 + vite 8.2.2, vitest 4.1.11 + vite 7.3.6, and vitest 3.2.7 + vite 7.3.6.

Supersedes #46, which called the suite-level hooks rather than the per-benchmark ones and only covered a single cycle.

Thanks @ematipico for reporting this.

[greptile-apps]

Greptile Summary

The PR restores per-benchmark setup and teardown hooks in Vitest analysis and memory modes while keeping those hooks outside CodSpeed’s instrumentation window.

• Constructs the Tinybench Bench and Task objects expected by benchmark hooks.
• Calls setup and teardown for both warmup and measured cycles.
• Adds coverage for hook arguments, ordering, and measurement boundaries.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete correctness, compatibility, or security defect identified in the changed paths.

The analysis runner now preserves benchmark setup and teardown around both execution cycles, and the added test verifies that the measured setup and teardown remain outside CodSpeed’s instrumentation markers.

Important Files Changed

Filename	Overview
packages/vitest-plugin/src/analysis.ts	Adds Tinybench-compatible benchmark objects and invokes per-benchmark hooks around warmup and measured cycles without moving them into the instrumentation window.
packages/vitest-plugin/src/tests/instrumented.test.ts	Verifies setup and teardown arguments and their ordering relative to benchmark instrumentation.

Sequence Diagram

sequenceDiagram
  participant R as Analysis runner
  participant B as Tinybench hooks
  participant C as CodSpeed instrumentation
  participant F as Benchmark function
  R->>B: setup(task, "warmup")
  R->>F: warmup invocation
  R->>B: teardown(task, "warmup")
  R->>B: setup(task, "run")
  R->>C: start measurement
  C->>F: measured invocation
  R->>C: stop measurement
  R->>B: teardown(task, "run")

Loading

_{Reviews (1): Last reviewed commit: "fix(vitest-plugin): call the bench setup..." | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will regress 5 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 15 improved benchmarks
❌ 5 regressed benchmarks
✅ 218 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Memory	recursive fibo 15 with hooks	432 B	656 B	-34.15%
❌	Simulation	wait 500ms	29 ms	37 ms	-21.47%
❌	Simulation	wait 1sec	58.1 ms	73.3 ms	-20.72%
❌	WallTime	test sync baz 10	96 ns	108 ns	-11.11%
❌	WallTime	test_iterative_fibo_10	96 ns	108 ns	-11.11%
⚡	Simulation	recursive fibo 10	1,372.3 µs	300.5 µs	×4.6
⚡	Simulation	test_recursive_fibo_10	725 µs	277.6 µs	×2.6
⚡	Memory	two	2.6 KB	2 KB	+32.54%
⚡	Memory	short body	784 B	656 B	+19.51%
⚡	Memory	short body	784 B	656 B	+19.51%
⚡	WallTime	fibo darwin	10 ms	8.6 ms	+16.8%
⚡	WallTime	switch 1	84 ns	72 ns	+16.67%
⚡	WallTime	switch 1	84 ns	72 ns	+16.67%
⚡	WallTime	switch 2	96 ns	84 ns	+14.29%
⚡	WallTime	switch 2	96 ns	84 ns	+14.29%
⚡	Memory	one	2.6 KB	2.3 KB	+11.33%
⚡	WallTime	test sync baz 10	120 ns	108 ns	+11.11%
⚡	Memory	wait 1sec	20.3 KB	18.3 KB	+11.01%
⚡	WallTime	test_recursive_cached_fibo_10	2.3 µs	2.1 µs	+10.86%
⚡	Memory	long body	1.3 KB	1.2 KB	+10.39%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing feat/bench-setup-teardown (3bc98df) with feat/bump-vitest-4.1 (87ca2bf)}