Add a focused Criterion benchmark suite by gmemuriuki · Pull Request #283 · symposium-dev/symposium

gmemuriuki · 2026-09-01T19:26:42Z

What does this PR do?

This PR adds the first focused benchmark suite for Symposium.
Rather than starting with a broad collection of microbenchmarks, it answers one concrete
performance question: how much latency does Symposium's in-process PreToolUse path adds to
unchanged workspace?

The initial suite measures four cases:

workspace dependency resolution with an empty Symposium cache
dependency resolution through a new disk resolver with a valid disk cache
PreToolUse dispatch with minimal configuration
PreToolUse dispatch with a representative local plugin registry.

Pairing the component and hook measurements makes the result easier interpret. In particular
it shows how much work the workspace cache avoids and makes the existing Cargo workspace-lookup
cost visible.

what changed

Added a non-publishable symposium-benchsuite workspace package using Criterion.
Added deterministic Cargo workspace, configuration, and local registry fixtures.
Added reusable fixture staging, sandboxing, validation, and mock-Cargo support.
Added integration tests covering resolver memoization, disk-cache reuse, and lockfile invalidation.
Documented a measurement contract beside every benchmark target.
Added normal CI checks that compile benchmark targets on native Linux, macOS, and Windows jobs.
Added a dedicated benchmark workflow for manual runs and relevant pull requests.
Updated existing workflows to use Node 24-compatible GitHub Actions.

Disclosure questions

AI disclosure.

Other: I used Codex and Claude throughout this work for research, design discussion, code review, and some implementation and editing. I reviewed and approved the resulting changes and made the final design decisions.

Questions for reviewers.

Document the initial workloads, fixture organization, and measurement contracts. Define the CI strategy and lifecycle for introducing performance gates after the suite has collected stable results.

Add a non-publishable workspace package for benchmarks and shared support code. Disable automatic benchmark discovery, exclude benchmark files from published crates, and document the suite structure and lifecycle.

Add a reusable Cargo workspace and local plugin registry for the benchmark suite. Document the workload invariants so missing or malformed fixture data fails setup instead of producing misleading results.

Resolve checked-in fixtures independently from the benchsuite package path. Add focused tests for the project and registry fixtures.

Copy benchmark fixtures into fresh sandbox directories. Reject unsupported entries and existing destinations, and cover recursive copying and no-merge behavior with focused tests.

Create temporary configuration and cache directories for benchmark workloads, and stage fixtures on demand. Reject duplicate staging to prevent accidental fixture merges.

Record Cargo invocations around repeated resolver loads. Verify that the initial cache miss runs once and subsequent loads use the in-memory result.

Verify resolver memoization, disk-cache reuse, and deterministic Cargo.lock invalidation. Return the generated mock Cargo script path from testlib so call-log assertions do not depend on its directory.

Clear only WorkspaceDeps cache entries during benchmark setup while preserving unrelated sandbox caches. Cover idempotent clearing and failure diagnostics in the benchsuite tests.

Centralize fixture metadata and validate staged layouts and Cargo workspace shapes before measurement. Reject missing members, non-path dependencies, and paths outside the staged fixture so setup failures cannot appear as faster samples.

Move isolated filesystem setup and fixture staging into a focused module. Keep workspace cache clearing narrow so benchmarks preserve unrelated configuration and binary cache state.

Keep the crate root as a small public facade over fixture and sandbox responsibilities. Let benchmark targets continue to own their scenario semantics and timed operations.

Stage and validate the reference project, then clear only Symposium's workspace cache before each timed load. Verify cache population and reset behavior up front so path drift cannot turn misses into silently faster samples.

Add a metadata-rejecting Cargo guard so cache-hit failures cannot fall back to metadata unnoticed. Measure new resolvers against the prepared disk cache with bounded sampling, and document how to compare and report the results.

Build and validate an isolated unchanged-workspace PreToolUse workload. Reject invalid setup before it can become a faster measurement.

Measure the unchanged-workspace PreToolUse path. Validate configuration, cache state, and output before sampling. Use flat sampling so warm-up variance cannot change the model.

Use flat sampling for long-running cache misses. Keep linear sampling for the shorter disk-cache hit case. Prevent latency drift from silently changing either model.

Require the predicate-gated hook script during fixture staging. A missing command now fails before measurements can begin.

Document the production-valid configurations selected by each case. Record the relative registry path and required hook script.

Add a representative three-plugin local-registry dispatch case. Share case preparation while validating exact registries, plugins, cache state, predicate state, and guarded no-op output. Record that Cargo subprocesses limit the measurement resolution.

Describe the local-registry case as an end-to-end measurement. Note that workspace lookup can mask smaller in-process changes.

Compile the benchmark suite on Linux, macOS, and Windows so target drift is caught by ordinary pull request CI. Keep the musl job focused on its cross-compilation target. Full smoke and measurement runs remain in the dedicated benchmark workflow.

Run every workload once as a correctness smoke test before collecting Criterion measurements in a pinned Linux and Rust environment. Write experimental medians to the job summary and retain raw results for post-hoc inspection without caching baselines or gating merges.

Align the design with path coverage, smoke validation, Node 24 actions, result labeling, and rerun-safe artifact handling. Remove the precedent lists so the document stays focused on decisions and the benchmark operating contract.

Move checkout and dependency caching to their Node 24 runtime lines before GitHub removes Node 20 from hosted runners.

Move checkout, artifact upload, and GitHub release publishing to their supported Node 24 runtime lines.

Fluzko · 2026-09-02T20:17:49Z

The #[cfg(windows)] batch shims never executes in CI: the Windows job only does cargo check, and the benchmarks workflow is Linux-only.
The native jobs should run cargo test -p symposium-benchsuite --lib too? It's nearly cero time and would actually exercise the shim plus the fixture validation on macOS and Windows

Fluzko · 2026-09-02T20:23:35Z

+cargo test -p symposium-benchsuite --benches
+```
+
+Smoke runs execute workloads without collecting full measurements. Normal pull request CI compiles benchmark targets and runs the small support-library and cache-invariant tests. Full smoke and measurement runs belong to the benchmark workflow.


I think that support library tests are not ran. They only run inside the path-filtered Benchmarks workflow (where --benches happens to also pull in the lib unittests).
Also says that cargo package --list "verifies" that benches are excluded, but nothing in xtask/ or .github/ runs it.

gmemuriuki added 30 commits September 1, 2026 13:41

docs: add benchmark suite design

3d10439

Document the initial workloads, fixture organization, and measurement contracts. Define the CI strategy and lifecycle for introducing performance gates after the suite has collected stable results.

build: scaffold the benchmark suite

69942b2

Add a non-publishable workspace package for benchmarks and shared support code. Disable automatic benchmark discovery, exclude benchmark files from published crates, and document the suite structure and lifecycle.

test: add deterministic benchmark fixtures

2631a7b

Add a reusable Cargo workspace and local plugin registry for the benchmark suite. Document the workload invariants so missing or malformed fixture data fails setup instead of producing misleading results.

bench: add fixture discovery support

27b4d28

Resolve checked-in fixtures independently from the benchsuite package path. Add focused tests for the project and registry fixtures.

bench: add fixture copying support

8fc4ff1

Copy benchmark fixtures into fresh sandbox directories. Reject unsupported entries and existing destinations, and cover recursive copying and no-merge behavior with focused tests.

bench: add isolated benchmark sandboxes

7e6515e

Create temporary configuration and cache directories for benchmark workloads, and stage fixtures on demand. Reject duplicate staging to prevent accidental fixture merges.

test: add workspace cache fixture

a1c82d0

test: verify workspace cache memoization

6bcdc16

Record Cargo invocations around repeated resolver loads. Verify that the initial cache miss runs once and subsequent loads use the in-memory result.

test: strengthen workspace cache coverage

2174be1

Verify resolver memoization, disk-cache reuse, and deterministic Cargo.lock invalidation. Return the generated mock Cargo script path from testlib so call-log assertions do not depend on its directory.

bench: add workspace cache clearing

cfc7240

Clear only WorkspaceDeps cache entries during benchmark setup while preserving unrelated sandbox caches. Cover idempotent clearing and failure diagnostics in the benchsuite tests.

bench: add symposium dependency to benchsuite

8c8b272

bench: add typed fixture validation

dea9684

Centralize fixture metadata and validate staged layouts and Cargo workspace shapes before measurement. Reject missing members, non-path dependencies, and paths outside the staged fixture so setup failures cannot appear as faster samples.

refactor: extract benchmark sandbox support

abe7307

Move isolated filesystem setup and fixture staging into a focused module. Keep workspace cache clearing narrow so benchmarks preserve unrelated configuration and binary cache state.

refactor: split benchmark support into modules

85dc995

Keep the crate root as a small public facade over fixture and sandbox responsibilities. Let benchmark targets continue to own their scenario semantics and timed operations.

docs: describe benchmark support architecture

07a4f1a

bench: configure workspace dependency benchmarks

5f35ad3

bench: measure workspace cache misses

f8eb0b8

Stage and validate the reference project, then clear only Symposium's workspace cache before each timed load. Verify cache population and reset behavior up front so path drift cannot turn misses into silently faster samples.

docs: document workspace dependency benchmark

090ba7a

bench: measure workspace disk cache hits

d5961b9

Add a metadata-rejecting Cargo guard so cache-hit failures cannot fall back to metadata unnoticed. Measure new resolvers against the prepared disk cache with bounded sampling, and document how to compare and report the results.

bench: set up hook dispatch target

ecc9d9a

bench: add sandbox configuration setup

3c2a18a

bench: trim redundant comments from fixture and sandbox

d474ec8

bench: record metadata attempts in the Cargo guard

1df9104

bench: prepare minimal hook dispatch workload

5c7db6f

Build and validate an isolated unchanged-workspace PreToolUse workload. Reject invalid setup before it can become a faster measurement.

bench: measure minimal hook dispatch latency

189474c

Measure the unchanged-workspace PreToolUse path. Validate configuration, cache state, and output before sampling. Use flat sampling so warm-up variance cannot change the model.

bench: make workspace sampling modes explicit

1748742

Use flat sampling for long-running cache misses. Keep linear sampling for the shorter disk-cache hit case. Prevent latency drift from silently changing either model.

bench: add minimal hook configuration fixture

a8e4583

bench: add local registry configuration fixture

f1929a4

bench: validate the local registry command fixture

21181b3

Require the predicate-gated hook script during fixture staging. A missing command now fails before measurements can begin.

docs: describe hook benchmark fixture configuration

cf26b13

Document the production-valid configurations selected by each case. Record the relative registry path and required hook script.

gmemuriuki added 11 commits September 1, 2026 20:04

docs: clarify hook benchmark resolution

3f0b2ad

Describe the local-registry case as an end-to-end measurement. Note that workspace lookup can mask smaller in-process changes.

docs: list the local registry hook benchmark

19fd124

docs: consolidate hook benchmark contracts

3a410b2

docs: update benchmark automation design

94f5bd8

Align the design with path coverage, smoke validation, Node 24 actions, result labeling, and rerun-safe artifact handling. Remove the precedent lists so the document stays focused on decisions and the benchmark operating contract.

ci: update CI actions for Node 24

ed6e30c

Move checkout and dependency caching to their Node 24 runtime lines before GitHub removes Node 20 from hosted runners.

ci: update Pages actions for Node 24

fd396fb

ci: update release actions for Node 24

4d7df5e

Move checkout, artifact upload, and GitHub release publishing to their supported Node 24 runtime lines.

ci: update release checkout action

2e45ff3

gmemuriuki changed the title ~~Feat/benchmark~~ Add a focused Criterion benchmark suite Sep 1, 2026

Fluzko reviewed Sep 2, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add a focused Criterion benchmark suite - #283

Add a focused Criterion benchmark suite#283
gmemuriuki wants to merge 41 commits into
symposium-dev:mainfrom
gmemuriuki:feat/benchmark

gmemuriuki commented Sep 1, 2026

Uh oh!

Fluzko Sep 2, 2026

Uh oh!

Fluzko Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

gmemuriuki commented Sep 1, 2026

What does this PR do?

what changed

Uh oh!

Fluzko Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Fluzko Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants