Skip to content

bench: add reproducible engine and context shaping workload - #103

Open
milanb17 wants to merge 1 commit into
mainfrom
codex/engine-performance-bench
Open

milanb17 wants to merge 1 commit into
mainfrom
codex/engine-performance-bench

Conversation

@milanb17

@milanb17 milanb17 commented Oct 8, 2026

Copy link
Copy Markdown

Summary

Add a reproducible Criterion workload for investigating engine/context-folding performance:

cargo bench --locked --bench engine
├── init/queries/{default,no_context}
├── engine/pr998/{default,no_context}
└── shape/pr998/context

Bundles four TypeScript/TSX before/after pairs from Whiteboard PR #998 (215,152 bytes), with provenance, hashes, and their MIT license. The benchmark reuses the actual CLI component host; the only host-source edit makes a module path explicit so it can also compile in the bench target.

docs/benchmarks.md describes baseline comparisons, timing boundaries, and rebuilding bundled plugin components after changing their Rust sources. These are native CLI/Wasmtime measurements, not browser-WASM timings. No-context also removes its query contributions; isolated shaping uses fresh clones of the same default-query trees.

Evidence

  • Before: no checked-in cargo bench workload for this regression; reproducing the earlier measurement required an external fixture/harness.
  • After: all five benchmarks completed in an optimized build on Rust 1.99.0. Initial Criterion point estimates: query compilation 616 ms default / 100 ms without context; four-file processing 2.266 s default / 161 ms without context; isolated context shaping 1.849 s. These are local observations, not hardware-independent thresholds.
  • Passed cargo bench --locked --bench engine -- --save-baseline initial and cargo bench --locked --bench engine -- --test (all five report success).
  • Passed formatting, diff whitespace checks, and all eight fixture hash/byte-count checks. Preflight rejects structural fallbacks, plugin failures, and source-text changes.

Merge Danger

Door: two-way. Blast Radius: development-only benchmark/dependency/fixture additions and one equivalent module-path annotation; no intended engine behavior change. Initial compilation includes Wasmtime and parser dependencies; full sampling takes longer than the quick --test check.

AI-assisted: OpenAI Codex.

Benchmark query compilation, full core/shape processing, and isolated context shaping with Criterion and bundled PR998 inputs. Document timing boundaries and plugin rebuild requirements.

AI-assisted: OpenAI Codex.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant