docs(examples): make profile matrix, experiment evidence, and GEPA recipes reachable

[drewstone]

Why

Three built capabilities were unreachable from the places developers look:

• runProfileMatrix had a front-door row that linked to a doc, not a runnable example — the only row that broke the "each row links to a runnable example" promise.
• ExperimentTracker (rep-level runId + EvidenceRef from feat: make experiments evidence-addressable and preflightable #622) had no front-door row and no example.
• The composed GEPA recipes (sequential, adaptive-sequential, best-of, vote, omni) were documented in docs/campaign-proposers.md but no example could run one.

What

• examples/profile-matrix/ — offline runnable of runProfileMatrix with two profiles, a judge, and integrity: 'off' explained. The README states when to use it, how, and why the integrity guard exists.
• examples/experiment-evidence/ — offline runnable of ExperimentTracker + fileExperimentStore: parent/candidate experiments, deterministic reps carrying runId + EvidenceRef, a KEEP verdict, and the persisted log path.
• examples/compare-optimization-methods/ — new GEPA_RECIPE env knob builds any of the six recipe kinds at the same total budget (evaluations and dollars split across stages). README gains a recipe table and the source-revision install caveat.
• examples/selfimprove-quickstart/README.md — one paragraph naming what the promotion gate refuses (no-op winner, non-significant paired delta, overfit gap), pointing at held-out-gate.
• Front door: runProfileMatrix row now links to its example; new ExperimentTracker row. Example index updated.

Verification

• pnpm typecheck — clean.
• pnpm typecheck:examples — clean.
• pnpm tsx examples/profile-matrix/index.ts — runs; prints per-profile means over 6 records; per-cell expectUsage warnings are expected and documented.
• pnpm tsx examples/experiment-evidence/index.ts — runs; prints ITERATE/KEEP verdicts and the persisted store path; README output matches.
• No src/ changes; no export-surface changes (the examples import ExperimentTracker from the existing /experiment subpath).

[tangletools]

✅ Auto-approved drewstone PR — a0b4b0b4

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-18T20:33:29Z}

[tangletools]

⚠️ Value Audit did not run — no verdict

This is not an approval and not a clean bill of health. Neither interrogation lens returned a judgment, so this PR has no value-audit evidence for or against it.

Status	audit-incomplete (could not run)
Why	value: cli-bridge admission rejected (queue saturated); usefulness: cli-bridge admission rejected (queue saturated)
Lenses answered	0 of 2
What to do	re-run once the CLI bridge has capacity: pr-reviewerctl trigger <repo>#<pr> --force

💰 Value — error

value agent never ran: the CLI bridge refused admission (no model was started).

• Model: opencode/deepseek/deepseek-v4-pro
• Bridge attempts: 4
• Bridge error: opencode/kimi-for-coding/k2p7: Bridge returned 503: bridge at capacity (queue_timeout, lane=reserved): active=20/20 queued=16/48 — no model was started

🎯 Usefulness — error

usefulness agent never ran: the CLI bridge refused admission (no model was started).

• Model: opencode/deepseek/deepseek-v4-pro
• Bridge attempts: 4
• Bridge error: opencode/zai-coding-plan/glm-5.2: Bridge returned 503: bridge at capacity (queue_timeout, lane=reserved): active=20/20 queued=15/48 — no model was started

🔎 Heuristic Signals

🟡 Cruft: console debug added examples/experiment-evidence/index.ts

• console.log(

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260818T203946Z}

[tangletools]

✅ No Blockers — a0b4b0b4

Review health 100/100 · Reviewer score 51/100 · Confidence 85/100 · 10 findings (2 medium, 8 low)

glm: Correctness 51 · Security 51 · Testing 51 · Architecture 51

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 5/5 planned shots over 9 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM vote recipe gets more candidate evaluations than skillopt while the matched-limits guard and artifact claim equality — examples/compare-optimization-methods/index.ts

For GEPA_RECIPE=vote the harness enforces gepaRecipeEvaluationLimit = sum(runs.maxEvaluations) + runs.lengthselectionScenarioCount (src/campaign/gepa-optimization-config.ts:159-161), passed as the enforced ceiling at src/campaign/gepa-optimization-method.ts:255-259,299-305. At defaults (GEPA_MAX_EVALUATIONS=33, SELECTION=3) that is 216 + 23 = 38 enforced evaluations for gepa vs 33 for skillopt (~15% more). assertMatchedMethodLimits at index.ts:168-172 compares only the nominal GEPA_MAX_EVALUATIONS (33 == 33) so it passes, and the artifact at index.ts:390 records candidateCaseEvaluations.gepa = 33, misreporting the enforced 38. In a budget-matched comparison example this gives GEPA-vote a silent evaluation advantage and leaves a wrong durable record. Fix: subtract runs.lengthSELECTION.l

🟠 MEDIUM README claims the quickstart's gate refuses search-to-holdout overfit candidates; the default gate has no such check — examples/selfimprove-quickstart/README.md

Added sentence: '...and refuses a candidate whose search-to-holdout gap says it won the optimizer but lost the exam.' The quickstart's release decision comes from defaultProductionGate (src/contract/self-improve.ts:684-689; examples/selfimprove-quickstart/index.ts:109 calls improve() with no gate override). defaultProductionGate (src/campaign/gates/default-production-gate.ts) runs heldout-significance, dimension-regression, budget, red-team, reward-hacking, and canary checks — none compares search-split to holdout-split scores. runImprovementLoop (src/campaign/presets/run-improvement-loop.ts) adds no gap check either. The overfit-gap refusal exists only in HeldOutGate ('overfit_gap', src/held-out-gate.ts:642), which this example does not use. Impact: a reader concludes the default selfImpr

🟡 LOW Table header still promises a runnable example for rows that link to docs — README.md

Line 82 says 'Each links to a runnable example', and this PR retargets runProfileMatrix from docs/eval-surface-map.md to a real example. But rows at lines 101-104 (runAnalystBenchmark -> docs/trace-analysis.md, deltaRepair -> docs/trace-repair-grader.md, replayVerify -> docs/trajectory-replay.md, analyzeSupervisorRun -> docs/adapters-observability.md) still link to doc pages, so the header claim stays false for four rows. Impact: minor reader confusion, same class of inconsistency this diff fixes once. Fix: either point those rows at example dirs when they exist or reword

🟡 LOW GEPA_RECIPE missing from Controls table and not trimmed — examples/compare-optimization-methods/README.md

The Controls table (README lines 115-139) lists every other environment variable but omits the newly added GEPA_RECIPE (values, default 'engine'). Also index.ts:101 reads GEPA_RECIPE without .trim(), unlike OPTIMIZER_PYTHON and BASE_URL in the same file; a value with stray whitespace fails validation with a confusing error rather than being normalized. Fail-loud behavior keeps this a nit. Fix: add a GEPA_RECIPE row and trim the value before validation.

🟡 LOW README budget claim inaccurate for adaptive-sequential and floor rounding — examples/compare-optimization-methods/README.md

'The example splits GEPA_MAX_EVALUATIONS and GEPA_MAX_PROPOSER_COST_USD evenly across stages, so every recipe runs at the same total budget.' For adaptive-sequential, GEPA_MAX_EVALUATIONS is NOT split — index.ts:128 sets one shared maxEvaluations, contradicting both this line and the table's own 'one shared evaluation budget' row (line 67). For sequential with odd E the total is 2*floor(E/2) = E-1; omni loses up to 2. Only dollar budgets are exactly equal. Fix: state that evaluations are split per stage except adaptive-sequential (shared) and vote (adds selection evaluations), and note floor rounding.

🟡 LOW README describes NOISE only as the unstable-sample verdict, omitting the in-band branch — examples/experiment-evidence/README.md

Line 7 says the tracker returns 'NOISE when the sample is too unstable to judge' (line 31 repeats 'calls an unstable sample NOISE instead of KEEP'). src/experiment-tracker.ts:257-261 also returns NOISE for a STABLE candidate whose medianDelta sits inside [-regressionThreshold, +keepThreshold] — the more common NOISE case in practice. A user reading only this README would misread a stable in-band NOISE as instability. Fix: one clause, e.g. 'NOISE when the sample is too unstable to judge or the delta sits inside the noise b

🟡 LOW Line exceeds biome lineWidth; biome check examples/experiment-evidence/ fails — examples/experiment-evidence/index.ts

The console.log template literal at line 79 is 103 chars; biome's formatter would wrap it (verified: pnpm exec biome check examples/experiment-evidence/ reports 1 format error). biome.json files.includes covers examples/**/*.ts, so any repo-wide biome run or lint-staged pass on this file flags it. Not CI-blocking (ci.yml runs pnpm lint = biome check src) and 8 pre-existing example files share the drift, so nit-level. Fix: wrap the call as biome suggests (pnpm exec biome check --write examples/experiment-evidence/).

🟡 LOW Example has no CI typecheck coverage — examples/profile-matrix/index.ts

tsconfig.json include is ["src"] and pnpm typecheck (tsc --noEmit) never touches examples/, while tsx executes by transpile-only. A future API rename in runProfileMatrix/AgentProfile/JudgeConfig would not fail CI for this file — it would only fail at example run time. It typechecks clean today (verified manually with the repo's strict flags), and this is a repo-wide pre-existing property shared by all examples, not introduced here. No action required for merge; a follow-up examples tsconfig would close the gap.

🟡 LOW 'walks each check with a promoting and a refused candidate' overstates the held-out-gate example — examples/selfimprove-quickstart/README.md

The linked example (examples/held-out-gate/index.ts:72-130) demonstrates exactly three cases: a promoting clear win, a few-runs coverage refusal, and an overfit-gap refusal. It contains no byte-identical candidate case, no CI-threshold-failure case, and no cost-ceiling case, so 'each check' (which per the prior sentence includes the byte-identical hold and the CI threshold) is not covered by a refused candidate. Impact: a reader expects a demonstration that does not exist. Fix: 'walks a promoting candidate and two refusals (too few runs, overfit gap).'

🟡 LOW Byte-identical hold attributed to 'the gate'; it is the loop's no-op guard upstream of the gate — examples/selfimprove-quickstart/README.md

Added sentence: 'The gate holds a candidate that is byte-identical to the baseline.' The behavior exists — run-improvement-loop.ts:147 computes winnerIsBaseline via surfaceHash (sha256 content prefix, src/campaign/surface-identity.ts:113) and lines 282-314 force decision 'hold' with contributing gate 'no-op-guard' without calling opts.gate.decide — but the actor is the improvement loop, not the gate, and HeldOutGate itself has no byte-identical rejection code (src/held-out-gate.ts rejectionCodes at 505-686). Impact: minor misattribution; a reader inspecting gate implementations will not find the check. Fix: 'the loop holds a winner byte-identical

---

_{tangletools · 2026-08-18T20:52:05Z · trace}

[tangletools]

✅ Approved — 10 non-blocking findings — a0b4b0b4

Full multi-shot audit completed 5/5 planned shots over 9 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-08-18T20:52:05Z · immutable trace}

[tangletools]

✅ No Blockers — 6bc0a112

Review health 100/100 · Reviewer score 61/100 · Confidence 85/100 · 9 findings (1 medium, 8 low)

glm: Correctness 61 · Security 61 · Testing 61 · Architecture 61

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 5/5 planned shots over 10 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM README claims a search-to-holdout overfit-gap refusal the quickstart's gate does not perform — examples/selfimprove-quickstart/README.md

The sentence says the gate 'refuses a candidate whose search-to-holdout gap says it won the optimizer but lost the exam.' The gate improve() actually wires for this example is defaultProductionGate (src/contract/self-improve.ts:684-689). Its checks are heldout significance, dimension-regression, budget, red-team, reward-hacking, and canary (src/campaign/gates/default-production-gate.ts:28-34) — there is no overfit-gap check, and the GateContext receives only holdout cells with no search scores (src/campaign/presets/run-improvement-loop.ts:236-247, 315-330). The overfit-gap check exists only in HeldOutGate (src/held-out-gate.ts:21-25, rejectionCode 'overfit_gap'), which this example does not use. A reader who copies this quickstart's pattern believes they have overfit protection they do not

🟡 LOW 'every recipe runs at the same total budget' overclaims for vote, odd N, and adaptive-sequential — examples/compare-optimization-methods/README.md

Verified numerically against gepaRecipeEvaluationLimit (src/campaign/gepa-optimization-config.ts:144-163): with the default N=33, vote's enforced budget is 38 (adds runs.length * selectionScenarioCount = 6 selection re-evaluations), sequential/best-of get 32 for any odd N (floor loses one), and adaptive-sequential shares rather than splits evaluations (its table row says this correctly, the blanket sentence does not). Impact is claim precision in a fairness comparison, not runtime failure. Fix: qualify the sentence, e.g. 'approximately the same total budget; vote adds selection-set re-evaluations and odd counts lose one evaluation'.

🟡 LOW GEPA_RECIPE missing from the Controls table — examples/compare-optimization-methods/README.md

The new env var is documented only in the 'Choose a GEPA recipe' section; every other variable appears in the Controls table with default and meaning. Add a row: GEPA_RECIPE | engine | GEPA composition recipe (engine|sequential|adaptive-sequential|best-of|vote|omni).

🟡 LOW README Controls row for GEPA_MAX_PROPOSER_COST_USD now describes stale per-stage semantics — examples/compare-optimization-methods/README.md

Controls table says 'Maximum GEPA model spend inside one engine stage', but gepaStage() at index.ts:111 now divides the value across stages (GEPA_MAX_PROPOSER_COST_USD / stages), making it the whole-recipe proposer budget. A reader tuning per-stage spend will set the wrong number. Fix: change the meaning to 'Total GEPA model spend across all recipe stages' (default is MAX_OPTIMIZER_MODEL_COST_USD, not fixed 5).

🟡 LOW Math.max(1, floor(N/stages)) can exceed the declared evaluation limit at tiny N — examples/compare-optimization-methods/index.ts

GEPA_MAX_EVALUATIONS=1 with sequential yields 2 total stage evaluations (max(1, floor(1/2)) per stage); N=2 with omni yields 3. assertMatchedMethodLimits (index.ts:168-172) and the artifact's limits.candidateCaseEvaluations.gepa (index.ts:390) still report the lower declared value, so a matched comparison against skillopt=N silently gives GEPA up to 3x the budget. Requires an explicit tiny env value (default is 33), hence low. Fix: clamp the per-stage value without the max(1,...) floor, or reject GEPA_MAX_EVALUATIONS < stages for composed recipes.

🟡 LOW NOISE verdict described incompletely — also covers delta inside the band — examples/experiment-evidence/README.md

README says NOISE occurs 'when the sample is too unstable to judge' and index.ts:21 comment says 'iqr >= 10 is NOISE'. The tracker also returns NOISE for a STABLE sample whose medianDelta is inside [-regressionThreshold, +keepThreshold] (src/experiment-tracker.ts:257-261). A reader copying this example would expect a stable +8-delta-but-threshold-10 config to read ITERATE or something else, and get NOISE. Fix: extend the README sentence to 'NOISE when the sample is too unstable to judge or the median delta is inside the noise band'. Cosmetic; example data never exercises the path (verified by running it).

🟡 LOW Example is unguarded by CI — typecheck include is src-only — examples/experiment-evidence/index.ts

tsconfig.json include is ["src"] and vitest does not execute examples, so nothing in CI would catch this example breaking on future ExperimentTracker API drift; tsx runs transpile-only. Pre-existing convention for all 26 example dirs (none import via package name; all use '../../src/...' relative imports), so not a regression introduced by this PR. Verified today by execution: runs clean, exit 0. Optional hardening for the repo later: a CI smoke step that tsx-runs each example.

🟡 LOW README names a non-existent export 'HeldOutGate' — examples/profile-matrix/README.md

README says result.records feeds directly into analyzeRuns(), HeldOutGate, and scorecards. The public export is the function heldOutGate (src/campaign/gates/heldout-gate.ts:44, exported via src/campaign/index.ts:146 as heldOutGate/HeldOutGateOptions); 'HeldOutGate' as written matches no exported symbol. Impact: cosmetic only — a reader grepping the name finds the right place anyway. Fix: write heldOutGate() to match the export, as the README already does for analyzeRuns() and runCampaign().

🟡 LOW 'walks each check with a promoting and a refused candidate' overstates the linked example — examples/selfimprove-quickstart/README.md

examples/held-out-gate/index.ts walks exactly three cases: one clean promoting win (case 1), a few-runs coverage refusal (case 2), and the overfit refusal (case 3). It does not demonstrate a paired-delta-threshold refusal, a complete-scores refusal, or the cost ceiling, and no single check is shown with both a promoting and a refused variant. The held-out-gate README itself says 'walks through three decisions'. Fix: 'walks a promoting candidate and two refused candidates through the gate's checks' or similar.

---

_{tangletools · 2026-08-18T20:55:23Z · trace}

[tangletools]

✅ Approved — 9 non-blocking findings — 6bc0a112

Full multi-shot audit completed 5/5 planned shots over 10 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-08-18T20:55:23Z · immutable trace}

[tangletools]

🟢 Value Audit — sound

Verdict	sound
Coverage	2 of 2 lenses (value, usefulness)
Concerns	2 (1 low, 1 weak-concern)
Heuristic	0.0s
Duplication	0.1s
Interrogation	78.0s (2 bridge agents)
Total	78.1s

💰 Value — sound

Adds two offline runnable examples (runProfileMatrix, ExperimentTracker) and a GEPA_RECIPE knob that makes the six documented GEPA recipe kinds runnable at equal budget — verified running and typechecking clean, faithful to the src APIs, no duplication found; ship.

• What it does: Three examples-surface changes, no src/ changes: (1) examples/profile-matrix/index.ts runs runProfileMatrix over two AgentProfiles × three scenarios with a deterministic offline dispatch, a judge, and integrity: 'off' (documented as offline-only); (2) examples/experiment-evidence/index.ts drives ExperimentTracker + fileExperimentStore through parent/candidate experiments with 3 reps each carrying
• Goals it achieves: Restore the front door's stated promise — 'Every row is a function you call. Each links to a runnable example' (README.md:93) — which the runProfileMatrix row broke by linking to a doc; give ExperimentTracker (landed in #622, 70597fa) a first runnable example; and make the composed GEPA recipes that docs/campaign-proposers.md:250-254 already documents actually executable from the existing optimize
• Assessment: Good on its merits. I verified both new examples run offline and produce byte-for-byte the output their READMEs document (experiment-evidence prints ITERATE/KEEP with the persisted store path; profile-matrix prints per-profile means over 6 records with the expected per-cell expectUsage warnings), and pnpm typecheck:examples is clean. The examples follow the established grain: same README shape (Wh
• Better / existing approach: none — this is the right approach. I searched for an existing example or preset to extend (grep'd examples/ and src/campaign/presets/ for runProfileMatrix/ExperimentTracker usage; the segmented-profile-matrix preset exists but has no example and covers the segmented variant, not the base primitive the front-door row names), and for overlap with the scorecard/evaluate-a-change/held-out-gate example
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound

Three offline-runnable examples plus a GEPA_RECIPE knob that make already-shipped capabilities (runProfileMatrix, ExperimentTracker, composed GEPA recipes) reachable from the repo's two real discovery surfaces; every API used exists, every example runs, and the pattern matches the established exampl

• Integration: Verified wired and reachable. All imported APIs exist: runProfileMatrix (src/campaign/presets/run-profile-matrix.ts:440), ExperimentTracker/fileExperimentStore (src/experiment-tracker.ts:307,356; exported at src/experiment/index.ts:53-55), GepaOptimizationRecipe with all six kinds (src/campaign/gepa-optimization-method.ts:100-138). Both new examples execute clean in this checkout (exit 0; profile-
• Fit with existing patterns: Follows the established grain precisely: examples//{index.ts,README.md} + a row in examples/README.md's task-based index, identical to held-out-gate, sealed-experiment, and verify-without-answer-key. No src/ or export-surface changes. The experiment-evidence example pins provenanceReader and now, mirroring the test seam pattern (src/experiment-tracker.test.ts:181); profile-matrix's integrity
• Real-world viability: Holds up beyond the happy path within its stated scope. Deterministic and offline by construction (fixed scores, pinned clock, tmpdir store). Error paths are fail-loud where they should be: invalid GEPA_RECIPE throws with the valid list; the tracker validates rep runId/evidence URIs (src/experiment-tracker.ts:406-414). Known edges are documented rather than hidden: the expectUsage warnings are cal
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: console debug added examples/experiment-evidence/index.ts

• console.log(

🎯 Usefulness Audit

🟡 docs/experiment.md points at source, not the new example [integration] ``

docs/experiment.md:95 lists ExperimentTracker (run ledger with KEEP/ITERATE/NOISE/REGRESSION) pointing only at src/experiment-tracker.ts. A one-line cross-link to examples/experiment-evidence/ would complete the doc-to-runnable loop the same way docs/campaign-proposers.md:200 links its example. Front door and example index already make it reachable, so this does not gate shipping.

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260818T210732Z}

[tangletools]

✅ No Blockers — 6bc0a112

Review health 100/100 · Reviewer score 54/100 · Confidence 85/100 · 9 findings (2 medium, 7 low)

glm: Correctness 54 · Security 54 · Testing 54 · Architecture 54

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 5/5 planned shots over 10 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM vote recipe breaks the matched evaluation-budget guarantee between methods — examples/compare-optimization-methods/index.ts

For GEPA_RECIPE=vote the library budget is sum(run maxEvaluations) + runs.length * selectionScenarioCount (src/campaign/gepa-optimization-config.ts:159-161). With defaults (GEPA_MAX_EVALUATIONS=33 from TRAIN=5/SELECTION=3, SELECTION_N=3), gepa-vote is allowed 16+16+2*3 = 38 evaluations while skillopt stays capped at SKILLOPT_MAX_EVALUATIONS=33. assertMatchedMethodLimits (index.ts:168-172) compares only the declared env values, so the guard passes and the comparison runs at unequal budget, contradicting README line 72 ('every recipe runs at the same total budget') and the GEPA_MAX_EVALUATIONS control note 'Must match SkillOpt when both run'. Second

🟠 MEDIUM Overfit-gap clause misattributes a HeldOutGate-only check to the quickstart's release decision — examples/selfimprove-quickstart/README.md

The sentence says the gate for this quickstart's release decision 'refuses a candidate whose search-to-holdout gap says it won the optimizer but lost the exam'. improve() wires defaultProductionGate (src/contract/self-improve.ts:684-687); its checks (dimension-regression, budget, red-team, reward-hacking, canary, paired-bootstrap heldout significance — src/campaign/gates/default-production-gate.ts:20-26) include no search-to-holdout gap check, and GateContext carries only holdout artifacts so the gate never sees search-split scores. That check lives solely in HeldOutGate (src/held-out-gate.ts:21-25), which this example does not use. A reader concludes the printed 'Release decision: ship' carries overfit protection it does not have. Fix: reword to attribute the gap check to the linked held-

🟡 LOW New row links to docs under a 'runnable example' preamble — README.md

The 'Which Front Door' section states 'Each links to a runnable example', but the new analyzeTraces() row links to docs/trace-analysis.md#answer-one-question instead of an examples/ directory. Four pre-existing rows (runAnalystBenchmark, deltaRepair, replayVerify, analyzeSupervisorRun) already do the same, so this diff follows existing convention rather than introducing the inconsistency. Fix if desired: soften the preamble to 'Each links to a runnable example or doc', or add an examples/ask-one-question/ directory. Cosmetic only; no broken link.

🟡 LOW GEPA_RECIPE missing from the README Controls table — examples/compare-optimization-methods/README.md

The new GEPA_RECIPE variable is documented only in the 'Choose a GEPA recipe' section; the Controls table (lines 115-139) lists every other environment variable including all GEPA_* controls. A reader scanning the table will miss the recipe selector. Fix: add a row 'GEPA_RECIPE | engine | GEPA composition recipe: engine|sequential|adaptive-sequential|best-of|vote|omni.'

🟡 LOW comparison.json does not record GEPA_RECIPE or the recipe-effective evaluation limit — examples/compare-optimization-methods/index.ts

limits.candidateCaseEvaluations.gepa always reports GEPA_MAX_EVALUATIONS regardless of recipe, and no field records GEPA_RECIPE. Two runs differing only in GEPA_RECIPE (e.g. engine vs vote, with effective budgets 33 vs 38) produce identical limits blocks in the front-door artifact; the recipe snapshot exists only inside the gepa run directory (src/campaign/gepa-optimization-method.ts:270) reachable via provenance.artifactDir. For a package whose charter is registered-rule = executed-rule, the headline artifact should name the recipe kind and the effective limit it ran under. Fix: add gepaRecipe (kind) and the gepaRecipeEvaluationLimit value to the limits block.

🟡 LOW Example has no CI typecheck coverage — examples/profile-matrix/index.ts

tsconfig.json include is ["src"] only, so pnpm typecheck never touches examples/. The file compiles clean under strict mode today (verified with tsc --ignoreConfig --noEmit --strict), but a future API rename in src/ would break it silently while the README's run command fails. Repo-wide pattern for examples, not introduced by this PR. Fix optionally: add an examples tsconfig or a smoke test that imports each example.

🟡 LOW execSync git call fails outside a git checkout — examples/profile-matrix/index.ts

execSync('git rev-parse --short HEAD') throws (stack trace via the main().catch, exit 1) when the example runs outside a git repo, e.g. from a published tarball. Fails loud per repo doctrine and the example lives in-repo, so impact is minimal. No change required; a one-line comment or a friendlier error would polish it.

🟡 LOW 'walks each check with a promoting and a refused candidate' overstates the linked example — examples/selfimprove-quickstart/README.md

examples/held-out-gate/index.ts:72-130 walks exactly three fixed cases: a clean win that promotes, a few_runs coverage rejection, and an overfit refusal; its own README (examples/held-out-gate/README.md:20-25) says 'The example walks through three decisions'. No check is exercised with both a promoting and a refused candidate, several rejection codes (incomplete_coverage, missing_split_scores, indeterminate_delta, cost_ceiling) are never triggered, and there is no byte-identical/no-op candidate case matching the previous sentence. Fix: 'walks a promoting candidate and two refusals (coverage, overfit)'.

🟡 LOW Byte-identical hold attributed to 'the gate'; it is the loop's no-op guard — examples/selfimprove-quickstart/README.md

'The gate holds a candidate that is byte-identical to the baseline' — the forced 'hold' on a byte-identical winner comes from the no-op guard in runImprovementLoop (src/campaign/presets/run-improvement-loop.ts:147,304-312, contributing gate 'no-op-guard') before opts.gate.decide is ever called. Outcome is accurate; attribution is loose. Fix: 'the loop holds a byte-identical winner before the gate runs'.

---

_{tangletools · 2026-08-18T21:16:58Z · trace}