feat(experiment): the evidence registry — one home for measured claims

CHANGELOG.md

@@ -6,6 +6,10 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ## [Unreleased]
 
+### Added
+
+- The evidence registry: one canonical home for measured claims. `evidence/records/*.json` hold typed records (claim, instrument, exact command, arms, denominator, result, evidence state `CERTIFIED`/`MEASURED-ONCE`/`RESOLVED-NULL`/`UNVERIFIED`/`KILLED`, artifacts, cost, confounds, optional experiment seal digest), validated by `evidenceRecordSchema` from `./experiment`. `pnpm run evidence:render` generates `evidence/INDEX.md` from the records; `pnpm run evidence:check` (inside `verify:package`) fails on an invalid record or a stale index, so the human index can never drift from the data. Initial migration: seven records spanning trace-repair (gated-stop confirm, free-lunch), multishot (golden oracle v1), trace-analysis (GEPA-certified prompt, prime-vs-dspy), creative-cad, and vertical-bench (parity no-flip).
+
 ---
 
 ## [0.149.0] — 2026-08-18

README.md

@@ -12,6 +12,8 @@ Model calls happen only through the clients and agents you configure.
 
 New to the package? Read [concepts](./docs/concepts.md) first — it takes five minutes and defines every word used here.
 
+Looking for a measured result (a lift, a null, a parity verdict)? The canonical registry is [`evidence/`](./evidence/README.md) — machine-readable records, a generated index, and a freshness gate.
+
 ## Install
 
 ```sh

docs/experiment.md

@@ -2,6 +2,8 @@
 
 `@tangle-network/agent-eval/experiment` turns an experiment's registration into the object that runs it.
 
+The registry of measured claims those experiments produce lives in [`evidence/`](../evidence/README.md); a sealed experiment's digest is the `experimentDigest` its registry record carries.
+
 ## The covenant
 
 1. **The registered rule is the executed rule.**

evidence/README.md

@@ -0,0 +1,37 @@
+# Where evidence lives
+
+One home, one shape, one index.
+This directory is the canonical registry for measured claims about our agents, prompts, policies, and instruments — the numbers that used to rot in per-repo docs, `.evolve/` diaries, results directories, PR comments, and gists.
+
+## The rule
+
+- **A measured claim that anyone will cite later gets a record here.**
+  One JSON file per claim in [`records/`](./records/), validated by the exported schema (`evidenceRegistryRecordSchema` from `@tangle-network/agent-eval/experiment`).
+- **The registry lives in agent-eval** because the measurement substrate owns evidence legitimacy.
+  Every other repo keeps at most a pointer file (`docs/EVIDENCE.md`, three lines: "measured results for this repo live in agent-eval `evidence/`; do not restate numbers here").
+- **Humans read [`INDEX.md`](./INDEX.md); machines read `records/*.json`.**
+  The index is generated (`pnpm run evidence:render`) and drift-checked (`pnpm run evidence:check`, inside `verify:package`), so prose can never drift from data.
+- **Session diaries stay diaries.** `.evolve/experiments.jsonl` remains the append-only lab notebook.
+  A registry record is the distilled, addressable claim a notebook line earned.
+
+## The record
+
+Each record states, in typed fields: the one-sentence **claim** (written so it can fail), the **instrument**, the exact **command** (`null` = not preserved, a named gap), the **arms**, the denominator **n**, the **result** with uncertainty, **artifacts** (run dirs, PRs, gists — never empty), **cost** (`null` = not captured, never a silent zero), **confounds** (stated before anyone reads the verdict), and an **evidence state**:
+
+| state | meaning |
+| --- | --- |
+| `CERTIFIED` | pre-registered rule + sealed/held-out data, defended or re-run in the suite |
+| `MEASURED-ONCE` | one honest measurement on the real path, not replicated |
+| `RESOLVED-NULL` | adequate instrument, effect did not appear under the registered rule |
+| `UNVERIFIED` | stated somewhere load-bearing, no independent check yet |
+| `KILLED` | refuted, invalidated, or superseded |
+
+When a run was governed by a sealed experiment (`@tangle-network/agent-eval/experiment`), the record carries the seal as `experimentDigest` — the registered rule is the addressing scheme.
+
+## How to add a record
+
+1. Write `records/<id>.json` (`id` = kebab-case filename).
+2. `pnpm run evidence:render` — validates and regenerates `INDEX.md`.
+3. Commit both. A stale index or invalid record fails `verify:package`.
+
+A result that changes state (replicated, refuted, superseded) gets its state moved or a new record with `supersedes` — never a silent edit of the numbers.

evidence/records/cad-gepa-openscad-directive.json

@@ -0,0 +1,26 @@
+{
+  "id": "cad-gepa-openscad-directive",
+  "date": "2026-06-08",
+  "claim": "GEPA optimization of the OpenSCAD system directive lifts the CAD agent's held-out build quality over the hand-written directive.",
+  "domain": "creative-cad",
+  "instrument": "creative-agent CAD self-improvement loop: GEPA over the OpenSCAD directive, held-out scenario split, produced-state scoring of compiled models",
+  "command": null,
+  "arms": ["hand-written-directive", "gepa-optimized-directive"],
+  "n": {
+    "value": 1,
+    "unit": "held-out split",
+    "detail": "single held-out evaluation of the winning directive; split size recorded in the creative-agent run artifacts"
+  },
+  "result": "+9.5pp held-out over the baseline directive; the winning directive shipped with automatic video capture of the built models.",
+  "evidenceState": "MEASURED-ONCE",
+  "artifacts": [
+    "creative-agent: .evolve/evals/ and the CAD self-improve loop run artifacts (repo tangle-network/creative-agent)"
+  ],
+  "costUsd": null,
+  "confounds": [
+    "single held-out evaluation, no replication or challenger round",
+    "exact invocation and split size were not preserved in a durable record - this registry entry is the pointer that stops the number from degrading into folklore; the next CAD run should re-derive and upgrade or kill this record"
+  ],
+  "sourceRepo": "creative-agent",
+  "notes": "Weakest record in the initial migration by design: it demonstrates the UNVERIFIED->MEASURED-ONCE boundary. The +9.5pp number traces to the CAD loop session record, but the run directory is not pinned here yet."
+}

evidence/records/ctb-analyst-gepa-certified-prompt.json

@@ -0,0 +1,29 @@
+{
+  "id": "ctb-analyst-gepa-certified-prompt",
+  "date": "2026-08-02",
+  "claim": "The GEPA-optimized DSPy-RLM analyst prompt (sha d3829fb8) is the certified shipping prompt on CodeTraceBench: challengers lose on pooled sealed splits under the pre-registered rule.",
+  "domain": "trace-analysis",
+  "instrument": "sealed certification runs: pre-registration written before the winner was known, pooled OH-cert32 + T2-cert32 sealed splits, scored micro-F1 against human step labels, paired CIs",
+  "command": "gepa-analyst-campaign certification phase: 4 serial runs, 2 reps, mutex-held seat (see .evolve/certification-2026-08-02-preregistration.md for the registered rule)",
+  "arms": ["stock-d3829fb8", "g2-challenger-1bb303e7"],
+  "n": {
+    "value": 64,
+    "unit": "sealed cases",
+    "detail": "OH-cert32 + T2-cert32, 2 reps (128 case-runs; G2 failed 3/128, stock 0/128); both cert32 splits spent for this config family"
+  },
+  "result": "G2 REJECTED by rule 1: pooled-cert micro-F1 stock 0.2489 vs G2 0.1928 (-5.6pp). Per-family stock OH 0.2896 / T2 0.2162. Paired CIs [-0.126, 0.058] OH, [-0.143, 0.023] T2. G2's selection-split edge (+0.064 weighted) did not transfer - third consecutive tuning-split-vs-fresh-split reversal in the campaign. The shipping prompt stands, defended twice (round-1 and round-2 challengers).",
+  "evidenceState": "CERTIFIED",
+  "artifacts": [
+    "~/bench-cache/ctb-20260801/cert2/ (4 result.json + logs + winner-instructions.txt)",
+    ".evolve/certification-2026-08-02-preregistration.md",
+    ".evolve/experiments.jsonl (arm cert2-stock-vs-g2)"
+  ],
+  "costUsd": 31.07,
+  "confounds": [
+    "cross-family gap is the dominant residual: shipping analyst OH 0.29 / T2 0.22 vs mini-SWE wide 0.56-0.63 (~2x), later diagnosed as localization (far-class mass), not prompt quality",
+    "gold-mass-weighted selection metric recorded as overfit-prone at n=12",
+    "CodeTraceBench frozen as an optimization target on 2026-08-04: ~60% of residual misses are missing labels (H1, n=20 adjudication), so absolute F1 under-reads the analyst"
+  ],
+  "sourceRepo": "agent-eval",
+  "notes": "Companion baseline: dev-pool scored-F1 baseline for the dspy-rlm analyst stack is 32.73%/30.99% (retired direct runner excluded). Prime-agent comparison against these stored baselines is recorded separately (prime-vs-dspy-analyst-38-rows)."
+}

evidence/records/multishot-golden-oracle-v1.json

@@ -0,0 +1,28 @@
+{
+  "id": "multishot-golden-oracle-v1",
+  "date": "2026-08-16",
+  "claim": "The multishot loop's behaviour is frozen as deterministic golden records that any engine can be verified against, and the records are load-bearing: they detect every field-level and engine-level deviation.",
+  "domain": "multishot",
+  "instrument": "golden-record oracle: scripted transports and executors, request ledger + outcome per scenario, double capture with mandatory agreement, exhaustive single-field mutation and engine-mutant detection in the suite",
+  "command": "pnpm vitest run src/multishot/golden/golden.test.ts",
+  "arms": ["recorded-loop-v1", "candidate-engine"],
+  "n": {
+    "value": 14,
+    "unit": "records",
+    "detail": "13 shot scenarios + 1 matrix scenario, each captured twice with agreement required; v1 captured from the loop at 0.145.21"
+  },
+  "result": "Consumer parity proven before freezing: tax 4/4 deep-equal, gtm 4/4 shot-level and 8/8 matrix-level. Oracle load-bearing: every single leaf of {outcome, requests} (and {matrix, requests, judgeRequests, files}) mutated one at a time is reported for all 14 records; 8/8 engine mutants detected, each naming the deviating field.",
+  "evidenceState": "CERTIFIED",
+  "artifacts": [
+    "src/multishot/golden/records/v1.json",
+    "src/multishot/golden/golden.test.ts",
+    "docs/multishot-golden-records.md",
+    "https://github.com/tangle-network/agent-eval/pull/627"
+  ],
+  "costUsd": 0,
+  "confounds": [
+    "deterministic-only coverage: scripted transports, fixed personas and budgets; live-provider behaviour (latency, substitution, SSE faults) is out of the oracle's scope by design"
+  ],
+  "sourceRepo": "agent-eval",
+  "notes": "CERTIFIED because the check re-runs on every suite run and records are frozen: the recorder refuses to overwrite a version, so a behaviour change mints a new version file and the diff is the reviewable evidence. Later consumer parity counts quoted as 14/14 and 15/15 live in the loop-retention PR chain (#617 and its consumers) - PR-comment-only until migrated here."
+}

evidence/records/prime-vs-dspy-analyst-38-rows.json

@@ -0,0 +1,28 @@
+{
+  "id": "prime-vs-dspy-analyst-38-rows",
+  "date": "2026-08-08",
+  "claim": "Prime-agent as a one-shot trace analyst beats the GEPA-certified DSPy-RLM analyst on stored-baseline CodeTraceBench rows.",
+  "domain": "trace-analysis",
+  "instrument": "batch comparison over stored dspy baselines: prime-RLM one-shot with a bounded structured-output repair turn, scored per-case F1 against human step labels",
+  "command": "prime-vs-dspy batch over 38 positive-labeled OH+T2 dev rows via cli-bridge prime backend (systemd-run scoped spawn, nix-built prime-agent 0.7.0)",
+  "arms": ["prime-agent-0.7.0", "dspy-rlm-stored-baseline"],
+  "n": {
+    "value": 38,
+    "unit": "rows",
+    "detail": "positive-labeled OH+T2 dev rows; 35/38 completed ok, 5 needed the repair turn"
+  },
+  "result": "Prime wins the dev-pool comparison: mean per-case scored F1 0.3218 vs 0.1833; 18W/4L/16T (all ties 0-0); two-sided sign test p=0.0043. Direction consistent across both baseline sources including the derivation-verified stock-smoke slice (OH 0.377 vs 0.231, T2 0.416 vs 0.163).",
+  "evidenceState": "MEASURED-ONCE",
+  "artifacts": [
+    "https://gist.github.com/drewstone/cd6ee0f0ca659334215001d792d0dacf",
+    ".evolve/experiments.jsonl (arm prime-38-row-comparison)"
+  ],
+  "costUsd": 5.3,
+  "confounds": [
+    "compared against STORED dspy baselines, not a same-day paired rerun; provider state differed between the arms' measurement dates",
+    "cost asymmetry ~4.7x dspy (220K input tokens per case mean), so the win is not cost-normalized",
+    "dev-pool rows, not the sealed cert splits; 3/38 prime rows failed structured output"
+  ],
+  "sourceRepo": "traces",
+  "notes": "Until this registry, the durable copy of this result was a secret gist plus an uncommitted .evolve line in a working tree."
+}

evidence/records/tb-gated-stop-ab-confirm.json

@@ -0,0 +1,31 @@
+{
+  "id": "tb-gated-stop-ab-confirm",
+  "date": "2026-08-15",
+  "claim": "At one matched total token budget on TB-Repair, an agent that may stop only after an executable held-out check passes finishes more rows than the same agent on unconditional continuation.",
+  "domain": "trace-repair",
+  "instrument": "tb-gated-stop-ab confirmatory draw: sealed registered experiment, servedModel abort gate, paired cluster-bootstrap contrast over best intermediate state",
+  "command": "pnpm tsx scripts/tb-gated-stop-ab.ts (design/screen/pilot/confirm phases; seal e13a4f8d re-executed by the confirm runner)",
+  "arms": ["blind-continue", "gated-continue"],
+  "n": {
+    "value": 151,
+    "unit": "rows",
+    "detail": "22 clusters; settlingDraw at power 0.8009 (search sim) / 0.8013 (registered sim) for effect 0.10; admitted ceiling 216 rows at power 0.9189"
+  },
+  "result": "+0.0596 pass-rate delta (best intermediate state in both arms), 95% cluster-bootstrap CI [-0.0061, +0.1210]. The interval crosses zero: direction positive, not significant at the registered level.",
+  "evidenceState": "MEASURED-ONCE",
+  "artifacts": [
+    "docs/trace-repair-gated-stop.md",
+    "benchmarks/trace-repair/gated-stop-ab/design.json",
+    "~/bench-cache/gated-stop-ab/ (design.json, pilot.json, confirm-runs.json, confirm-refusal.json, finalize.out)",
+    "https://github.com/tangle-network/agent-eval/pull/601"
+  ],
+  "costUsd": null,
+  "confounds": [
+    "95% CI crosses zero; the draw was powered for effect 0.10 and the observed effect is 0.06",
+    "provider retired the pinned glm-5.2 mid-campaign; the servedModel gate aborted the stale seal at $0 and the pin moved to glm-5.3 (seal ad82d342 -> e13a4f8d), so screen-era and confirm-era rows saw different provider states",
+    "confirm run absorbed HTTP 429 rate-limit interruptions on several rows (retried; logged in confirm-run.out)"
+  ],
+  "sourceRepo": "agent-eval",
+  "experimentDigest": "e13a4f8d9ff437dbf7d24898d64cba55c90c682c3902ad30bb528ca241939626",
+  "notes": "Aggregate router spend was not captured as one number for the confirm draw; per-row token and grade costs are in confirm-runs.json."
+}

evidence/records/tb-repair-free-lunch.json

@@ -0,0 +1,29 @@
+{
+  "id": "tb-repair-free-lunch",
+  "date": "2026-08-10",
+  "claim": "Unconditional continuation from the recorded end state with a real model budget (no analyst, no hint, no gate) rescues failed TB-Repair rows, so admission condition 3 has a free lunch to subtract.",
+  "domain": "trace-repair",
+  "instrument": "tb-repair-freelunch runner: pinned mini-swe-agent continuation policy, network none inside the continuation container, deterministic oracle grading, servedModel check per pass",
+  "command": "bash ~/bench-cache/freelunch-20260810/run3.sh -> tsx scripts/tb-repair-freelunch.ts (policy tb-repair-continuation-v1, glm-5.2 pinned, seed 20260810, $10 ceiling)",
+  "arms": ["zero-call-control", "budgeted-continuation"],
+  "n": {
+    "value": 64,
+    "unit": "rollouts",
+    "detail": "32 prefix-faithful rows x uniform n=2; denominator chain 48 evaluated -> 43 deterministic oracle -> 35 clean exit -> 32 prefix-fidelity ok"
+  },
+  "result": "3 of 64 rollouts rescued (4.7%); 2 of 32 rows rescued at least once (6.2%). Zero-call control: 0 passes in 96 rollouts. Exit statuses: 15 submitted, 46 step-budget-exhausted, 3 model-error.",
+  "evidenceState": "MEASURED-ONCE",
+  "artifacts": [
+    "docs/trace-repair-free-lunch.md",
+    "~/bench-cache/freelunch-20260810/ (freelunch-pass1.json, freelunch-pass2.json, rollouts.jsonl, PREREG.md, status.txt)",
+    "https://github.com/tangle-network/agent-eval/pull/558"
+  ],
+  "costUsd": 9.64,
+  "confounds": [
+    "one full pass was served a substituted model (deepseek-v3.2 request answered as deepseek-v4-flash) and was aborted pre-count; only glm-5.2-served rollouts are in the denominator",
+    "uniform n stopped at 2 because a third pass would cross the $10 ceiling",
+    "3 prefix-divergent rows quarantined to a secondary set and excluded from the primary denominator"
+  ],
+  "sourceRepo": "agent-eval",
+  "notes": "The free-lunch rate is the floor any repair intervention must beat; it is subtracted, not celebrated."
+}

evidence/records/vb-cell-engine-parity-no-flip.json

@@ -0,0 +1,29 @@
+{
+  "id": "vb-cell-engine-parity-no-flip",
+  "date": "2026-08-15",
+  "claim": "The VerticalBench graph cell-engine reaches behavioural parity with the loop cell-engine on a live cell, so the default engine can flip to the graph.",
+  "domain": "vertical-bench",
+  "instrument": "instrumented live parity cell: tempo-payments tempo-tip20-balance-viewer, opencode glm-5.3 (router-served id verified), shots=2, verify=full, judges deepseek-v4-flash, identical env across arms",
+  "command": "blueprint-agent experiments harness, parity phase 3 of tangle-network/agent-runtime#694 (loop arm + four instrumented graph attempts)",
+  "arms": ["loop-cell-engine", "graph-cell-engine"],
+  "n": {
+    "value": 1,
+    "unit": "leaf",
+    "detail": "1 rep per arm; 4 instrumented graph cycles, each archived with its defect"
+  },
+  "result": "NO FLIP. Parity holds on dispatched shot-1 prompt (byte-identical), realness-firewall activation, verification score on the same graded state (blended 0.31 both arms), and toolchain detection. Parity fails on coder turn depth (graph settles recon-only in ~3-4m vs the loop working the full 30m wall; opencode SSE errored 2/4 cells), fixed-budget adherence (structural in the loop vs probabilistic in the graph: 1/3 post-contract cells dispatched shot 2), and board classification (loop 2/2 fair attempts, graph 0/2 - quarantined, never silently scored).",
+  "evidenceState": "RESOLVED-NULL",
+  "artifacts": [
+    "blueprint-agent: scripts/experiments/results/p3-parity-20260815/ (loop + four graph attempts, each archived with its defect)",
+    "https://github.com/tangle-network/blueprint-agent/pull/2330"
+  ],
+  "costUsd": null,
+  "confounds": [
+    "n=1 leaf, 1 rep per arm; stochastic model trajectories",
+    "arms ran hours apart on a live provider",
+    "the loop's cells were wall-capped at 30m, so its 0.31 grades a half-build",
+    "glm-5.3 had no rate in SANDBOX_MODEL_COSTS/cost-estimator at run time, so spend did not accumulate against caps (left loud, not guessed)"
+  ],
+  "sourceRepo": "blueprint-agent",
+  "notes": "Two structural gaps block the flip and are named in the engine header: TODO(graph-cell/bridge-turn-depth) and TODO(graph-cell/fixed-budget-adherence). The flip is one line once both close. Until this registry, these numbers lived only in the PR body."
+}

package.json

@@ -180,7 +180,9 @@
     "check:model-ids": "node scripts/check-model-id-requests.mjs",
     "check:analyst-benchmark": "node scripts/check-analyst-benchmark-implementation.mjs",
     "openapi": "node dist/cli.js openapi --out dist/openapi.json",
-    "verify:package": "pnpm check:analyst-benchmark && pnpm run check:skill && pnpm run check:model-ids && publint && attw --pack --profile esm-only . && node scripts/verify-package-exports.mjs"
+    "verify:package": "pnpm check:analyst-benchmark && pnpm run check:skill && pnpm run check:model-ids && publint && attw --pack --profile esm-only . && node scripts/verify-package-exports.mjs && pnpm run evidence:check",
+    "evidence:render": "tsx scripts/render-evidence-index.ts",
+    "evidence:check": "tsx scripts/render-evidence-index.ts --check"
   },
   "dependencies": {
     "@asteasolutions/zod-to-openapi": "^9.1.0",