fabbrik · Oct 31, 2026 · Oct 31, 2026 · Oct 31, 2026
diff --git a/.gitattributes b/.gitattributes
@@ -1,2 +1,3 @@
 *.sql text eol=lf
 contract/** text eol=lf
+dogfood/** text eol=lf
diff --git a/.gitignore b/.gitignore
@@ -47,3 +47,6 @@ artifacts/
 # The CI-triage tool (experiments/AgentExperience.CiTriage): fetched CI logs, and what --scripted writes
 experiments/AgentExperience.CiTriage/.cache/
 experiments/AgentExperience.CiTriage/results/scripted/
+
+# aexp: the local lesson store, open runs, sessions, hook log and the copied config (dogfood/README.md)
+.aexp/
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -612,6 +612,26 @@ AgentExperience.NET is a **preview**. It is not production ready, and public API
   the file keeps its mode (and, when another user owns it, its owner and group, by an in-place rewrite).
 - **Library change**: none. Guide: [Claude Code](docs/guide/claude-code.md).
 
+### Dogfood on this repository (story 28.10)
+
+- **`aexp benefit --markdown <file>`** writes the holdout's estimate as a results page: counts per arm, rates, the
+  difference with its 95% interval, the verdict, the pending, unverified and mixed-arm runs, the reading rules, and what
+  the page can and cannot say (among them that runs of one Claude Code session are correlated and the store cannot count
+  sessions). No path or run ID, only the date, so it can be committed. Inside the project root only, never through a
+  link. `aexp benefit` without it prints what it did before.
+- **[`dogfood/`](dogfood/README.md)**: everything to run the Claude Code recipe on this repository's own sessions, up to
+  switching it on, which stays a person's step. A config (`dogfood/aexp.config.json`: SQLite under `.aexp/`, holdout
+  0.5, recall on, only file paths kept, no recipe consents, each choice commented), the verifier (`dogfood/verify.sh`:
+  `dotnet build`, then every Docker-free test project from CONTRIBUTING.md, kept equal to that list by a test), and a
+  pre-registration written before any session ran (`dogfood/preregistration.json`: the session as the randomized unit,
+  the run as the analysed one, 20 verified runs per arm before the one reading, a stated second look at 50, a deadline,
+  exclusions and limits, and the SHA-256 of both files, which a test checks). `.gitignore` now excludes `.aexp/`.
+- **README**: leads with what the library is for, the block a second run gets after a first run failed and recovered,
+  the evidence so far (now with the real-task and CI-triage results: no overall benefit shown; lesson content beat a
+  placebo, p = 0.035; wrong lessons did worse than none; CI triage `NoEvidence` for lack of runs) and the three ways in.
+  Detail moved down; a "Real work" section says the dogfood run has no result yet.
+- **Library change**: none.
+
 ## 0.1.0-preview.17
 
 This preview measures whether injected lessons help. An opt-in holdout withholds the block on a deterministic share of

diff --git a/README.md b/README.md
@@ -2,17 +2,41 @@
 
 [![CI](https://github.com/fabbrik/AgentExperience.NET/actions/workflows/ci.yml/badge.svg)](https://github.com/fabbrik/AgentExperience.NET/actions/workflows/ci.yml) [![NuGet](https://img.shields.io/nuget/vpre/AgentExperience.Core.svg?label=nuget%20(preview))](https://www.nuget.org/packages/AgentExperience.Core) [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](./LICENSE)
 
-**Memory of what *worked*, verified by your own checks, for Microsoft Agent Framework agents.**
-
-When your own code says a run worked (a test run, an exit code, a person's sign-off; never the model's word), the
-library keeps what the agent tried as a lesson and hands it to the next run on a similar task. A preview, not
+**Memory of what verifiably worked, for AI agents, measured by its own holdout.** When your own check says a run worked
+(a test run, an exit code, a person's sign-off; never the model's word), the library keeps what the agent tried as a lesson
+for the next run on a similar task, and withholds it from a share of runs to measure whether it helps. A preview, not
 production ready. How it differs from MAF's own memory: [Compared with MAF memory](docs/compared-with-maf-memory.md).
 
+## What an agent sees
+
+Run 1 called `run_tests`: exit 3, then a pass, which your check verified. Run 2, on a similar ticket, starts with this
+block (shortened; [the whole block](#the-whole-block) is from another sample):
+
+```text
+=== BEGIN HISTORICAL REFERENCE (UNTRUSTED REFERENCE MATERIAL) ===
+…
+Lesson: Verified after 1 attempt. Worked: attempt 0. Checks: [run_tests].
+Tried:
+  - attempt 0: run_tests [failed: exit 3], run_tests [returned] → completed
+…
+=== END HISTORICAL REFERENCE ===
+```
+
+## Does it help?
+
+On one synthetic task, yes, for both models tried. On realistic tasks no overall benefit is shown yet, and wrong lessons did worse than none ([the evidence so far](#the-evidence-so-far)). Mean failed attempts per task in a [live experiment](experiments/AgentExperience.LiveReuse/README.md):
+
+| Model | No memory | Memory | Placebo | Stale |
+| --- | ---: | ---: | ---: | ---: |
+| Gemini `gemini-3.1-flash-lite` ([report](experiments/AgentExperience.LiveReuse/results/gemini-gemini-3.1-flash-lite-2026-09-27.md); pre-registered, confirmatory) | 2.17 | **0.00** | 2.42 | 2.75 |
+| Claude `claude-haiku-4-5` ([report](experiments/AgentExperience.LiveReuse/results/anthropic-claude-haiku-4-5-2026-09-28.md); exploratory replication) | 2.75 | **0.83** | 2.42 | 2.67 |
+
 ## Quick start
 
-Needs `0.1.0-preview.11` or later. Add `AgentExperience.MicrosoftAgentFramework`, `AgentExperience.Storage.InMemory`
-(`--prerelease`) and `Microsoft.Extensions.DependencyInjection`; bring `chatClient` (any `IChatClient`) and
-`RunTestsAsync` (your test run, returning text such as `exit=0`). Development and tests only: see [In production](#in-production).
+Three ways in: a **Microsoft Agent Framework** agent (below), **any agent** through the `aexp` CLI or MCP server
+([CLI and MCP](#any-agent-cli-and-mcp)), or **Claude Code** through hooks ([Claude Code](#claude-code)).
+
+For MAF (`0.1.0-preview.11` or later), add `AgentExperience.MicrosoftAgentFramework`, `AgentExperience.Storage.InMemory` (`--prerelease`) and `Microsoft.Extensions.DependencyInjection`; bring `chatClient` (any `IChatClient`) and `RunTestsAsync` (your test run, returning text such as `exit=0`). Development and tests only: see [In production](#in-production).
 
 ```csharp
 using AgentExperience.MicrosoftAgentFramework;
@@ -35,37 +59,34 @@ await agent.RunAsync("Ticket #4812: a refund is stuck on a lock. Triage it.");
 await agent.RunAsync("Ticket #4977: a refund is stuck on a lock again. Triage it.");  // gets the lesson
 ```
 
-## What an agent sees
+## Any agent: CLI and MCP
 
-The quick start's second run gets this block (shortened; [the whole block](#the-whole-block) is from another sample):
+Agents outside the Agent Framework (Claude Code, Codex, Cursor, a script) get the same loop through `aexp`, a .NET tool
+that is both a command line and an MCP stdio server, over a local SQLite file by default (published from the next
+preview, `0.1.0-preview.18`; until then, [build it from this repository](dogfood/README.md)):
 
-```text
-=== BEGIN HISTORICAL REFERENCE (UNTRUSTED REFERENCE MATERIAL) ===
-…
-Lesson: Verified after 1 attempt. Worked: attempt 0. Checks: [run_tests].
-Tried:
-  - attempt 0: run_tests [failed: exit 3], run_tests [returned] → completed
-…
-=== END HISTORICAL REFERENCE ===
+```bash
+dotnet tool install --global AgentExperience.Cli --prerelease
+claude mcp add aexp -- aexp mcp      # or any MCP client: the server is `aexp mcp`, run in the project
 ```
 
-## Does it help?
-
-On one synthetic task, for the two models tried, yes. Mean failed attempts per task in a [live experiment](experiments/AgentExperience.LiveReuse/README.md):
+A `.aexp/config.json` in the project names the scope and the one command `aexp` may run, the verifier
+(`"verify": { "command": ["dotnet", "test"] }`, exit code 0 = pass), which runs only once a person has reviewed the
+config and approved it with `aexp trust` (an edit revokes the approval). The MCP tools `experience_recall`,
+`experience_failure_hint`, `experience_record_call`, `experience_finish` and `experience_lessons` (or the commands
+`aexp run start|call|end`, `aexp recall`, `aexp finalize`, `aexp lessons`) capture a run, verify it with that command,
+and hand the next run the lesson and, after a failure, the hint, through the same writer, rules and holdout as the
+adapter. No tool takes identity, scope, a path or a command, and no model is called. See
+[CLI and MCP](docs/guide/cli-and-mcp.md).
 
-| Model | No memory | Memory | Placebo | Stale |
-| --- | ---: | ---: | ---: | ---: |
-| Gemini `gemini-3.1-flash-lite` ([report](experiments/AgentExperience.LiveReuse/results/gemini-gemini-3.1-flash-lite-2026-09-27.md); pre-registered, confirmatory) | 2.17 | **0.00** | 2.42 | 2.75 |
-| Claude `claude-haiku-4-5` ([report](experiments/AgentExperience.LiveReuse/results/anthropic-claude-haiku-4-5-2026-09-28.md); exploratory replication) | 2.75 | **0.83** | 2.42 | 2.67 |
+## Claude Code
 
-The experiment (opt-in, real models) gives the model a tool with one hidden working strategy, on 12 task instances, and
-compares memory against no memory, a placebo block with the strategy withheld, and stale experience.
+A recipe with no code: `aexp setup claude-code` adds hooks that record every tool call, check
+each turn that changed something with your tests when it stops, and hand the next session the lesson and the failure
+hint; `aexp export --claude-md` keeps the lessons measured as helping in a managed section of `CLAUDE.md`. See
+[Claude Code](docs/guide/claude-code.md).
 
-Read it with its limits: one run per model, one synthetic task family, 12 instances, and the working strategy reaches
-the block verbatim, so it shows that a model acts on an injected lesson, not that the library helps on real tasks in
-general. A harder experiment, where the lesson has to transfer to a different system through the library's own
-retrieval, is [in progress, no live results yet](experiments/AgentExperience.LiveReuse/README.md#transfer-experiment).
-The samples and the reuse baseline use scripted models: they show the loop works, not that it helps.
+This repository's own sessions are set up the same way, to judge the library on real work: see [Real work](#real-work).
 
 ## Try it in a minute
 
@@ -89,6 +110,49 @@ The stand-in is a script that tries first whatever strategy an injected block na
 not that it helps a real model ([Does it help?](#does-it-help) is about that). Set `OPENAI_API_KEY` to run it against
 one; see [the sample's README](samples/AgentExperience.Sample.QuickStart/README.md).
 
+## The evidence so far
+
+Each result links its report, which says whether it was pre-registered. None yet is from real work.
+
+### A synthetic task
+
+The experiment behind the table [above](#does-it-help) (opt-in, real models) gives the model a tool with one hidden
+working strategy, on 12 task instances, and compares memory against no memory, a placebo block with the strategy
+withheld, and stale experience.
+
+Read it with its limits: one run per model, one synthetic task family, 12 instances, and the working strategy reaches
+the block verbatim, so it shows that a model acts on an injected lesson, not that the library helps on real tasks in
+general. The samples and the reuse baseline use scripted models: they show the loop works, not that it helps.
+
+### Realistic tasks
+
+In the [real-task experiment](experiments/AgentExperience.LiveReuse/README.md#real-task-experiment) a model fixes small
+services broken by a dependency upgrade, and the lessons reach it through the library's own retrieval. The only run that
+could be evaluated (confirmatory; lessons learned by Claude Haiku 4.5, used by gpt-oss-120b, 11 services,
+[report](experiments/AgentExperience.LiveReuse/results/realtask-openrouter-anthropic-claude-haiku-4.5-to-openai-gpt-oss-120b-2026-10-27-run4.md))
+shows **no overall benefit**: against no memory, the lessons saved tool calls on 5 services and cost them on 4
+(p = 0.50). What a lesson says did matter: against a placebo block with the fix file withheld, the real lessons used
+fewer tool calls on 7 of the 8 services that differed (p = 0.035). Lessons from other breakages did worse than no
+memory (0.303 of trials verified, against 0.455). Wrong lessons cost more than none, which is what the holdout is there
+to catch. Every other real-task run was inconclusive (too many services excluded) or stopped early (by the provider or a budget cap).
+
+In the [transfer experiment](experiments/AgentExperience.LiveReuse/README.md#transfer-experiment) a lesson has to help
+on a different system that shares one trait. Its one evaluated run (Claude Haiku 4.5) shows no demonstrated benefit
+(p = 0.0625), and the placebo and mismatched blocks did about as well as the real one, so even the direction cannot be
+credited to what the lessons said.
+
+### CI triage
+
+[The library on this repository's failed CI runs](experiments/AgentExperience.CiTriage/README.md), with its holdout
+judging ([results](experiments/AgentExperience.CiTriage/results/ci-triage-2026-10-29.md)): 1 of 7 triaged correctly
+with lessons, 3 of 9 without. The verdict is `NoEvidence` for lack of runs: each arm needs 20.
+
+### Real work
+
+[`dogfood/`](dogfood/README.md) runs the Claude Code recipe on the sessions that build this repository, checked by its
+own Docker-free tests, with lessons withheld from half the sessions. It is pre-registered and not switched on yet, so
+there is **no result yet**. This section will state the result whichever way it comes out.
+
 ## The whole block
 
 The excerpt above comes from the quick start. This whole block comes from the
@@ -200,30 +264,6 @@ working attempt also say what its call produced, as `[returned: "pinned lib to 2
 capture, sanitized, bounded and signed, off by default (`--recipe` on the quick start shows one; see
 [Showing what worked](docs/guide/injection.md#showing-what-worked-the-recipe)).
 
-## Any agent: CLI and MCP
-
-Agents outside the Agent Framework (Claude Code, Codex, Cursor, a script) get the same loop through `aexp`, a .NET tool
-that is both a command line and an MCP stdio server, over a local SQLite file by default:
-
-```bash
-dotnet tool install --global AgentExperience.Cli --prerelease
-claude mcp add aexp -- aexp mcp      # or any MCP client: the server is `aexp mcp`, run in the project
-```
-
-A `.aexp/config.json` in the project names the scope and the one command `aexp` may run, the verifier
-(`"verify": { "command": ["dotnet", "test"] }`, exit code 0 = pass), which runs only once a person has reviewed the
-config and approved it with `aexp trust` (an edit revokes the approval). The MCP tools `experience_recall`,
-`experience_failure_hint`, `experience_record_call`, `experience_finish` and `experience_lessons` (or the commands
-`aexp run start|call|end`, `aexp recall`, `aexp finalize`, `aexp lessons`) capture a run, verify it with that command,
-and hand the next run the lesson and, after a failure, the hint, through the same writer, rules and holdout as the
-adapter. No tool takes identity, scope, a path or a command, and no model is called. See
-[CLI and MCP](docs/guide/cli-and-mcp.md).
-
-For Claude Code there is a recipe with no code: `aexp setup claude-code` adds hooks that record every tool call, check
-each turn that changed something with your tests when it stops, and hand the next session the lesson and the failure
-hint; `aexp export --claude-md` keeps the lessons measured as helping in a managed section of `CLAUDE.md`. See
-[Claude Code](docs/guide/claude-code.md).
-
 ## Seeing what it learned
 
 After the quick start, `provider.GetAgentExperienceLessons().ListForDevelopmentIdentityAsync()` lists the lessons it

diff --git a/docs/guide/claude-code.md b/docs/guide/claude-code.md
@@ -180,7 +180,7 @@ default). Read it as evidence about your project's sessions, not a promise. The
 the turns of one session are correlated: they share the session's arm, its task and often its outcome. The interval
 treats them as independent, so it is narrower than the evidence supports; a session of many turns weighs as many. Look
 at how many sessions are behind the counts (`ls .aexp/sessions` shows the live ones; the hook log names each session's
-runs) before trusting a verdict reached with few sessions. `aexp benefit` cannot count sessions yet. `aexp lessons` shows each lesson's own verdict. See [the holdout](injection.md) for how the arms are assigned.
+runs) before trusting a verdict reached with few sessions. `aexp benefit` cannot count sessions yet. `aexp benefit --markdown <file>` writes the same estimate as a results page to commit, with the reading rules and these limits beside it; [`dogfood/`](../../dogfood/README.md) is this repository's own setup and pre-registration, as a worked example. `aexp lessons` shows each lesson's own verdict. See [the holdout](injection.md) for how the arms are assigned.
 
 ## Lessons in CLAUDE.md or AGENTS.md
 

diff --git a/docs/guide/cli-and-mcp.md b/docs/guide/cli-and-mcp.md
@@ -134,15 +134,15 @@ aexp finalize --run "$run"                                 # runs the verifier;
 | `aexp lessons [--all] [--html <file>]` | A table of the lessons (`--all` adds the ones retrieval would not use), or the [HTML view](deployment.md#seeing-what-it-learned). |
 | `aexp export --markdown <file>` | Exports the lessons the holdout measured as helping (the [measured export](injection.md)). |
 | `aexp export --claude-md [file]` / `--agents-md [file]` | Writes the same lessons into a managed section of `CLAUDE.md` / `AGENTS.md` (the project root's when no file is named), leaving the rest of the file byte for byte ([Claude Code](claude-code.md#lessons-in-claudemd-or-agentsmd)). |
-| `aexp benefit` | The holdout's estimate: runs given lessons against runs they were withheld from. |
+| `aexp benefit [--markdown <file>]` | The holdout's estimate: runs given lessons against runs they were withheld from. With `--markdown`, writes it as a results page instead: counts and rates per arm, the difference with its 95% interval, the verdict, the runs left out, the reading rules and what it can and cannot say (no path or run ID, so it can be committed). |
 | `aexp mcp` | Serves the MCP tools over stdin and stdout. |
 | `aexp trust` | Prints what the config would run, and approves the config (see above). |
 | `aexp runs prune --older-than <age>` | Deletes open runs (never ended) started longer ago than `7d`, `12h` or `30m`, unless a command is using them, and Claude Code session files not touched for as long. |
 | `aexp setup claude-code [--print]` | Adds the Claude Code hooks to `.claude/settings.json`, a default config and the `.gitignore` entry ([Claude Code](claude-code.md)). |
 | `aexp hook <event> [--timeout <s>]` | A Claude Code hook: reads the event's JSON on standard input, prints the hook output; always exits 0 ([Claude Code](claude-code.md)). |
 
 Exit codes: 0 done, 1 failed, 2 a usage error (including a run that is not open), 3 a configuration the tool refuses
-(including an untrusted one). `lessons --html` and `export --markdown` write inside the project root only, never through
+(including an untrusted one). `lessons --html`, `export --markdown` and `benefit --markdown` write inside the project root only, never through
 a link.
 
 ## How it works