diff --git a/.gitattributes b/.gitattributes index e9b72d9f..ce56793e 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1,3 @@ *.sql text eol=lf contract/** text eol=lf +dogfood/** text eol=lf diff --git a/.gitignore b/.gitignore index 9a5737ee..4eb50518 100644 --- a/.gitignore +++ b/.gitignore @@ -47,3 +47,6 @@ artifacts/ # The CI-triage tool (experiments/AgentExperience.CiTriage): fetched CI logs, and what --scripted writes experiments/AgentExperience.CiTriage/.cache/ experiments/AgentExperience.CiTriage/results/scripted/ + +# aexp: the local lesson store, open runs, sessions, hook log and the copied config (dogfood/README.md) +.aexp/ diff --git a/CHANGELOG.md b/CHANGELOG.md index b28809c7..c9bcf0e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -612,6 +612,26 @@ AgentExperience.NET is a **preview**. It is not production ready, and public API the file keeps its mode (and, when another user owns it, its owner and group, by an in-place rewrite). - **Library change**: none. Guide: [Claude Code](docs/guide/claude-code.md). +### Dogfood on this repository (story 28.10) + +- **`aexp benefit --markdown `** writes the holdout's estimate as a results page: counts per arm, rates, the + difference with its 95% interval, the verdict, the pending, unverified and mixed-arm runs, the reading rules, and what + the page can and cannot say (among them that runs of one Claude Code session are correlated and the store cannot count + sessions). No path or run ID, only the date, so it can be committed. Inside the project root only, never through a + link. `aexp benefit` without it prints what it did before. +- **[`dogfood/`](dogfood/README.md)**: everything to run the Claude Code recipe on this repository's own sessions, up to + switching it on, which stays a person's step. A config (`dogfood/aexp.config.json`: SQLite under `.aexp/`, holdout + 0.5, recall on, only file paths kept, no recipe consents, each choice commented), the verifier (`dogfood/verify.sh`: + `dotnet build`, then every Docker-free test project from CONTRIBUTING.md, kept equal to that list by a test), and a + pre-registration written before any session ran (`dogfood/preregistration.json`: the session as the randomized unit, + the run as the analysed one, 20 verified runs per arm before the one reading, a stated second look at 50, a deadline, + exclusions and limits, and the SHA-256 of both files, which a test checks). `.gitignore` now excludes `.aexp/`. +- **README**: leads with what the library is for, the block a second run gets after a first run failed and recovered, + the evidence so far (now with the real-task and CI-triage results: no overall benefit shown; lesson content beat a + placebo, p = 0.035; wrong lessons did worse than none; CI triage `NoEvidence` for lack of runs) and the three ways in. + Detail moved down; a "Real work" section says the dogfood run has no result yet. +- **Library change**: none. + ## 0.1.0-preview.17 This preview measures whether injected lessons help. An opt-in holdout withholds the block on a deterministic share of diff --git a/README.md b/README.md index 6e47bac2..dac82d7d 100644 --- a/README.md +++ b/README.md @@ -2,17 +2,41 @@ [![CI](https://github.com/fabbrik/AgentExperience.NET/actions/workflows/ci.yml/badge.svg)](https://github.com/fabbrik/AgentExperience.NET/actions/workflows/ci.yml) [![NuGet](https://img.shields.io/nuget/vpre/AgentExperience.Core.svg?label=nuget%20(preview))](https://www.nuget.org/packages/AgentExperience.Core) [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](./LICENSE) -**Memory of what *worked*, verified by your own checks, for Microsoft Agent Framework agents.** - -When your own code says a run worked (a test run, an exit code, a person's sign-off; never the model's word), the -library keeps what the agent tried as a lesson and hands it to the next run on a similar task. A preview, not +**Memory of what verifiably worked, for AI agents, measured by its own holdout.** When your own check says a run worked +(a test run, an exit code, a person's sign-off; never the model's word), the library keeps what the agent tried as a lesson +for the next run on a similar task, and withholds it from a share of runs to measure whether it helps. A preview, not production ready. How it differs from MAF's own memory: [Compared with MAF memory](docs/compared-with-maf-memory.md). +## What an agent sees + +Run 1 called `run_tests`: exit 3, then a pass, which your check verified. Run 2, on a similar ticket, starts with this +block (shortened; [the whole block](#the-whole-block) is from another sample): + +```text +=== BEGIN HISTORICAL REFERENCE (UNTRUSTED REFERENCE MATERIAL) === +… +Lesson: Verified after 1 attempt. Worked: attempt 0. Checks: [run_tests]. +Tried: + - attempt 0: run_tests [failed: exit 3], run_tests [returned] → completed +… +=== END HISTORICAL REFERENCE === +``` + +## Does it help? + +On one synthetic task, yes, for both models tried. On realistic tasks no overall benefit is shown yet, and wrong lessons did worse than none ([the evidence so far](#the-evidence-so-far)). Mean failed attempts per task in a [live experiment](experiments/AgentExperience.LiveReuse/README.md): + +| Model | No memory | Memory | Placebo | Stale | +| --- | ---: | ---: | ---: | ---: | +| Gemini `gemini-3.1-flash-lite` ([report](experiments/AgentExperience.LiveReuse/results/gemini-gemini-3.1-flash-lite-2026-09-27.md); pre-registered, confirmatory) | 2.17 | **0.00** | 2.42 | 2.75 | +| Claude `claude-haiku-4-5` ([report](experiments/AgentExperience.LiveReuse/results/anthropic-claude-haiku-4-5-2026-09-28.md); exploratory replication) | 2.75 | **0.83** | 2.42 | 2.67 | + ## Quick start -Needs `0.1.0-preview.11` or later. Add `AgentExperience.MicrosoftAgentFramework`, `AgentExperience.Storage.InMemory` -(`--prerelease`) and `Microsoft.Extensions.DependencyInjection`; bring `chatClient` (any `IChatClient`) and -`RunTestsAsync` (your test run, returning text such as `exit=0`). Development and tests only: see [In production](#in-production). +Three ways in: a **Microsoft Agent Framework** agent (below), **any agent** through the `aexp` CLI or MCP server +([CLI and MCP](#any-agent-cli-and-mcp)), or **Claude Code** through hooks ([Claude Code](#claude-code)). + +For MAF (`0.1.0-preview.11` or later), add `AgentExperience.MicrosoftAgentFramework`, `AgentExperience.Storage.InMemory` (`--prerelease`) and `Microsoft.Extensions.DependencyInjection`; bring `chatClient` (any `IChatClient`) and `RunTestsAsync` (your test run, returning text such as `exit=0`). Development and tests only: see [In production](#in-production). ```csharp using AgentExperience.MicrosoftAgentFramework; @@ -35,37 +59,34 @@ await agent.RunAsync("Ticket #4812: a refund is stuck on a lock. Triage it."); await agent.RunAsync("Ticket #4977: a refund is stuck on a lock again. Triage it."); // gets the lesson ``` -## What an agent sees +## Any agent: CLI and MCP -The quick start's second run gets this block (shortened; [the whole block](#the-whole-block) is from another sample): +Agents outside the Agent Framework (Claude Code, Codex, Cursor, a script) get the same loop through `aexp`, a .NET tool +that is both a command line and an MCP stdio server, over a local SQLite file by default (published from the next +preview, `0.1.0-preview.18`; until then, [build it from this repository](dogfood/README.md)): -```text -=== BEGIN HISTORICAL REFERENCE (UNTRUSTED REFERENCE MATERIAL) === -… -Lesson: Verified after 1 attempt. Worked: attempt 0. Checks: [run_tests]. -Tried: - - attempt 0: run_tests [failed: exit 3], run_tests [returned] → completed -… -=== END HISTORICAL REFERENCE === +```bash +dotnet tool install --global AgentExperience.Cli --prerelease +claude mcp add aexp -- aexp mcp # or any MCP client: the server is `aexp mcp`, run in the project ``` -## Does it help? - -On one synthetic task, for the two models tried, yes. Mean failed attempts per task in a [live experiment](experiments/AgentExperience.LiveReuse/README.md): +A `.aexp/config.json` in the project names the scope and the one command `aexp` may run, the verifier +(`"verify": { "command": ["dotnet", "test"] }`, exit code 0 = pass), which runs only once a person has reviewed the +config and approved it with `aexp trust` (an edit revokes the approval). The MCP tools `experience_recall`, +`experience_failure_hint`, `experience_record_call`, `experience_finish` and `experience_lessons` (or the commands +`aexp run start|call|end`, `aexp recall`, `aexp finalize`, `aexp lessons`) capture a run, verify it with that command, +and hand the next run the lesson and, after a failure, the hint, through the same writer, rules and holdout as the +adapter. No tool takes identity, scope, a path or a command, and no model is called. See +[CLI and MCP](docs/guide/cli-and-mcp.md). -| Model | No memory | Memory | Placebo | Stale | -| --- | ---: | ---: | ---: | ---: | -| Gemini `gemini-3.1-flash-lite` ([report](experiments/AgentExperience.LiveReuse/results/gemini-gemini-3.1-flash-lite-2026-09-27.md); pre-registered, confirmatory) | 2.17 | **0.00** | 2.42 | 2.75 | -| Claude `claude-haiku-4-5` ([report](experiments/AgentExperience.LiveReuse/results/anthropic-claude-haiku-4-5-2026-09-28.md); exploratory replication) | 2.75 | **0.83** | 2.42 | 2.67 | +## Claude Code -The experiment (opt-in, real models) gives the model a tool with one hidden working strategy, on 12 task instances, and -compares memory against no memory, a placebo block with the strategy withheld, and stale experience. +A recipe with no code: `aexp setup claude-code` adds hooks that record every tool call, check +each turn that changed something with your tests when it stops, and hand the next session the lesson and the failure +hint; `aexp export --claude-md` keeps the lessons measured as helping in a managed section of `CLAUDE.md`. See +[Claude Code](docs/guide/claude-code.md). -Read it with its limits: one run per model, one synthetic task family, 12 instances, and the working strategy reaches -the block verbatim, so it shows that a model acts on an injected lesson, not that the library helps on real tasks in -general. A harder experiment, where the lesson has to transfer to a different system through the library's own -retrieval, is [in progress, no live results yet](experiments/AgentExperience.LiveReuse/README.md#transfer-experiment). -The samples and the reuse baseline use scripted models: they show the loop works, not that it helps. +This repository's own sessions are set up the same way, to judge the library on real work: see [Real work](#real-work). ## Try it in a minute @@ -89,6 +110,49 @@ The stand-in is a script that tries first whatever strategy an injected block na not that it helps a real model ([Does it help?](#does-it-help) is about that). Set `OPENAI_API_KEY` to run it against one; see [the sample's README](samples/AgentExperience.Sample.QuickStart/README.md). +## The evidence so far + +Each result links its report, which says whether it was pre-registered. None yet is from real work. + +### A synthetic task + +The experiment behind the table [above](#does-it-help) (opt-in, real models) gives the model a tool with one hidden +working strategy, on 12 task instances, and compares memory against no memory, a placebo block with the strategy +withheld, and stale experience. + +Read it with its limits: one run per model, one synthetic task family, 12 instances, and the working strategy reaches +the block verbatim, so it shows that a model acts on an injected lesson, not that the library helps on real tasks in +general. The samples and the reuse baseline use scripted models: they show the loop works, not that it helps. + +### Realistic tasks + +In the [real-task experiment](experiments/AgentExperience.LiveReuse/README.md#real-task-experiment) a model fixes small +services broken by a dependency upgrade, and the lessons reach it through the library's own retrieval. The only run that +could be evaluated (confirmatory; lessons learned by Claude Haiku 4.5, used by gpt-oss-120b, 11 services, +[report](experiments/AgentExperience.LiveReuse/results/realtask-openrouter-anthropic-claude-haiku-4.5-to-openai-gpt-oss-120b-2026-10-27-run4.md)) +shows **no overall benefit**: against no memory, the lessons saved tool calls on 5 services and cost them on 4 +(p = 0.50). What a lesson says did matter: against a placebo block with the fix file withheld, the real lessons used +fewer tool calls on 7 of the 8 services that differed (p = 0.035). Lessons from other breakages did worse than no +memory (0.303 of trials verified, against 0.455). Wrong lessons cost more than none, which is what the holdout is there +to catch. Every other real-task run was inconclusive (too many services excluded) or stopped early (by the provider or a budget cap). + +In the [transfer experiment](experiments/AgentExperience.LiveReuse/README.md#transfer-experiment) a lesson has to help +on a different system that shares one trait. Its one evaluated run (Claude Haiku 4.5) shows no demonstrated benefit +(p = 0.0625), and the placebo and mismatched blocks did about as well as the real one, so even the direction cannot be +credited to what the lessons said. + +### CI triage + +[The library on this repository's failed CI runs](experiments/AgentExperience.CiTriage/README.md), with its holdout +judging ([results](experiments/AgentExperience.CiTriage/results/ci-triage-2026-10-29.md)): 1 of 7 triaged correctly +with lessons, 3 of 9 without. The verdict is `NoEvidence` for lack of runs: each arm needs 20. + +### Real work + +[`dogfood/`](dogfood/README.md) runs the Claude Code recipe on the sessions that build this repository, checked by its +own Docker-free tests, with lessons withheld from half the sessions. It is pre-registered and not switched on yet, so +there is **no result yet**. This section will state the result whichever way it comes out. + ## The whole block The excerpt above comes from the quick start. This whole block comes from the @@ -200,30 +264,6 @@ working attempt also say what its call produced, as `[returned: "pinned lib to 2 capture, sanitized, bounded and signed, off by default (`--recipe` on the quick start shows one; see [Showing what worked](docs/guide/injection.md#showing-what-worked-the-recipe)). -## Any agent: CLI and MCP - -Agents outside the Agent Framework (Claude Code, Codex, Cursor, a script) get the same loop through `aexp`, a .NET tool -that is both a command line and an MCP stdio server, over a local SQLite file by default: - -```bash -dotnet tool install --global AgentExperience.Cli --prerelease -claude mcp add aexp -- aexp mcp # or any MCP client: the server is `aexp mcp`, run in the project -``` - -A `.aexp/config.json` in the project names the scope and the one command `aexp` may run, the verifier -(`"verify": { "command": ["dotnet", "test"] }`, exit code 0 = pass), which runs only once a person has reviewed the -config and approved it with `aexp trust` (an edit revokes the approval). The MCP tools `experience_recall`, -`experience_failure_hint`, `experience_record_call`, `experience_finish` and `experience_lessons` (or the commands -`aexp run start|call|end`, `aexp recall`, `aexp finalize`, `aexp lessons`) capture a run, verify it with that command, -and hand the next run the lesson and, after a failure, the hint, through the same writer, rules and holdout as the -adapter. No tool takes identity, scope, a path or a command, and no model is called. See -[CLI and MCP](docs/guide/cli-and-mcp.md). - -For Claude Code there is a recipe with no code: `aexp setup claude-code` adds hooks that record every tool call, check -each turn that changed something with your tests when it stops, and hand the next session the lesson and the failure -hint; `aexp export --claude-md` keeps the lessons measured as helping in a managed section of `CLAUDE.md`. See -[Claude Code](docs/guide/claude-code.md). - ## Seeing what it learned After the quick start, `provider.GetAgentExperienceLessons().ListForDevelopmentIdentityAsync()` lists the lessons it diff --git a/docs/guide/claude-code.md b/docs/guide/claude-code.md index 4b71b46e..f9210af1 100644 --- a/docs/guide/claude-code.md +++ b/docs/guide/claude-code.md @@ -180,7 +180,7 @@ default). Read it as evidence about your project's sessions, not a promise. The the turns of one session are correlated: they share the session's arm, its task and often its outcome. The interval treats them as independent, so it is narrower than the evidence supports; a session of many turns weighs as many. Look at how many sessions are behind the counts (`ls .aexp/sessions` shows the live ones; the hook log names each session's -runs) before trusting a verdict reached with few sessions. `aexp benefit` cannot count sessions yet. `aexp lessons` shows each lesson's own verdict. See [the holdout](injection.md) for how the arms are assigned. +runs) before trusting a verdict reached with few sessions. `aexp benefit` cannot count sessions yet. `aexp benefit --markdown ` writes the same estimate as a results page to commit, with the reading rules and these limits beside it; [`dogfood/`](../../dogfood/README.md) is this repository's own setup and pre-registration, as a worked example. `aexp lessons` shows each lesson's own verdict. See [the holdout](injection.md) for how the arms are assigned. ## Lessons in CLAUDE.md or AGENTS.md diff --git a/docs/guide/cli-and-mcp.md b/docs/guide/cli-and-mcp.md index fc6caeef..db442abb 100644 --- a/docs/guide/cli-and-mcp.md +++ b/docs/guide/cli-and-mcp.md @@ -134,7 +134,7 @@ aexp finalize --run "$run" # runs the verifier; | `aexp lessons [--all] [--html ]` | A table of the lessons (`--all` adds the ones retrieval would not use), or the [HTML view](deployment.md#seeing-what-it-learned). | | `aexp export --markdown ` | Exports the lessons the holdout measured as helping (the [measured export](injection.md)). | | `aexp export --claude-md [file]` / `--agents-md [file]` | Writes the same lessons into a managed section of `CLAUDE.md` / `AGENTS.md` (the project root's when no file is named), leaving the rest of the file byte for byte ([Claude Code](claude-code.md#lessons-in-claudemd-or-agentsmd)). | -| `aexp benefit` | The holdout's estimate: runs given lessons against runs they were withheld from. | +| `aexp benefit [--markdown ]` | The holdout's estimate: runs given lessons against runs they were withheld from. With `--markdown`, writes it as a results page instead: counts and rates per arm, the difference with its 95% interval, the verdict, the runs left out, the reading rules and what it can and cannot say (no path or run ID, so it can be committed). | | `aexp mcp` | Serves the MCP tools over stdin and stdout. | | `aexp trust` | Prints what the config would run, and approves the config (see above). | | `aexp runs prune --older-than ` | Deletes open runs (never ended) started longer ago than `7d`, `12h` or `30m`, unless a command is using them, and Claude Code session files not touched for as long. | @@ -142,7 +142,7 @@ aexp finalize --run "$run" # runs the verifier; | `aexp hook [--timeout ]` | A Claude Code hook: reads the event's JSON on standard input, prints the hook output; always exits 0 ([Claude Code](claude-code.md)). | Exit codes: 0 done, 1 failed, 2 a usage error (including a run that is not open), 3 a configuration the tool refuses -(including an untrusted one). `lessons --html` and `export --markdown` write inside the project root only, never through +(including an untrusted one). `lessons --html`, `export --markdown` and `benefit --markdown` write inside the project root only, never through a link. ## How it works diff --git a/dogfood/README.md b/dogfood/README.md new file mode 100644 index 00000000..b10f2840 --- /dev/null +++ b/dogfood/README.md @@ -0,0 +1,86 @@ +# Dogfood: the library on this repository's own coding sessions + +Every result so far comes from a lab experiment or a replay. This folder holds what it takes to judge the library on +real work instead: the [Claude Code recipe](../docs/guide/claude-code.md) running on the sessions that build this +repository, checked by this repository's own tests, measured by the library's own holdout. + +**Status: not switched on. No session has run, so there is no result yet.** Switching it on is a person's step, by +design: it installs hooks into local Claude Code settings and approves the command `aexp` may run. + +| File | What it is | +| --- | --- | +| [`aexp.config.json`](aexp.config.json) | The config, with a comment on every choice. Copied to `.aexp/config.json`, which is gitignored. | +| [`verify.sh`](verify.sh) | The verifier: `dotnet build`, then every Docker-free test project from [CONTRIBUTING.md](../CONTRIBUTING.md#development) with `--no-build`. Exit 0 only if all pass. A test keeps its list equal to CONTRIBUTING's. | +| [`preregistration.json`](preregistration.json) | The design, written before any session ran: unit, outcome, arms, when to read the result, what is excluded, the limits. | + +## What it measures + +- **Arms.** The holdout withholds lessons (and failure hints) from half the sessions, drawn once per Claude Code session. +- **Unit.** A session is what is randomized. A run is one turn of a session that changed something; the `Stop` hook + runs `verify.sh` on it. Runs of one session share its arm, so they are clustered; the estimate counts runs and treats + them as independent. `aexp` cannot count sessions. The pre-registration says how that limits the reading. +- **Outcome.** The verifier's exit code at the end of the run: 0 is verified, anything else failed. +- **Reading.** Once, when both arms first have 20 verified runs. A second look at 50 per arm, stated as a second look. + +## Switching it on (a person's steps) + +Do these yourself, in a terminal, not through an agent. Until a preview ships `aexp benefit --markdown`, install the +tool from this checkout instead of the first command below: + +```bash +dotnet pack src/AgentExperience.Cli -o artifacts/aexp -p:VersionSuffix=dogfood.1 +dotnet tool install --global AgentExperience.Cli --version 0.1.0-dogfood.1 --add-source artifacts/aexp +``` + +Setup, three commands, from the repository root: + +```bash +dotnet tool install --global AgentExperience.Cli --prerelease # puts aexp on PATH, where the hooks call it +mkdir -p .aexp && cp dogfood/aexp.config.json .aexp/config.json # setup keeps an existing config +aexp setup claude-code # hooks into .claude/settings.json (gitignored here) +``` + +Run `aexp setup claude-code --print` first to see what it would write. Then: + +1. **Review the verifier.** Read `.aexp/config.json` and `dogfood/verify.sh`. `cmp dogfood/aexp.config.json + .aexp/config.json` must print nothing: the copy is the reviewed file. +2. **Approve it:** `aexp trust`. It prints the command and the program it resolves to. Never let an agent run it. +3. **Protect it.** Add the deny rules `aexp setup claude-code` printed to `~/.claude/settings.json`, plus one for the + verifier script, which lives in the repository where the agent works: + + ```json + "Edit(//**/dogfood/verify.sh)", "Edit(//**/dogfood/aexp.config.json)" + ``` + + These are Claude Code's own checks, not an operating-system boundary: a Bash command can still write those files. + Every such change shows in the pull request's diff. `aexp trust` covers the config's bytes, not the script's. +4. **Work as usual.** Run one dogfood session at a time: two `Stop` hooks building the same checkout at once can + fail each other's build. + +A `Stop` that took longer than 540 s, or ran while the config was untrusted, stores the run without a verdict and says +so; `aexp finalize --run ` finishes it later. When this was written the script took about two minutes after a +warm build. + +## Reading the result + +```bash +aexp benefit # counts per arm and the verdict, any time +aexp benefit --markdown dogfood/results-2027-01-15.md # the results page, on the day of a reading +``` + +The page has the counts per arm, the rates, the difference with its 95% interval, the verdict, the pending and +unverified runs, the reading rules, and what the page can and cannot say. Commit it as written, and update the +README's [Real work](../README.md#real-work) section with its verdict, whichever way it comes out. `aexp lessons` shows +each lesson's own comparison. + +Read only at the pre-registered points. `aexp benefit` prints the verdict with the counts, so you will see it before a +reading point; that must not move the reading. To count sessions roughly, keep copies of `.aexp/hook.log` (it names +each session's runs and keeps about the last 512 KB); a count from it is approximate and must be labelled so. + +## Stopping + +- **For a while:** remove the `aexp hook` entries from `.claude/settings.json`. The store keeps what it has. +- **For good:** remove the hooks, delete `.aexp/`, remove the deny rules, and remove the approval from `trusted.json` + in your user config directory (`~/.config/aexp/`). Write the results page first if a reading point was reached. +- **Changing the design** (the config, the verifier, the rate) after sessions ran: add an amendment to + `preregistration.json` with the date and why, before the next session. diff --git a/dogfood/aexp.config.json b/dogfood/aexp.config.json new file mode 100644 index 00000000..555e255b --- /dev/null +++ b/dogfood/aexp.config.json @@ -0,0 +1,33 @@ +// The dogfood config: the Claude Code recipe on this repository. Copy it to .aexp/config.json (gitignored) before +// 'aexp setup claude-code', which keeps an existing config. Read dogfood/README.md first. Changing a copied config +// revokes 'aexp trust'; changing this file after dogfooding started changes the pre-registered design +// (dogfood/preregistration.json): record it there as an amendment. +{ + // The store, open runs, sessions and the hook log stay in .aexp/, which .gitignore excludes. + "store": "sqlite:.aexp/experience.db", + "identity": { + "tenant": "fabbrik", + "application": "agentexperience-net", + "project": "dev", + "agent": "claude-code" + }, + "verify": { + // The Docker-free test set as one script: build, then every project CONTRIBUTING.md runs without Docker. + // Resolved against the project root (the directory holding .aexp/), never through PATH, never through a shell. + "command": ["dogfood/verify.sh"], + // Under the Stop hook's 600 s, so the hook can still finalize. A slower run is stored; 'aexp finalize' finishes it. + "timeoutSeconds": 540, + "checkId": "docker-free-tests" + }, + // Only file paths are kept from tool calls. A Bash command line is not: it can carry a token or a password. + "sanitization": { "allowArguments": ["file_path"] }, + // No recipe consents. A consent would keep a tool's result text (file contents, command output) as an excerpt of what + // worked, and command output can carry secrets (environment dumps, connection strings, tokens in URLs). The lessons + // still name the tools, the failure class and the file paths. + "recipes": [], + // Half the sessions get no lessons: the arm is drawn once per Claude Code session, from its ID. + "injection": { "holdoutRate": 0.5 }, + "recall": { "enabled": true }, + // Turns made only of these Bash commands count as reading: they do not run the test suite when they stop. + "claudeCode": { "readOnlyBashPrefixes": ["git status", "ls"] } +} diff --git a/dogfood/preregistration.json b/dogfood/preregistration.json new file mode 100644 index 00000000..3d4b0e59 --- /dev/null +++ b/dogfood/preregistration.json @@ -0,0 +1,69 @@ +{ + "experiment": "dogfood", + "version": "dogfood@1", + "registeredOn": "2026-10-30", + "registeredBefore": "No dogfood session had run when this was written: the hooks were not installed and no config was trusted.", + "claim": "Claude Code sessions that build this repository verify their work more often when the library hands them the lessons (and failure hints) earlier verified sessions left, than sessions from which the holdout withholds them.", + "notAClaimAbout": "Whether the lessons make sessions faster or cheaper (secondary, never gated), whether they help other repositories, other agents or other models, or whether a passing test suite means the change was right.", + "design": { + "config": { + "file": "dogfood/aexp.config.json", + "sha256": "1ab26a261f55958f16038ffac6f967712071df0666b5a99eeda9b4c5c65c1fda", + "use": "Copied to .aexp/config.json and approved by the maintainer with 'aexp trust'. The copy must equal this file byte for byte (cmp) for the whole experiment." + }, + "verifier": { + "file": "dogfood/verify.sh", + "sha256": "992ac372a3f521f0cd621535f25aa72de8c1a3a9233891dabde3664fbdb8b153", + "what": "dotnet build, then every Docker-free test project of CONTRIBUTING.md with --no-build; exit 0 only if all pass; stops at the first failure; killed after 540 s.", + "hashNote": "Both digests are SHA-256 of the file with line endings normalized to LF. A test fails when either file changes without this file changing too." + }, + "agent": "Claude Code with the hooks of 'aexp setup claude-code' on the maintainer's machine, whatever model the maintainer's sessions use (the model is not recorded per run and not controlled).", + "work": "The maintainer's ordinary work on this repository: stories, fixes, reviews. No task is chosen or written for the experiment." + }, + "randomization": { + "unit": "One Claude Code session.", + "how": "The holdout draws a session's arm once, from the session ID with the library's default salt, at rate 0.5 (injection.holdoutRate). Every run of the session gets a run ID the library assigns to that same arm, so a session is given lessons on every turn or on none. Hints follow the same arm.", + "assigned": "Only runs that had a lesson or a hint to give are recorded to the holdout. Runs with nothing to give are in neither arm and are not counted." + }, + "unitOfAnalysis": { + "unit": "One run: a turn of a session, from a prompt to the first Stop after the agent changed something (any tool outside the read-only list, or a Bash command not starting with a read-only prefix).", + "clustering": "Runs are clustered by session: the runs of one session share its arm, its task and often its outcome. The library's estimate treats every run as independent, so its interval is narrower than the evidence supports, and a long session weighs as much as many short ones. The store does not record which session a run came from, so the number of sessions behind the counts cannot be reported by aexp; the reading says so beside the verdict. The hook log (.aexp/hook.log, rotated past 256 KB) names each recent session's runs and may be used to give an approximate session count, labelled as approximate." + }, + "outcome": { + "primary": "The verifier's verdict at the end of the run: exit 0 is Verified (success), any other exit is Failed (failure). Decided by the Stop hook, or by 'aexp finalize --run ' for a run the Stop hook stored without a verdict (verifier timed out, config not trusted at the time).", + "notTheOutcome": "The model's own report, a tool call's result, or any later edit." + }, + "comparison": { + "estimate": "AgentExperienceBenefit.EstimateAsync, as 'aexp benefit' and 'aexp benefit --markdown' print it: treatment minus holdout success rate, with the 95% Newcombe hybrid-score interval built from each arm's Wilson interval.", + "verdict": "Helps if the interval lies wholly above zero, Hurts if wholly below, NoEvidence otherwise.", + "minimumVerifiedRunsPerArm": 20 + }, + "looks": { + "first": "Read once, when both arms first have at least 20 runs with a verified outcome. The reading is the page 'aexp benefit --markdown dogfood/results-.md' writes that day, committed as it is. That is the result.", + "second": "A second look when both arms first have at least 50 verified runs, stated as a second look. Two looks at a 95% interval each raise the chance of a false Helps or Hurts above 5%; the second page says so, and a verdict that appears only at the second look is reported as weaker than one at the first.", + "noMore": "No other reading is reported as a result. 'aexp benefit' prints the counts and the verdict together, so the verdict may be seen before a reading point; that never moves a reading point or decides whether a reading is made.", + "deadline": "If both arms have not reached 20 verified runs by 2027-04-30, the page is written that day and reported as it is (NoEvidence for lack of runs, most likely)." + }, + "secondaryMeasures": { + "toolCallsPerSession": "Tool calls per run, and per session where the session can be told, by arm.", + "failedCallsBeforeFirstGreen": "Failed tool calls before the first verified run of a session, by arm.", + "rule": "Reported, never gated, never used to decide a verdict. No tool computes them yet: if none exists at the first reading, they are reported as not measured rather than computed after seeing the primary result." + }, + "exclusions": { + "pending": "Runs stored but never finalized (no verifier verdict): counted as Pending on the page, left out of the rates.", + "unknown": "Runs finalized without a verdict: counted as Unverified, left out of the rates.", + "mixedArm": "Runs recorded in both arms (the rate or salt changed mid-run): counted as Mixed arm, left out of the rates. The config fixes both, so there should be none.", + "readOnlySessions": "Sessions that never changed anything never run the verifier and leave no run: they are not counted anywhere.", + "unpinned": "A run whose ID could not be pinned to its session's arm (4,096 tries) gets no block or hint and is not recorded to the holdout." + }, + "reporting": "The repository README's 'Real work' section states the result of each reading, whichever way it comes out, and links the committed page. Until the first reading it says there is no result yet.", + "limitations": [ + "One repository, one maintainer, one agent product. The sessions are the maintainer's real work, so their tasks differ in kind and difficulty and are not a sample of anything wider.", + "The agent can edit the code the verifier runs: the tests, and dogfood/verify.sh itself. Permission rules (dogfood/README.md) make that visible and harder, not impossible; a reviewer sees every change in the pull request.", + "A run whose tests pass is a success even if its change was wrong or untested, and a run whose tests were already failing for another reason is a failure.", + "The maintainer sees which sessions got a block (it is in the session's context), so the comparison is not blind.", + "Runs are clustered by session and the interval ignores that (see unitOfAnalysis.clustering).", + "Claude Code's model, settings and the library itself change over the months this runs; the arms share those changes, the comparison does not control for them." + ], + "amendments": [] +} diff --git a/dogfood/verify.sh b/dogfood/verify.sh new file mode 100755 index 00000000..7b16e31b --- /dev/null +++ b/dogfood/verify.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# The dogfood verifier (dogfood/aexp.config.json names it): builds the solution, then runs every test project that needs +# no Docker, exactly as CONTRIBUTING.md's "Without Docker" list does, each with --no-build. Exit 0 only if the build +# and every project pass; it stops at the first failure. aexp bounds it with verify.timeoutSeconds and kills it after. +# A test (ContributingWithoutDockerTests) keeps the list below equal to CONTRIBUTING.md's. +set -u +cd "$(dirname "$0")/.." || exit 2 +export DOTNET_CLI_TELEMETRY_OPTOUT=1 DOTNET_NOLOGO=1 + +dotnet build --nologo --verbosity quiet || exit 1 + +run() { + dotnet test "$@" --no-build --nologo --verbosity quiet || exit 1 +} + +run tests/AgentExperience.Abstractions.Tests +run tests/AgentExperience.Core.Tests +run tests/AgentExperience.MicrosoftAgentFramework.Tests +run tests/AgentExperience.Storage.InMemory.Tests +run tests/AgentExperience.Storage.Sqlite.Tests +run tests/AgentExperience.Cli.Tests +run tests/AgentExperience.ReuseBaseline +run tests/AgentExperience.Release.Tests +run tests/AgentExperience.Sample.QuickStart.Tests +run experiments/AgentExperience.LiveReuse.Tests +run experiments/AgentExperience.CiTriage.Tests +run tests/AgentExperience.CompatibilityProof --filter "FullyQualifiedName!~Postgres" +run tests/AgentExperience.Sample.EndToEnd.Tests --filter "FullyQualifiedName!~Postgres" +run tests/AgentExperience.RetrievalQuality --filter "FullyQualifiedName!~Postgres" +run tests/AgentExperience.Contract.Tests --filter "FullyQualifiedName!~Postgres" + +exit 0 diff --git a/src/AgentExperience.Cli/AexpCli.cs b/src/AgentExperience.Cli/AexpCli.cs index 8a9f28d7..e8e4a990 100644 --- a/src/AgentExperience.Cli/AexpCli.cs +++ b/src/AgentExperience.Cli/AexpCli.cs @@ -46,7 +46,7 @@ aexp recall --run --failure --tool --error aexp export [--markdown ] [--claude-md [file]] [--agents-md [file]] Export lessons measured as helping (to a file, or the managed section of CLAUDE.md / AGENTS.md). - aexp benefit The holdout's estimate. + aexp benefit [--markdown ] The holdout's estimate (or a results page). aexp mcp Serve the MCP tools over stdio. aexp trust Show the config's verifier, and approve the config. aexp runs prune --older-than Delete open runs older than e.g. 7d, 12h, 30m. @@ -236,8 +236,23 @@ await output.WriteLineAsync( return Ok; case ["benefit"]: - args.Only(); - await output.WriteAsync(Formatting.Benefit(await engine.BenefitAsync(cancellationToken).ConfigureAwait(false))).ConfigureAwait(false); + args.Only("markdown"); + var estimate = await engine.BenefitAsync(cancellationToken).ConfigureAwait(false); + if (args.Value("markdown") is { } resultsPage) + { + var target = InsideProject(engine, resultsPage, console); + await File.WriteAllTextAsync( + target, + Formatting.BenefitMarkdown(estimate, engine.Settings, console.Clock.GetUtcNow()), + new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), + cancellationToken).ConfigureAwait(false); + await output.WriteLineAsync($"Wrote the holdout's estimate ({estimate.Overall.Verdict}) to {resultsPage}.").ConfigureAwait(false); + } + else + { + await output.WriteAsync(Formatting.Benefit(estimate)).ConfigureAwait(false); + } + return Ok; case ["mcp"]: @@ -461,4 +476,74 @@ static string Arm(ExperienceBenefitArm arm) => text.AppendLine(CultureInfo.InvariantCulture, $"Runs per arm needed for a verdict: {estimate.MinimumRunsPerArm}; not yet verified: {overall.Pending}"); return text.ToString(); } + + /// + /// The estimate as a results page: counts per arm, rates, the difference with its interval, the verdict, the runs left + /// out, the reading rules and what the page can and cannot say. No path, run ID or timestamp beyond the date, so the + /// page can be committed. + /// + public static string BenefitMarkdown(ExperienceBenefitEstimate estimate, AexpSettings settings, DateTimeOffset readOn) + { + static string Count(int value) => value.ToString(CultureInfo.InvariantCulture); + static string Signed(double value) => value.ToString("+0.00;-0.00;0.00", CultureInfo.InvariantCulture); + static string Arm(string name, ExperienceBenefitArm arm) => + $"| {name} | {Count(arm.Runs)} | {Count(arm.Successes)} | " + + (arm.Rate is { } rate ? (rate * 100).ToString("0", CultureInfo.InvariantCulture) + "%" : "n/a") + " |\n"; + + var minimum = estimate.MinimumRunsPerArm; + var b = new StringBuilder(); + b.Append("# Holdout results, ").Append(readOn.UtcDateTime.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)).Append("\n\n"); + b.Append("- Scope: application `").Append(settings.Scope.ApplicationId).Append("`, project `").Append(settings.Scope.ProjectId).Append("`\n"); + b.Append("- Holdout rate: ").Append(settings.HoldoutRate.ToString("0.0##", CultureInfo.InvariantCulture)) + .Append(" (lessons withheld from that share of the runs that had one to give)\n"); + b.Append("- Verifier: check `").Append(settings.Verifier.CheckId).Append("`, the configured command's exit code (0 = verified)\n"); + b.Append("- Verified runs each arm needs for a verdict: ").Append(Count(minimum)).Append('\n'); + + b.Append("\n## The library's estimate\n\n"); + if (estimate.Outcome != ExperienceStoreOutcome.Found) + { + b.Append("The store answered `").Append(estimate.Outcome.ToString()).Append("`, so nothing was read and there is no estimate.\n"); + } + else if (!estimate.Tracked) + { + b.Append("This store does not track the holdout, so there is no estimate.\n"); + } + else + { + var o = estimate.Overall; + b.Append("From `AgentExperienceBenefit.EstimateAsync` over the holdout's rows and each run's finalized outcome.\n\n"); + b.Append("| Arm | Runs | Successes | Rate |\n| --- | ---: | ---: | ---: |\n"); + b.Append(Arm("treatment", o.Treatment)); + b.Append(Arm("holdout", o.Holdout)); + b.Append("\n`treatment` runs were given the lessons (and hints) that matched; `holdout` runs had them withheld. Runs with nothing to give are in neither arm.\n"); + b.Append("\n- Difference (treatment minus holdout): ").Append(o.Difference is { } d ? Signed(d) : "n/a") + .Append(", 95% interval ").Append(o.Lower is { } lower && o.Upper is { } upper ? Signed(lower) + " to " + Signed(upper) : "n/a").Append('\n'); + b.Append("- Verdict: **").Append(o.Verdict.ToString()).Append("**\n"); + b.Append("- Pending: ").Append(Count(o.Pending)) + .Append(", Unverified: ").Append(Count(o.Unverified)) + .Append(", Mixed arm: ").Append(Count(o.MixedArm)).Append("\n\n"); + + b.Append(o.Verdict switch + { + ExperienceBenefitVerdict.Helps => "The verdict is `Helps`: the 95% interval of the difference lies wholly above zero.", + ExperienceBenefitVerdict.Hurts => "The verdict is `Hurts`: the 95% interval of the difference lies wholly below zero.", + _ when o.Treatment.Runs < minimum || o.Holdout.Runs < minimum => + $"The verdict is `NoEvidence` for lack of runs: each arm needs at least {Count(minimum)} runs with a verified outcome, and this has {Count(o.Treatment.Runs)} treatment and {Count(o.Holdout.Runs)} holdout. It says nothing yet about whether the lessons help.", + _ => "The verdict is `NoEvidence`: the interval includes zero. That is not \"no effect\"; the data cannot support a claim either way.", + }).Append('\n'); + } + + b.Append("\n## Reading rules\n\n"); + b.Append("- A run counts once the verifier decided it: exit 0 is a success, any other exit a failure. Pending runs (not finalized yet) and unverified runs (no verdict) are counted above and left out of the rates.\n"); + b.Append("- `Helps` needs the 95% interval of the difference (Newcombe's hybrid score, from each arm's Wilson interval) wholly above zero, `Hurts` wholly below. Anything else, or fewer than ") + .Append(Count(minimum)).Append(" verified runs in an arm, is `NoEvidence`.\n"); + b.Append("- Read it at the point fixed in advance. Reading again and again until the verdict looks good overstates it: each look is another test.\n"); + + b.Append("\n## What this can and cannot say\n\n"); + b.Append("- A success means the verifier passed, nothing more. A run whose tests pass is a success even if the tests did not cover its change, and the agent can edit the code the verifier runs.\n"); + b.Append("- The interval treats every run as independent. Under the Claude Code hooks the session is what is randomized and each turn that changed something is a run: the runs of one session share its arm and are correlated, so the interval is narrower than the evidence supports and a long session weighs as much as many short ones. The store does not record which session a run came from, so this page cannot count sessions.\n"); + b.Append("- Arms are assigned only on runs that had a lesson or a hint to give, so the comparison is about runs where memory had something to say, not about every run.\n"); + b.Append("- One lesson's own comparison is in `aexp lessons`; this page pools them all.\n"); + return b.ToString(); + } } diff --git a/src/AgentExperience.Cli/README.md b/src/AgentExperience.Cli/README.md index ed59a7ea..b5e01927 100644 --- a/src/AgentExperience.Cli/README.md +++ b/src/AgentExperience.Cli/README.md @@ -49,6 +49,7 @@ aexp run call --run "$run" --tool run_tests --result "all passed" aexp run end --run "$run" aexp finalize --run "$run" # runs the configured check; may run in a later process aexp lessons +aexp benefit --markdown results.md # the holdout's estimate, as a page to commit ``` ## With Claude Code diff --git a/tests/AgentExperience.Cli.Tests/ClaudeCodeHookTests.BenefitPage.cs b/tests/AgentExperience.Cli.Tests/ClaudeCodeHookTests.BenefitPage.cs new file mode 100644 index 00000000..45779c5c --- /dev/null +++ b/tests/AgentExperience.Cli.Tests/ClaudeCodeHookTests.BenefitPage.cs @@ -0,0 +1,103 @@ +using System.Runtime.CompilerServices; +using System.Text.Json.Nodes; +using System.Text.RegularExpressions; +using AgentExperience.Cli.ClaudeCode; + +namespace AgentExperience.Cli.Tests; + +/// +/// aexp benefit --markdown: the results page, golden, over a SQLite store seeded through the Claude Code hooks with +/// one session in each holdout arm (so every count is fixed: the arms follow the session IDs, not chance). +/// +public sealed partial class ClaudeCodeHookTests +{ + private const string UpdateGoldenVariable = "AEXP_UPDATE_GOLDEN"; + + [Fact] + public async Task Benefit_markdown_writes_the_results_page_for_a_seeded_store() + { + using var project = new TempProject(config => config["injection"] = new JsonObject { ["holdoutRate"] = 0.5 }); + await LearnAsync(project, "d0d0d0d0-d0d0-4d0d-8d0d-d0d0d0d0d0d0"); + + var salt = MicrosoftAgentFramework.Injection.ExperienceHoldoutOptions.DefaultSalt; + string Find(Abstractions.HoldoutArm arm) => Enumerable.Range(0, 1000) + .Select(i => $"d1d1d1d1-d1d1-4d1d-8d1d-{i:D12}") + .First(id => SessionHoldout.ArmOf(id, salt, 0.5) == arm); + + // Treatment: three turns, the second fails its tests. Holdout: two turns, the first fails, then a third turn the + // session ends before it stops, so it is stored without a verdict (pending). + await TurnsAsync(project, Find(Abstractions.HoldoutArm.Treatment), [true, false, true]); + var holdout = Find(Abstractions.HoldoutArm.Holdout); + await TurnsAsync(project, holdout, [false, true]); + Assert.Null(await HookAsync(project, "UserPromptSubmit", Session(holdout, Task + " last turn"))); + Assert.Null(await HookAsync(project, "PostToolUse.write", Session(holdout))); + Assert.Null(await HookAsync(project, "SessionEnd", Session(holdout))); + + var written = await project.OkAsync("benefit", "--markdown", "results.md"); + Assert.Equal("Wrote the holdout's estimate (NoEvidence) to results.md." + Environment.NewLine, written); + + var page = await File.ReadAllTextAsync(Path.Combine(project.Root, "results.md")); + Assert.Matches(new Regex(@"\A# Holdout results, \d{4}-\d{2}-\d{2}\n"), page); + var normalized = Regex.Replace(page, @"\A# Holdout results, \d{4}-\d{2}-\d{2}\n", "# Holdout results, \n"); + Assert.DoesNotContain(project.Root, page, StringComparison.Ordinal); + + var golden = GoldenPath(); + if (Environment.GetEnvironmentVariable(UpdateGoldenVariable) == "1") + { + await File.WriteAllTextAsync(golden, normalized); + } + + Assert.Equal(await File.ReadAllTextAsync(golden), normalized); + + // Without --markdown the terminal summary is unchanged. + Assert.Contains("Given lessons: 2/3 verified", await project.OkAsync("benefit")); + } + + [Fact] + public async Task Benefit_markdown_writes_inside_the_project_only_and_takes_no_other_option() + { + using var project = new TempProject(); + var outside = Path.Combine(Path.GetTempPath(), $"aexp-outside-{Guid.NewGuid():N}.md"); + var refused = await project.RunAsync("benefit", "--markdown", outside); + Assert.Equal(AexpCli.Usage, refused.Exit); + Assert.False(File.Exists(outside)); + + Assert.Equal(AexpCli.Usage, (await project.RunAsync("benefit", "--html", "x.html")).Exit); + Assert.Equal(AexpCli.Usage, (await project.RunAsync("benefit", "--markdown")).Exit); + + // An empty store: no runs in either arm, the verdict for lack of runs, no difference. + await project.OkAsync("benefit", "--markdown", "empty.md"); + var page = await File.ReadAllTextAsync(Path.Combine(project.Root, "empty.md")); + Assert.Contains("| treatment | 0 | 0 | n/a |", page); + Assert.Contains("- Difference (treatment minus holdout): n/a, 95% interval n/a", page); + Assert.Contains("for lack of runs", page); + } + + /// One turn per entry: a prompt, a write, then Stop with the tests passing or failing as given. + private static async Task TurnsAsync(TempProject project, string session, bool[] passes) + { + for (var turn = 0; turn < passes.Length; turn++) + { + // A treatment session's prompt is answered with the block; a holdout session's with nothing. + await HookAsync(project, "UserPromptSubmit", Session(session, Task + " turn " + turn)); + Assert.Null(await HookAsync(project, "PostToolUse.write", Session(session))); + var fixedFile = Path.Combine(project.Root, "fixed"); + if (passes[turn]) + { + project.Fix(); + } + else + { + File.Delete(fixedFile); + } + + var stop = (string?)(await HookAsync(project, "Stop", Session(session)))?["systemMessage"]; + Assert.Contains(passes[turn] ? "exit 0 (pass)" : "exit 1", stop); + } + + project.Fix(); + } + + private static string GoldenPath([CallerFilePath] string source = "") => + Path.Combine(Path.GetDirectoryName(source)!, "Goldens", "benefit-results.md"); +} diff --git a/tests/AgentExperience.Cli.Tests/DogfoodConfigTests.cs b/tests/AgentExperience.Cli.Tests/DogfoodConfigTests.cs new file mode 100644 index 00000000..8cc84e77 --- /dev/null +++ b/tests/AgentExperience.Cli.Tests/DogfoodConfigTests.cs @@ -0,0 +1,108 @@ +using System.Security.Cryptography; +using System.Text; +using System.Text.Json.Nodes; +using AgentExperience.Cli.Configuration; +using AgentExperience.Cli.Engine; +using AgentExperience.Core.Holdout; + +namespace AgentExperience.Cli.Tests; + +/// +/// The repository's own dogfood setup (dogfood/): its config loads with the tool's loader once copied to +/// .aexp/config.json, its verifier resolves to the committed script, and the pre-registration names the exact files. +/// Nothing here installs a hook, approves a config or runs the script. +/// +public sealed class DogfoodConfigTests +{ + private static string Repository { get; } = FindRepository(); + + private static string Dogfood(string name) => Path.Combine(Repository, "dogfood", name); + + [Fact] + public void The_config_copied_to_aexp_loads_and_its_verifier_is_the_committed_script() + { + var root = Path.Combine(Path.GetTempPath(), $"aexp-dogfood-{Guid.NewGuid():N}"); + try + { + Directory.CreateDirectory(Path.Combine(root, ".aexp")); + Directory.CreateDirectory(Path.Combine(root, "dogfood")); + File.Copy(Dogfood("aexp.config.json"), Path.Combine(root, ".aexp", "config.json")); + var script = Path.Combine(root, "dogfood", "verify.sh"); + File.Copy(Dogfood("verify.sh"), script); + if (!OperatingSystem.IsWindows()) + { + File.SetUnixFileMode(script, File.GetUnixFileMode(Dogfood("verify.sh"))); + } + + var settings = AexpConfigLoader.Load(null, root, Environment.GetEnvironmentVariable); + + Assert.Equal(Path.GetFullPath(root), Path.GetFullPath(settings.ProjectRoot)); + Assert.Equal(("fabbrik", "agentexperience-net", "dev"), (settings.Scope.TenantId, settings.Scope.ApplicationId, settings.Scope.ProjectId)); + var store = Assert.IsType(settings.Store); + Assert.Equal(Path.Combine(Path.GetFullPath(root), ".aexp", "experience.db"), store.Path); + Assert.Equal(0.5, settings.HoldoutRate); + Assert.Null(settings.HoldoutSalt); + Assert.True(settings.RecallEnabled); + Assert.Equal(["file_path"], settings.AllowArguments); + Assert.Empty(settings.Recipes); + + // Below the Stop hook's 600 s (aexp setup claude-code), so the hook can finalize around it. + Assert.Equal(["dogfood/verify.sh"], settings.Verifier.Command); + Assert.True(settings.Verifier.Timeout < TimeSpan.FromSeconds(600)); + Assert.False(settings.Verifier.AcceptExitCode); + Assert.Equal(Path.GetFullPath(root), Path.GetFullPath(settings.Verifier.WorkingDirectory)); + Assert.Equal(script, VerifierRunner.ResolveProgram(settings.Verifier, settings.ProjectRoot, Environment.GetEnvironmentVariable)); + } + finally + { + Directory.Delete(root, recursive: true); + } + } + + [Fact] + public void The_committed_script_is_executable_and_runs_the_tests_without_a_shell_in_the_config() + { + var script = File.ReadAllText(Dogfood("verify.sh")); + Assert.StartsWith("#!/usr/bin/env bash\n", script, StringComparison.Ordinal); + Assert.DoesNotContain('\r', script); + Assert.Contains("dotnet build", script, StringComparison.Ordinal); + if (!OperatingSystem.IsWindows()) + { + Assert.True(File.GetUnixFileMode(Dogfood("verify.sh")).HasFlag(UnixFileMode.UserExecute), "dogfood/verify.sh is not executable."); + } + } + + [Fact] + public void The_preregistration_names_the_files_as_they_are_and_the_library_defaults_it_relies_on() + { + var registration = JsonNode.Parse(File.ReadAllText(Dogfood("preregistration.json")))!; + var design = registration["design"]!; + Assert.Equal("dogfood/aexp.config.json", (string?)design["config"]!["file"]); + Assert.Equal("dogfood/verify.sh", (string?)design["verifier"]!["file"]); + + // A change to either file is a change to the design: it must come with a new digest (and an amendment). + Assert.Equal(Sha256(Dogfood("aexp.config.json")), (string?)design["config"]!["sha256"]); + Assert.Equal(Sha256(Dogfood("verify.sh")), (string?)design["verifier"]!["sha256"]); + + Assert.Equal(ExperienceBenefitOptions.DefaultMinimumRunsPerArm, (int)registration["comparison"]!["minimumVerifiedRunsPerArm"]!); + Assert.Equal("One Claude Code session.", (string?)registration["randomization"]!["unit"]); + Assert.NotNull(registration["amendments"]!.AsArray()); + } + + /// SHA-256 of the file with line endings normalized to LF, in lowercase hex. + private static string Sha256(string path) => + Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(File.ReadAllText(path).ReplaceLineEndings("\n")))); + + private static string FindRepository() + { + for (var directory = new DirectoryInfo(AppContext.BaseDirectory); directory is not null; directory = directory.Parent) + { + if (File.Exists(Path.Combine(directory.FullName, "AgentExperience.NET.sln"))) + { + return directory.FullName; + } + } + + throw new InvalidOperationException("The repository root (AgentExperience.NET.sln) was not found above the test output."); + } +} diff --git a/tests/AgentExperience.Cli.Tests/Goldens/benefit-results.md b/tests/AgentExperience.Cli.Tests/Goldens/benefit-results.md new file mode 100644 index 00000000..733e9b91 --- /dev/null +++ b/tests/AgentExperience.Cli.Tests/Goldens/benefit-results.md @@ -0,0 +1,36 @@ +# Holdout results, + +- Scope: application `cli-tests`, project `repo` +- Holdout rate: 0.5 (lessons withheld from that share of the runs that had one to give) +- Verifier: check `tests`, the configured command's exit code (0 = verified) +- Verified runs each arm needs for a verdict: 20 + +## The library's estimate + +From `AgentExperienceBenefit.EstimateAsync` over the holdout's rows and each run's finalized outcome. + +| Arm | Runs | Successes | Rate | +| --- | ---: | ---: | ---: | +| treatment | 3 | 2 | 67% | +| holdout | 2 | 1 | 50% | + +`treatment` runs were given the lessons (and hints) that matched; `holdout` runs had them withheld. Runs with nothing to give are in neither arm. + +- Difference (treatment minus holdout): +0.17, 95% interval -0.45 to +0.65 +- Verdict: **NoEvidence** +- Pending: 1, Unverified: 0, Mixed arm: 0 + +The verdict is `NoEvidence` for lack of runs: each arm needs at least 20 runs with a verified outcome, and this has 3 treatment and 2 holdout. It says nothing yet about whether the lessons help. + +## Reading rules + +- A run counts once the verifier decided it: exit 0 is a success, any other exit a failure. Pending runs (not finalized yet) and unverified runs (no verdict) are counted above and left out of the rates. +- `Helps` needs the 95% interval of the difference (Newcombe's hybrid score, from each arm's Wilson interval) wholly above zero, `Hurts` wholly below. Anything else, or fewer than 20 verified runs in an arm, is `NoEvidence`. +- Read it at the point fixed in advance. Reading again and again until the verdict looks good overstates it: each look is another test. + +## What this can and cannot say + +- A success means the verifier passed, nothing more. A run whose tests pass is a success even if the tests did not cover its change, and the agent can edit the code the verifier runs. +- The interval treats every run as independent. Under the Claude Code hooks the session is what is randomized and each turn that changed something is a run: the runs of one session share its arm and are correlated, so the interval is narrower than the evidence supports and a long session weighs as much as many short ones. The store does not record which session a run came from, so this page cannot count sessions. +- Arms are assigned only on runs that had a lesson or a hint to give, so the comparison is about runs where memory had something to say, not about every run. +- One lesson's own comparison is in `aexp lessons`; this page pools them all. diff --git a/tests/AgentExperience.Release.Tests/Documentation/ContributingWithoutDockerTests.cs b/tests/AgentExperience.Release.Tests/Documentation/ContributingWithoutDockerTests.cs index 8797dd9b..4c89a5dc 100644 --- a/tests/AgentExperience.Release.Tests/Documentation/ContributingWithoutDockerTests.cs +++ b/tests/AgentExperience.Release.Tests/Documentation/ContributingWithoutDockerTests.cs @@ -80,6 +80,17 @@ public void A_project_run_without_Docker_runs_no_class_that_uses_a_container() Assert.True(problems.Count == 0, string.Join("\n", problems)); } + [Fact] + public void The_dogfood_verifier_runs_exactly_the_commands_listed_here() + { + var script = File.ReadAllText(Path.Combine(RepositoryRoot.Path, "dogfood", "verify.sh")).ReplaceLineEndings("\n"); + var scripted = ScriptRun().Matches(script) + .Select(match => (match.Groups["project"].Value, match.Groups["filter"].Success ? match.Groups["filter"].Value : null)) + .ToList(); + + Assert.Equal(RunCommands(Section()), scripted); + } + /// The "Without Docker" part of CONTRIBUTING.md, up to the next heading. private static string Section() { @@ -184,6 +195,10 @@ private static bool IsBuildOutput(string path) [GeneratedRegex(@"^dotnet test (?[\w./-]+)(?: --filter ""(?[^""]+)"")?[ \t]*$", RegexOptions.Multiline)] private static partial Regex RunCommand(); + // dogfood/verify.sh: one 'run [--filter "..."]' line per CONTRIBUTING.md command. + [GeneratedRegex(@"^run (?[\w./-]+)(?: --filter ""(?[^""]+)"")?[ \t]*$", RegexOptions.Multiline)] + private static partial Regex ScriptRun(); + [GeneratedRegex(@"`((?:tests|experiments|samples|benchmarks)/[\w./-]+)`")] private static partial Regex QuotedProject();