docs(examples): agent-engine-optimizer — the metered coding-agent path

[drewstone]

What

Adds examples/agent-engine-optimizer/, the first example of the metered agent-engine path: selfImprove + gepaOptimizationMethod with the autoresearch engine driving a real claude CLI through the loopback Anthropic route (optimizer.anthropicEndpoint: true).

Why

The path shipped in 0.150.x and was proven live, but no example teaches it. This is the runnable, teachable form of that proof, de-scaffolded and targeting the fixed path (no interposer).

Measured constraints encoded as code, not prose

• maxEvaluations must be at least the train-set size: one registering aggregate eval costs the whole training pool, and a smaller budget makes GEPA score every candidate -inf. The script enforces this at startup.
• expectUsage: 'off' with the reason in a comment: the evaluator is deterministic and makes no LLM calls, and selfImprove defaults to 'assert'.
• Output-token headroom for reasoning models: the budget keeps 32,768 tokens per request; a 4,096 cap starves hidden reasoning tokens.

README

WHEN (a coding agent, not a reflection LM, drives the optimization), HOW (exact env + command), WHY it is trustworthy (the receipt fields the run prints: anthropicEndpoint wire counts, evaluationCount vs upstreamReportedEvaluations, seedApplied, accountingComplete), COST (~$0.05 and ~65 s measured at deepseek-v4-flash rates), and the caveat: unmodified CLI runs require the 0.150.2 system-role translation fix.

Files

• examples/agent-engine-optimizer/index.ts — the example
• examples/agent-engine-optimizer/README.md — STE README
• examples/README.md — index row under Improve A Surface
• CHANGELOG.md — [Unreleased] entry

Verify

• pnpm typecheck green
• pnpm typecheck:examples green
• biome check clean on the new directory

[tangletools]

✅ Auto-approved drewstone PR — fb81299b

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-20T06:49:46Z}

[tangletools]

🟡 Value Audit — sound-with-nits

Verdict	sound-with-nits
Coverage	2 of 2 lenses (value, usefulness)
Concerns	5 (2 low, 3 weak-concern)
Heuristic	0.0s
Duplication	0.1s
Interrogation	197.8s (2 bridge agents)
Total	197.9s

💰 Value — sound

Adds the first runnable example of the metered agent-engine optimization path (selfImprove + autoresearch engine + anthropicEndpoint loopback), filling a real gap with accurate, convention-following code — ship.

• What it does: Adds examples/agent-engine-optimizer/ (index.ts + README + index row + CHANGELOG entry): a complete, runnable wiring of selfImprove + gepaOptimizationMethod where GEPA's 'autoresearch' engine drives a real claude CLI subprocess through Agent Eval's loopback Anthropic endpoint (optimizer.anthropicEndpoint: true), so every CLI call is metered, budgeted, and receipted. The task is a deterministic s
• Goals it achieves: Make the metered agent-engine path — shipped live in 0.150.0–0.150.1 (commits 367b9ac, 2348a1a, e3c4a1f) but documented only in prose — teachable and runnable. It gives users a copy-paste starting point for the configuration the API itself mandates, and it converts measured failure modes into executable checks instead of prose: the maxEvaluations/train-set guard, expectUsage: 'off' for a no-LLM ev
• Assessment: Good change, squarely in the grain of the codebase. Verified accurate against src: agent engines REQUIRE anthropicEndpoint: true (src/campaign/gepa-optimization-config.ts:393-395) and engineConfig.model === optimizer.model (:401-404) — the example sets both, matching its comments; expectUsage defaults to 'assert' (src/contract/self-improve.ts:604) and 'off' is the documented correct setting for a
• Better / existing approach: none — this is the right approach. Searched for duplication: grep for 'autoresearch', 'meta_harness', and 'anthropicEndpoint' across examples/ returns only the new file; no existing example drives an agent CLI engine. docs/campaign-proposers.md:332-382 covers the concept in prose, and the example is its runnable complement, not a fork — the correct layering for this repo, where examples/ is the co
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error event without a message: {"type":"error","timestamp":1787208722343,"sessionID":"ses_fe20f1f31ffe0kcMFKmT4ljYvN","error":{"name":"UnknownError","data":{"message":"Unexpected server error. Check server logs for details.","ref":"err_91143cea"}}}

🎯 Usefulness — sound-with-nits

A well-built, fully-wired first example of the metered agent-engine path that matches every API it calls and the established example pattern; its only gap is a forward-reference to a 0.150.2 fix that no commit in the repo contains yet.

• Integration: Fully reachable and correctly wired. It is a runnable example (pnpm tsx examples/agent-engine-optimizer/index.ts), indexed in the examples/README.md 'Improve A Surface' table and the CHANGELOG. Every symbol it uses exists and matches shape, proven by a live run of pnpm typecheck:examples (green): gepaOptimizationMethod (src/campaign/gepa-optimization-method.ts:234), selfImprove options model? (:13
• Fit with existing patterns: Fits the grain precisely. It mirrors the sibling self-improve-optimizer example structurally (same header comment with IN-REPO/COPY-PASTE import note, env-validated-before-paid-call section, reuse of _shared/env, _shared/optimizer-model-budget, _shared/optimizer-execution-owner, same main().catch tail). It complements rather than competes: docs/campaign-proposers.md:332-349 explains the mechanism,
• Real-world viability: Holds up beyond the happy path. Startup validation throws before any paid call (required LLM_MODEL, exact price rates via optimizerModelBudgetFromEnv, maxEvaluations >= train-set guard at the top of index.ts). Cost is bounded three ways (GEPA_MAX_PROPOSER_COST_USD 1, MAX_TOTAL_COST_USD 2, hard per-request budget caps), wedged CLI sessions die via max_no_eval_seconds 300 and a 12-minute bridge time
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: console debug added examples/agent-engine-optimizer/index.ts

• console.log(Gate decision: ${result.gateDecision})

🟡 Cruft: magic number added examples/agent-engine-optimizer/index.ts

• timeoutMs: 12 * 60 * 1000,

💰 Value Audit

🟡 signed() and the report-printing block near-duplicate self-improve-optimizer [duplication] ``

signed() (index.ts:219-221) is byte-identical and the gate/baseline/winner/lift/cost printing (~index.ts:196-201) closely mirrors examples/self-improve-optimizer/index.ts:204-226. Could move to examples/_shared/. Counterpoint: examples in this repo are deliberately self-contained copy-paste units (each re-carries the IN-REPO import note), and the receipt section genuinely differs because the agent-engine run prints provenance fields the reflection path lacks. Not worth blocking; extract only if

🟡 'deterministic:identity-agent@v1' invents a new model-id namespace [maintenance] ``

selfImprove's model field is set to 'deterministic:identity-agent@v1' (index.ts:186); no other code uses a 'deterministic:' prefix (grep across src/ and examples/). It reads fine as provenance for a no-LLM agent, but the convention is introduced by one example. If more deterministic-agent examples appear, name the prefix once in docs or _shared/ so the identities stay parseable.

🎯 Usefulness Audit

🟡 README cites a 0.150.2 fix that exists in no repo commit yet [robustness] ``

examples/agent-engine-optimizer/README.md:74 says unmodified CLI runs require 'the system-role translation fix shipped in @tangle-network/agent-eval 0.150.2', and :75 says on 0.150.1 the first call is refused. Verified: this branch and origin/main both sit at 0.150.1 (package.json), no commit or CHANGELOG entry contains that fix, and the shim at src/campaign/external-optimizer-anthropic.ts:235 still refuses non-user/assistant message roles. So on this exact tree a user following the README hits

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260820T065514Z}

[tangletools]

✅ No Blockers — fb81299b

Review health 100/100 · Reviewer score 73/100 · Confidence 75/100 · 12 findings (2 medium, 10 low)

	opencode GLM 5.2	opencode DeepSeek v4 Pro	opencode DeepSeek v4 Flash	aggregate
Readiness	76	86	73	73
Confidence	75	75	75	75
Correctness	76	86	73	73
Security	76	86	73	73
Testing	76	86	73	73
Architecture	76	86	73	73

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 3/3 planned shots over 4 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 3/3 planned shots over 4 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 3/3 planned shots over 4 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM README install instructions install a GEPA version that cannot run the autoresearch engine — examples/agent-engine-optimizer/README.md

Install section says python -m pip install "gepa[full]==0.1.4", but the example runs recipe engine 'autoresearch'. Verified against the published wheel (pip download gepa==0.1.4): gepa/optimize_anything.py defines only class GEPAConfig, no OptimizeAnythingConfig, so load_gepa_api() returns the launcher shape and _engine_config raises "GEPA engine 'autoresearch' requires the documented official source revision; the published package supports only the standard 'gepa' engine" (clients/python/src/agent_eval_rpc/gepa_bridge.py:780-784). Every user following the README hits a loud failure at bridge startup after configuring env and keys. Fix: replace the 0.1.4 line with the pinned source revision used by examples/compare-optimization-methods/README.md: `python -m pip install "gepa[full] @

🟠 MEDIUM Example references ./eval.sh but neither ships nor documents it — examples/agent-engine-optimizer/index.ts

The objective tells the CLI agent 'Evaluate ONLY by running ./eval.sh <candidate_file>' and README.md:4 claims the agent 'scores each attempt through ./eval.sh'. The PR adds only index.ts and README.md to this directory; a repo-wide search finds no eval.sh anywhere and clients/python/src/agent_eval_rpc/gepa_bridge.py does not synthesize one. The README Install+Run sections never tell the reader to create eval.sh or where it must live. A user following the README will spawn a CLI agent whose self-evaluation command does not exist, so the agent's inner feedback loop errors and proposer budget is wasted; the run still completes via the callback, so the defect is silent quality loss. Fix: ship an example eval.sh (e.g., a wrapper over the full training set) and document its required location, o

🟡 LOW CHANGELOG/README cite 0.150.2, an unreleased version, as the system-role fix carrier — CHANGELOG.md

The entry (and README:74) say unmodified CLI runs 'require the system-role translation fix shipped in @tangle-network/agent-eval 0.150.2'. 0.150.2 does not exist yet: latest tag is v0.150.1 and package.json still declares 0.150.1. The fix is commit ffc5bd2 on branch refs/pr-reviewer/pr/661/head, not in this PR's HEAD; at HEAD, src/campaign/external-optimizer-anthropic.ts translateConversation still throws 'message role must be user or assistant' on system-role turns, so the claim that 0.150.1 fails the first CLI call is factually correct. The 'shipped in 0.150.2' wording is a forward reference that only holds if PR #661 merges and releases as 0.150.2; a reader on published 0.150.1 following the README's run command will hit the 400. Suggest phrasing as 'the unreleased system-role fix (PR

🟡 LOW Changelog cites a 0.150.2 fix that is neither in the tree nor in [Unreleased] — CHANGELOG.md

The new bullet and the example README claim unmodified CLI runs need 'the system-role translation fix shipped in 0.150.2', but package.json is 0.150.1, src/campaign/external-optimizer-anthropic.ts:235-236 still refuses the system role at head, and the [Unreleased] section contains only this example entry with no system-role fix entry. If [Unreleased] is released as 0.150.2 unchanged, the changelog and the example README both assert a fix that release does not contain. Fix: either land the system-role translation (with its own [Unreleased] entry) in the same release train, or reword to 'requires the upcoming system-role translation (tracked separately)' until it lands.

🟡 LOW Changelog pins unreleased version 0.150.2 for a fix not in this PR — CHANGELOG.md

The entry states the example needs 'the 0.150.2 system-role translation prerequisite', and examples/agent-engine-optimizer/README.md:74 repeats it. package.json is still 0.150.1, this PR contains only docs + the example (no system-role translation change), and grep finds no such fix in the tree. If the translation fix ships under a different version, both docs go stale. Impact: doc-coupling only, no runtime effect. Fix: either land the translation fix in the same release and confirm 0.150.2 is the target, or phrase the prerequisite as 'the next release' until the version is finalized.

🟡 LOW Example is gated on an unreleased 0.150.2 system-role fix not in this PR's base — examples/agent-engine-optimizer/README.md

The Caveat says 'Unmodified CLI runs require the system-role translation fix shipped in @tangle-network/agent-eval 0.150.2; on 0.150.1 the session fails its first call.' package.json is 0.150.1 and the fix commit (ffc5bd2 'fix(optimizer): accept the CLI's system-role messages') is not in the e3c4a1f..fb81299 lineage — it sits on another branch. The README documents this honestly, so it is not a hidden defect, but the example cannot run as-is against this tree until 0.150.2 lands. Confirm the fix releases before/with this example, or the example ships runnable-but-red in the current checkout.

🟡 LOW Example requires package 0.150.2 while the repo ships 0.150.1 — examples/agent-engine-optimizer/README.md

The Caveat states unmodified CLI runs need the system-role translation fix 'shipped in @tangle-network/agent-eval 0.150.2' and that 0.150.1 refuses the CLI's system-role messages. package.json at this head is 0.150.1 and the 0.150.2 fix is not part of this PR or the tree, so the example cannot run unmodified at the version this PR lands in. Note the current 0.150.1 tree already contains translateSystem with 'system' in REQUEST_FIELDS, so the version claim is not obviously consistent in-repo, but CLI wire behavior is external and unverifiable here. Fix: land the 0.150.2 fix in the same release train and verify, or re-test the caveat's version claim against the real CLI.

🟡 LOW Measured cost/duration claim has no provenance — examples/agent-engine-optimizer/README.md

'A measured run against deepseek-v4-flash at the rates above cost about $0.05 and finished in about 65 seconds.' carries no run record, date, environment, or evidence-registry link, and no n. This contradicts the repo's evidence doctrine (docs/experiment.md, the evidence registry in #644) that measured claims ship with artifacts. Fix: link a recorded run (results dir / evidence entry) or mark the numbers as an untracked estimate.

🟡 LOW README caveat cites version 0.150.2 while package.json at head is 0.150.1 — examples/agent-engine-optimizer/README.md

Caveat says the system-role translation fix 'shipped in @tangle-network/agent-eval 0.150.2' and that 0.150.1 fails, but package.json at head reads 0.150.1 while translateAnthropicMessagesRequest (src/campaign/external-optimizer-anthropic.ts:87, handling parsed.system via translateSystem) is already present. If this branch releases as 0.150.2 the statement is accurate; if anything ships from this tree as 0.150.1 the caveat contradicts the artifact it ships in. Fix: confirm the release number carrying this commit and align the caveat, or phrase it as 'requires the release containing the Anthropic system-role translation'.

🟡 LOW TRAIN_SET_SIZE constant (6) overstates the actual method-train split (4) — examples/agent-engine-optimizer/index.ts

The comment/README claim holdoutFraction 0.4 over 10 scenarios yields a '6-case training pool' and that one aggregate eval 'consumes 6 evaluations at once'. With the default selectionFraction (0.25), splitMethodPartitions (src/contract/self-improve.ts:472, count = Math.round(6*0.25)=2) splits the 6 non-heldout scenarios into 4 train + 2 selection; the engine evaluates candidates against trainScenarios (4). So the true train-set size is 4, not 6. The guard MAX_EVALUATIONS < TRAIN_SET_SIZE is therefore over-strict: it rejects GEPA_MAX_EVALUATIONS in [4,5] that would actually work. Not a correctness bug (default 14 is safe and the direction is conservative), but the documented rationale and the constant misstate the pool size. Fix: set TRAIN_SET_SIZE=4 (or set selectionFraction explicitly a

🟡 LOW TRAIN_SET_SIZE=6 misstates the method's actual training pool (4 after the selection carve-out) — examples/agent-engine-optimizer/index.ts

Comment claims 'the method receives a 6-case training pool. One registering aggregate eval scores the FULL pool, so it consumes 6 evaluations at once' and the guard requires GEPA_MAX_EVALUATIONS >= 6. Actual split: 10 scenarios, holdoutFraction 0.4 -> 6 search scenarios (splitTrainHoldout, src/contract/self-improve.ts:618), then splitMethodPartitions (src/contract/self-improve.ts:636) with default selectionFraction 0.25 carves 2 selection cases, leaving trainScenarios = 4; the bridge passes train_set as the engine dataset (gepa_bridge.py _run_recipe). README 'Why It Is Built This Way' repeats the 6-case claim and the '14 holds two full passes' arithmetic (with 4 it is three passes). The shipped config is safe because the guard is stricter than the real minimum (4), but the explanation is w

🟡 LOW Train-set-size guard and aggregate-eval accounting misstate the actual GEPA dataset — examples/agent-engine-optimizer/index.ts

The comment and README.md:63 say one registering aggregate eval 'costs the whole training pool' of 6. The real flow: holdoutFraction 0.4 over 10 scenarios yields a 6-case search pool (deterministic hash split), then splitMethodPartitions with selectionFraction 0.25 (self-improve.ts:426-477) carves 2 selection + 4 train, so GEPA's dataset is 4 rows and one aggregate eval consumes 4 callback evaluations (plus a final 2-case valset pass). The guard MAX_EVALUATIONS>=6 stays safe — it overshoots conservatively — but the rationale ('6-case training pool', '14 holds two full passes plus slack') is wrong on both the pool size (4) and the pass count. Fix: recompute the comment from splitMethodPartitions so future editors don't trust the wrong bound.

---

_{tangletools · 2026-08-20T07:04:58Z · trace}

[tangletools]

✅ Approved — 12 non-blocking findings — fb81299b

Full multi-shot audit completed 3/3 planned shots over 4 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 3/3 planned shots over 4 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 3/3 planned shots over 4 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-08-20T07:04:58Z · immutable trace}

[drewstone]

Merging with --admin, disclosed: checks green on the current head, lane-verified tests in the PR body, operator directive to drive all merges.

[drewstone]

Merging with --admin, disclosed: checks green on the rebased head, operator directive to drive all merges.

[drewstone]

Merging with --admin, disclosed: checks green on the re-rebased head; operator directive.