refactor(execution): remove Eval-owned paid model transports

.claude/skills/agent-eval/SKILL.md

@@ -57,7 +57,7 @@ Three public entry points improve or benchmark a surface. Route by intent.
 `selfImprove()` is the agent entry: it gives the method disjoint train and selection partitions, re-scores the selected surface on a held-out split, and returns a `gateDecision`.
 `compareOptimizationMethods()` is the measurement entry: it gives every method equal inputs and scores the selected surfaces on final cases no method received.
 Neither entry ever passes final comparison cases to a method.
-When no package owns execution, build the owner with `createOpenAiCompatibleExecutionOwner` from `/campaign`.
+The execution owner is always caller code: `profileOptimizerModelCall` from `@tangle-network/agent-runtime/kernel`, or your own `ExternalOptimizerModelCall` (copy `examples/_shared/openai-compatible-owner.ts`).
 
 The canonical doc is `docs/campaign-proposers.md`.
 Runnable paths: `examples/self-improve-optimizer/` (selfImprove + official GEPA) and `examples/compare-optimization-methods/` (method comparison).

CHANGELOG.md

@@ -4,6 +4,30 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
+## [0.160.0] — 2026-08-21
+
+### Removed
+
+- Every Eval-owned paid model transport (#539). Agent Eval owns comparison, scoring, and durable evidence; it no longer executes a paid model, accepts a provider URL, or holds a credential. The caller supplies a `ChatClient`, and on Agent Runtime `profileChatClient` / `profileOptimizerModelCall` are that transport, so exact `AgentProfile` identity, retries, usage, cache accounting, and interruption safety stop being optional.
+  - `createChatClient` loses its `router`, `direct-provider`, and `cli-bridge` variants. `custom`, `sandbox-sdk`, and `mock` remain, and `ChatTransport` narrows to those three.
+  - The root barrel no longer exports `callLlm`, `callLlmJson`, `LlmClient`, `LlmClientOptions`, `assertLlmRoute`, `LlmRouteRequirements`, or `probeLlm`. `assertLlmRoute` and `probeLlm` are deleted outright: the caller holds the endpoint, so the caller owns both the route check and the reachability probe. The canonical contract stays public — `LlmCallRequest`, `LlmCallResult` (including `logprobs`, `toolCalls`, `servedModel`), `LlmMessage`, `LlmUsage`, `costReceiptFromLlm`, `costReceiptFromLlmError`, `maximumChargeForLlmRequest`, `isTransientLlmError`, `stripFencedJson`.
+  - `createOpenAiCompatibleExecutionOwner` is gone from `/campaign`. Agent Runtime already owns that role with `profileOptimizerModelCall`, which executes one exact `AgentProfile` and reports profile-digest evidence; two owners for one role was the defect. `examples/_shared/openai-compatible-owner.ts` is the caller-side reference implementation, and it is example code, not a published export.
+  - `multishot/router.ts` is deleted with `routerCompletion`, `requireRouterApiKey`, and `defaultRouterBaseUrl`. `runMultishot`, `runMultishotMatrix`, and `runJudge` now require a caller-supplied `MultishotTransport`; `JudgeConfig.transport` is required and `JUDGE_MODEL` is no longer read from the environment. `MultishotToolExecutor` receives `{ transport, signal }` instead of `{ apiKey, baseUrl, signal }`, and the optional `toolTransport` names the leg the built-in delegate tools run on. `estimateRouterCost` is now `estimateMultishotCost` in `multishot/cost.ts`.
+  - `preflightModels` and `assertModelsServed` take `request: ModelEndpointRequest` instead of `baseUrl` and `apiKey`. Agent Eval asks for a `list-models` or a `probe` check and reads the `Response`, so status, the provider's own `error.message`, `budgetExhausted`, and served-model substitution stay exactly as detectable as before.
+  - `runIntentMatchJudge`, `runSemanticConceptJudge`, `handleJudge`, `dispatchRpc`, and `createApp` take `chat: ChatClient` (plus optional `pricing`) instead of `llm: LlmClientOptions`. `/v1/judge` refuses with `llm_not_configured` (503) when no transport is configured, which replaces the old route assertion.
+  - The internal OpenAI-compatible client has no default endpoint. `DEFAULT_BASE_URL = 'https://router.tangle.tools/v1'` is deleted and `baseUrl` is required, so a misconfigured caller fails loudly instead of silently billing the public router — the failure `assertLlmRoute` existed to catch, now unrepresentable.
+  - `runEvalCampaign` takes `chatFactory: (wiring: CampaignChatWiring) => ChatClient` instead of `llmOpts`, and `CampaignRunContext.chat` replaces `ctx.llmOpts`. The campaign passes each run's `rawSink` and `runId` into the factory, so a transport that binds them still satisfies `assertRunCaptured`'s raw-coverage check. The campaign fingerprint now folds a caller-declared `executionRef` where it previously folded the base URL and provider it can no longer see.
+
+### Added
+
+- `paidJsonChat` collapses the three hand-rolled copies of "reserve the priced maximum, call the transport with a stable call id, settle the receipt, parse the JSON answer" that the two judges and the wire judge endpoint each carried. A malformed answer keeps its settled receipt: the call completed and was billed, so the spend stays known rather than becoming unknown.
+- `LlmChargeBounds`: the narrow bound inputs `maximumChargeForLlmRequest` actually reads, so a caller can price a request without naming a transport options type.
+- `examples/_shared/openai-compatible-owner.ts` exposes one OpenAI-compatible endpoint two ways — `openAiCompatibleChatClient` for judges and workers, `openAiCompatibleExecutionOwner` for the optimizer surface — as the reference for what caller-owned execution looks like.
+
+### Changed
+
+- The `agent-eval` binary is the one place in the package that reads a provider credential, and it is documented as such. `agent-eval serve` / `rpc` / `rpc-batch` build their own `ChatClient` from `AGENT_EVAL_LLM_*` (or the `OPENAI_*` / `TANGLE_*` equivalents) inside `src/cli-config.ts`. Both a base URL and a key are required; a half-configured server refuses instead of calling an unintended endpoint.
+
 ## [0.159.1] — 2026-08-21
 
 ### Added

README.md

@@ -110,21 +110,25 @@ Every row is a function you call. Each links to a runnable example.
 ## Configure Model Calls
 
 Benchmarks, user drivers, executors, built-in judges, completion checkers, and judge adapters all take the same `ChatClient`.
+You own model execution: Agent Eval issues no provider request and never receives a provider credential.
 
 ```ts
 import { createChatClient } from '@tangle-network/agent-eval'
 
 const chat = createChatClient({
-  transport: 'router',
-  apiKey: process.env.TANGLE_API_KEY!,
+  transport: 'custom',
   defaultModel: 'openai/gpt-4.1',
   maximumAttempts: 3,
+  chat: async (request, opts) => myProviderClient(request, opts),
 })
 ```
 
-Use `direct-provider` for an OpenAI-compatible endpoint, `cli-bridge` for a local subscription, `sandbox-sdk` for Sandbox, or `custom` to adapt another SDK.
+On Agent Runtime, `profileChatClient({ profile, executor, context })` from `@tangle-network/agent-runtime/kernel` is that transport: every call runs one exact `AgentProfile` and reports its measured usage, retries, and served model identity.
+Use `sandbox-sdk` for Sandbox and `mock` in tests.
 A custom adapter must return a `ChatResponse` and declare `maximumAttempts` before a capped cost account can dispatch it.
 
+`ChatResponse` carries the whole execution record across that boundary: the served model id, measured input/output/reasoning/cached tokens, billed USD or an explicit unknown, the finish reason, and the per-token log probabilities the expectation judge scores on.
+
 The official GEPA and SkillOpt optimizers run through a Python bridge.
 Install commands, version pins, and the reason for each pin:
 [GEPA](./docs/campaign-proposers.md#install-official-gepa),

clients/python/README.md

@@ -25,6 +25,10 @@ export AGENT_EVAL_LLM_MODEL=gpt-4.1-mini
 `OPENAI_BASE_URL`, `OPENAI_API_KEY`, and `OPENAI_MODEL` are also accepted.
 The endpoint receives the content, rubric, and context passed to `client.judge()`.
 
+The `agent-eval` binary is the only part of the package that reads a provider credential.
+It is a server process, so it configures its own endpoint the way every server does; the TypeScript library holds no key and executes no paid model.
+Without both a base URL and a key, `judge()` fails with `llm_not_configured` instead of calling an unintended endpoint.
+
 ## Judge Content
 
 ```python

clients/python/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agent-eval-rpc"
-version = "0.159.1"
+version = "0.160.0"
 description = "Python RPC client, official optimizer bridge, and DSPy metric adapter for @tangle-network/agent-eval."
 readme = "README.md"
 requires-python = ">=3.10"

clients/python/src/agent_eval_rpc/__init__.py

@@ -53,7 +53,7 @@
 try:
     __version__ = version("agent-eval-rpc")
 except PackageNotFoundError:
-    __version__ = "0.159.1"
+    __version__ = "0.160.0"
 
 __all__ = [
     "Client",

docs/building-doctrine.md

@@ -6,7 +6,7 @@ How every fleet agent that consumes `agent-eval` is built. Each rule is mechanic
 
 Every hard-coded model id or endpoint default is verifiable against the live router. Membership in `{baseUrl}/models` is the free check; an optional 1-token probe per model confirms the router will actually serve it. A default the router cannot serve is a config bug caught before the run, not a runtime surprise that silently degrades into a stub. Backend ids are namespaced by binding: cli-bridge ids (`claude-code/*`, `kimi-code/*`, `opencode/*`) never appear as defaults in code reachable from production: bridge use is an explicit env opt-in, never an implicit fallback.
 
-Enforced by: `preflightModels` (membership + optional probe) and `assertModelsServed` (gate that names every unreachable id with status + detail).
+Enforced by: `preflightModels` (membership + optional probe, over a caller-owned `request` function) and `assertModelsServed` (gate that names every unreachable id with status + detail).
 
 ## 1a. Reachable is not the same as identified
 
@@ -20,14 +20,14 @@ Snapshot resolution (`gpt-4o-mini` → `gpt-4o-mini-2024-07-18`) pins a floating
 Substitution (`gpt-4.1-mini` → `gemini-2.5-flash-lite`) mislabels every number the call produces.
 A response that echoes no model id at all is unproven, which fails closed rather than defaulting to agreement.
 
-Enforced by: `assertServedModel` / `assertServedModels` per call, `assertCrossFamilyServed` for panel diversity computed over the ids that answered, `LlmClientOptions.assertServedModel` to enforce it at the transport, and `assertModelsServed` (probe mode), which now fails a substituted id exactly as it fails a dead one.
+Enforced by: `assertServedModel` / `assertServedModels` per call, `assertCrossFamilyServed` for panel diversity computed over the ids that answered, `ChatResponse.servedModel` carrying the identity the caller's transport observed, and `assertModelsServed` (probe mode), which now fails a substituted id exactly as it fails a dead one.
 `assertCrossFamily` reads requested ids and therefore proves configuration only — reach for the served-side check wherever the diversity claim is load-bearing.
 
 ## 2. Probe the platform before peeling client layers
 
 When a request fails, one direct call against the live endpoint bisects platform-versus-client before any code-level debugging begins. A 401 from the router on a `model_not_found` is the platform telling you the default is dead; a connection refused is the platform being unreachable. Establish which side is at fault with a probe first, then debug only the side that is actually broken.
 
-Enforced by: `preflightModels({ probe: true })`: the probe is the platform-side bisection, carrying the router's own `error.message` back to the caller.
+Enforced by: `preflightModels({ probe: true, request })`: the probe is the platform-side bisection, carrying the endpoint's own `error.message` back to the caller.
 
 ## 3. Agent-produced findings are hypotheses
 

docs/campaign-proposers.md

@@ -313,21 +313,30 @@ With `optimizer`, every recipe stage must use the standard `gepa` engine or a me
 Agent Eval receives no provider key, enforces the declared request and token budget, and records the execution owner's exact usage and opaque finite JSON evidence.
 `maxProposerCostUsd` also limits each individual GEPA engine stage.
 
-When no execution package owns the call, build the callback with `createOpenAiCompatibleExecutionOwner` from `/campaign`:
+`optimizer.call` is always caller code.
+Agent Eval owns no model transport and never receives a provider credential.
 
-```ts
-import { createOpenAiCompatibleExecutionOwner } from '@tangle-network/agent-eval/campaign'
+On agent-runtime, use `profileOptimizerModelCall`, which executes one exact `AgentProfile` and reports profile-digest evidence:
 
-const call = createOpenAiCompatibleExecutionOwner({
-  baseUrl: 'https://api.openai.com/v1',
-  apiKey: process.env.LLM_API_KEY!,
-  model: 'gpt-4.1-mini',
+```ts
+import { profileOptimizerModelCall } from '@tangle-network/agent-runtime/kernel'
+
+const call = profileOptimizerModelCall({
+  profile: optimizerProfile,
+  context: 'prompt optimizer',
+  executor: {
+    backend: 'router',
+    routerBaseUrl: process.env.LLM_BASE_URL!,
+    routerKey: process.env.LLM_API_KEY!,
+  },
   pricing: { inputUsdPerMillion: 0.4, outputUsdPerMillion: 1.6 },
 })
 ```
 
-It executes each admitted request against any OpenAI-compatible `/chat/completions` endpoint and returns the typed outcome with a JSON-clean receipt.
-The credential stays inside the owner closure; the proxy still enforces every budget and identity check.
+Without agent-runtime, implement `ExternalOptimizerModelCall` over the OpenAI-compatible client you already have.
+`examples/_shared/openai-compatible-owner.ts` is a complete minimal implementation to copy.
+The callback resolves with one success or failure result and never rejects, because a rejection loses the execution record and fails the optimizer attempt.
+The credential stays in your process; the proxy still enforces every budget and identity check.
 
 ### Metered agent CLI engines
 

docs/multishot-golden-records.md

@@ -45,13 +45,13 @@ It refuses an `only` id the catalog does not hold, so a stale id after a rename
 An external conversation engine plugs into the matrix through one seam: `RunMultishotMatrixOptions.runShot`.
 The matrix runs that engine for every cell and reads its result through `MultishotCellOutput`, so the same golden checks grade any engine that implements the seam.
 
-The matrix pair is `assertMultishotMatrixGoldenScenario` / `checkMultishotMatrixGoldenScenario`; both take a `runDir` the engine may write into, and both install a deterministic judge wire on `globalThis.fetch` for the duration of the run.
-That wire is process-wide, so run matrix checks serially within one process and keep other fetch traffic out of it.
-Both rules are enforced, not just documented: a second concurrent install throws, and the wire fails loud on any request it does not recognise rather than answering it.
+The matrix pair is `assertMultishotMatrixGoldenScenario` / `checkMultishotMatrixGoldenScenario`; both take a `runDir` the engine may write into.
+Each judge carries a scripted `MultishotTransport`, exactly like the agent and driver legs, so a matrix check owns no process-wide resource and two checks may run at once.
+The judge transport fails loud on a system prompt it does not recognise rather than answering it.
 
 ## Determinism rules
 
-Every scenario is a closed system: scripted transports, scripted tool executors, a fixed persona and profile, fixed token budgets.
+Every scenario is a closed system: scripted transports on every leg including the judges, scripted tool executors, a fixed persona and profile, fixed token budgets.
 No network, no random number, and no clock in a COMPARED field — the fixture envelope carries a `recordedAt` stamp as provenance, and nothing compares it.
 Matrix cells run one at a time, so the request ledger is a property of the conversation engine rather than of how two engines interleave their microtasks.