refactor(execution): remove Eval-owned paid model transports

[drewstone]

Why

Agent Eval owns comparison, scoring, and durable evidence. It still had several ways to execute a paid model itself, each accepting a provider URL or a credential and issuing provider HTTP from Eval. Every one of those paths let a consumer bypass Runtime, which makes exact AgentProfile identity, retries, usage, cache accounting, and interruption safety optional rather than structural.

Three issuers existed on main:

1. src/llm-client.ts — callLlm / callLlmJson / probeLlm / LlmClient, default base URL https://router.tangle.tools/v1, reached through createChatClient's router / direct-provider / cli-bridge variants, createOpenAiCompatibleExecutionOwner, the two judges, wire /v1/judge, eval-campaign, and the CLI.
2. src/multishot/router.ts — its own fetch, plus TANGLE_ROUTER_BASE_URL / TANGLE_API_KEY / JUDGE_MODEL discovered from the environment.
3. src/integrity/preflight.ts — preflightModels({ baseUrl, apiKey }) issuing GET /models and POST /chat/completions.

The architectural call and its evidence are on the issue: #539 (comment)

The short version. agent-runtime already owns paid execution for this contract: profileOptimizerModelCall (src/runtime/profile-chat-client.ts:60, exported at src/runtime/index.ts:336, documented in agent-runtime/docs/improve.md:15,77) implements ExternalOptimizerModelCall against one exact AgentProfile, with a profile digest, a request digest, observed-vs-declared model refusal, transport attempts, and usdKnown provenance. Eval's own contract already says so — src/campaign/external-optimizer-contracts.ts: "For Discovery that owner is Runtime and the identity is an AgentProfile." So createOpenAiCompatibleExecutionOwner (added by #651) was a second, strictly weaker implementation of a role Runtime already owned. Two owners for one role is the defect; the fix is deletion, not relocation. No agent-runtime PR is needed for the move.

What

Eval accepts a caller-supplied ChatClient (or MultishotTransport, or ExternalOptimizerModelCall, or ModelEndpointRequest) and executes nothing.

• createChatClient: router, direct-provider, and cli-bridge deleted with wrapLlmClient. custom, sandbox-sdk, and mock remain; ChatTransport narrows to those three.
• llm-client: the transport half leaves the published surface. callLlm, callLlmJson, LlmClient, LlmClientOptions, LlmRouteRequirements are no longer exported from the root barrel, and src/llm-client.ts is not an export subpath, so no consumer import can reach them. assertLlmRoute and probeLlm are deleted — the caller holds the endpoint, so the caller owns both the route check and the reachability probe, and neither had an in-repo caller left. The canonical contract stays public: LlmCallRequest, LlmCallResult (with logprobs, toolCalls, servedModel), LlmMessage, LlmUsage, costReceiptFromLlm, costReceiptFromLlmError, maximumChargeForLlmRequest, isTransientLlmError, stripFencedJson.
• No default provider URL anywhere. DEFAULT_BASE_URL = 'https://router.tangle.tools/v1' is deleted from the internal client and baseUrl is required. With assertLlmRoute gone, a default endpoint would be exactly the silent fallback that assertion existed to catch; requiring the field makes the bad state unrepresentable instead of asserted-against. The only remaining literal provider hosts under src/ are in cli-config.ts (the binary's documented env mapping) and trace/raw-provider-sink.ts (a redaction hostname table).
• createOpenAiCompatibleExecutionOwner deleted from /campaign. examples/_shared/openai-compatible-owner.ts is the caller-side reference implementation and exposes the endpoint two ways — openAiCompatibleChatClient and openAiCompatibleExecutionOwner. examples/ is not in package.json files, so it ships to nobody: the bad state is unrepresentable, not merely discouraged.
• multishot: src/multishot/router.ts deleted. agentTransport and driverTransport are required, toolTransport names the leg the built-in delegate tools run on, JudgeConfig.transport is required, MultishotToolExecutor receives { transport, signal } instead of { apiKey, baseUrl, signal }, and JUDGE_MODEL is no longer read from the environment. estimateRouterCost → estimateMultishotCost in multishot/cost.ts.
• preflightModels / assertModelsServed: take request: ModelEndpointRequest — a caller-owned function answering a { kind: 'list-models' } or { kind: 'probe', model, maxOutputTokens } check with a Response. Passing the raw Response back is deliberate: it keeps status, the provider's own error.message, budgetExhausted, and served-model substitution exactly as detectable as before. A higher-level { observedModel, receipt } callback would have lost all four.
• judges, wire, campaign: runIntentMatchJudge, runSemanticConceptJudge, handleJudge, dispatchRpc, and createApp take chat: ChatClient (+ optional pricing). /v1/judge refuses with llm_not_configured (503) when no transport is configured — a stronger replacement for the old route assertion. runEvalCampaign takes chatFactory: (wiring) => ChatClient and puts chat on the run context; the campaign hands the factory each run's rawSink and runId, so a transport that binds them still satisfies assertRunCaptured's raw-coverage requirement. The fingerprint folds a caller-declared executionRef where it previously folded a base URL Eval can no longer see.
• The CLI is the sanctioned product path and is scoped in deliberately. agent-eval serve / rpc / rpc-batch is a deployed process whose caller is a JSON-RPC client in another language; it cannot be handed a ChatClient, and deleting it would remove the Python RPC product with no replacement. So src/cli-config.ts is the ONE place in the package that turns an environment credential into a transport, it lives inside the binary, and both README files say so. Both a base URL and a key are now required — a half-configured server refuses instead of calling an unintended endpoint.

Capability preserved across the caller boundary

The thing that must not be lost is the execution evidence, and it all rides on ChatResponse / CostReceiptInput, which are unchanged: model identity (model plus the separate servedModel, which is the only field that can witness a gateway substitution), timeoutMs, cancellation via ChatCallOpts.signal, a stable per-call id via idempotencyKey, retry count (maximumAttempts on the client, transportAttempts in raw), input/output/reasoning/cached usage, billed-or-unknown USD (costUsd: null and costUnknown / usageUnknown stay explicit — never a guessed zero), per-token logprobs, and finite JSON execution evidence on the optimizer path. llmJudge({ scoring: { method: 'expectation' } }) from #637 still works: logprobs ride on the canonical response, so any caller-owned transport can carry them.

No source-check script

CUTS.md killed check-provider-transport.mjs and I agree. The enforcement here is a deletion plus an un-export: the credential-bearing transports do not exist, and the module that still holds one is not reachable through any export subpath. A CI script grepping for process.env.*API_KEY after the fact would be theatre next to that.

Downstream breakage, named

docs/public-api.md records real consumers of the removed surface. Every one of them is the bypass this issue exists to close, and each has a named replacement — profileChatClient / profileOptimizerModelCall from @tangle-network/agent-runtime/kernel, or a custom ChatClient over the client they already have:

removed	consumers to migrate
callLlm, callLlmJson, LlmClient, LlmClientOptions	agent-builder, agent-dev-container, blueprint-agent, ai-trading-blueprint
assertLlmRoute	creative-agent, gtm-agent, starter-foundry
probeLlm	blueprint-agent
preflightModels signature	loops
createChatClient router/direct/cli-bridge	agent-dev-container, agent-lab
routerCompletion, requireRouterApiKey, defaultRouterBaseUrl, estimateRouterCost	gtm-agent, tax-agent
runMultishot / runJudge required transports	agent-runtime examples/p1-parity/arms.ts (drops its two dummy apiKey/baseUrl lines), gtm-agent, tuner-agent

agent-runtime/src imports none of these; only its examples/p1-parity/arms.ts does, and only the two dummy fields it passes because runMultishot used to resolve them eagerly. That example lands when runtime widens its agent-eval catalog pin.

Left for a follow-up, deliberately

profileOptimizerModelCall forwards only req.messages into streamAgentTurn; it drops tools / toolChoice and never returns toolCalls, so it cannot yet serve the tool-calling path #666 shipped for the claude CLI. Closing it needs routerChatWithTools-style pass-through threaded through streamAgentTurn and is blocked on agent-runtime widening its agent-eval catalog pin past 0.150.1, where tools was added to LlmCallRequest. Filed as tangle-network/agent-runtime#927. It is not a prerequisite here: the tool-carrying path is served by a caller-owned owner today.

Also left: multishot carries its own four-model price table (estimateMultishotCost) beside the package's real estimateCost / isModelPriced. Collapsing them changes recorded numbers, so it is a behavior change with its own test, not a rider on a transport removal.

Simplification

Simplification: deleted src/multishot/router.ts, src/campaign/openai-compatible-execution-owner.ts, and the route-assertion + probeLlm block in src/llm-client.ts; collapsed the three hand-rolled "reserve, call, settle, parse" copies in the two judges and the wire judge endpoint into one paidJsonChat (llmJudge keeps its own because its logprob-expectation scoring reads the response differently — named, not smuggled); collapsed the golden matrix check's process-wide globalThis.fetch judge wire into a scripted judge transport, which also removes the module-global install guard and the serial-checks-only constraint it forced.
Net: +1948 / -2494 lines (net -546), 77 files, 3 credential-bearing transports removed, 3 duplicated paid-call copies collapsed to 1, 1 process-wide fetch hijack removed.
Not done here: the runtime tool-forwarding gap (agent-runtime#927) and the multishot price-table duplication, both named above.

Tests: +7 (a caller-owned transport still carries cancellation into a judge and settles an incomplete receipt; /v1/judge refuses with llm_not_configured 503 when no transport is configured; a half-configured CLI route resolves to nothing rather than an unintended endpoint; the CLI binds its resolved route into the transport the wire handlers take; a model answer that is not JSON keeps its settled cost instead of reporting the spend as unknown — that one caught a real regression in this PR, where routing the judges through one helper had moved the parse failure outside runPaidCall and dropped the receipt), -30 deleted (openai-compatible-execution-owner.test.ts whole file, tests/llm-route-assertion.test.ts whole file, the probeLlm block in llm-client.test.ts, the golden matrix judge-wire serialization guard, the assertLlmRoute campaign smoke not.toThrow, the campaign's route-assertion refusal, the json_schema->json_object degrade test, and the multishot "zero HTTP with both seams injected" test — every one of them existed only to exercise a transport that no longer ships)

Proof

All run on agent-eval@10602dcd, rebased on origin/main@cd06dc95 (0.159.1 -> 0.160.0), macOS, node 24.11.1. CI on Linux passed both jobs (ci, gepa-release) on the first commit of this branch.

pnpm typecheck            clean
pnpm typecheck:examples   clean
pnpm typecheck:scripts    clean
pnpm lint                 723 files checked, no fixes applied
pnpm check:canonical-json src/ledger-core/canonical.ts is the only encoder (3 legacy verifiers waived, all matched)
pnpm build                ok, OpenAPI 3.1 spec written
pnpm verify:package       ok (analyst-benchmark digests, skill, model-ids, canonical-json, publint, attw, export surface, evidence index: 10 records, index matches)
uv lock --check           161 packages resolved, lock consistent (clients/python 0.160.0 in lockstep)

Full suite, this branch vs a clean origin/main worktree, same machine, same run:

branch   Tests  41 failed | 5387 passed | 3 skipped (5428)   8 files failed
main     Tests  40 failed | 5393 passed | 3 skipped (5436)   7 files failed

Failures unique to this branch: 1, and it is a load flake, not a regression — external optimizer process terminates the detached process group promptly when the caller aborts. src/campaign/external-optimizer-process.ts and external-optimizer-subprocess.ts are untouched by this branch (git diff --name-only against the base: empty), that same file already fails a different case on clean main under full-suite load, and the case passes 3/3 in isolation on this branch. The rest sit in the same files — analyst/benchmark-command*.test.ts, analyst/benchmark-verification-artifacts.test.ts, tests/campaign/external-optimizer-process.test.ts, tests/campaign/worktree.test.ts — the documented local-sandbox set (git-worktree adapters, process-spawn timeouts) that fails identically on clean main; the two extra on main are known subprocess flakes. CI on Linux is the authority.

Behavior-preservation proof for the two riskiest areas:

• multishot golden records unchanged. src/multishot/golden/golden.test.ts 53/53 pass with the frozen records untouched, after the agent, driver, tool, and now judge legs all run on scripted transports. The judge leg previously hijacked globalThis.fetch; it now uses a scripted MultishotTransport that returns the same costUsd: 0.0007 the old _response_cost produced, so the recorded judge ledger is byte-identical.

• execution evidence survives the caller boundary. src/reference-equivalence-judge.test.ts 16/16, including the rewritten cancellation case: a caller-owned transport receives the campaign's AbortSignal, the abort propagates, and the cost ledger still records { costUnknown: true, usageUnknown: true, error } rather than a zero-cost success.

• The published surface, read off the built artifact (import('./dist/index.js')): callLlm, callLlmJson, LlmClient, probeLlm, assertLlmRoute — all absent. createChatClient, preflightModels, costReceiptFromLlm, maximumChargeForLlmRequest, isTransientLlmError, stripFencedJson, llmJudge — all present. /campaign no longer exports createOpenAiCompatibleExecutionOwner; /multishot no longer exports routerCompletion, requireRouterApiKey, defaultRouterBaseUrl, or estimateRouterCost, and does export estimateMultishotCost.

• The product path still works, proven on the built artifact. node dist/cli.js rpc judge against a local OpenAI-compatible stub:

  # no credentials in the environment
  {"error":{"code":"llm_not_configured","message":"No model transport is configured. ..."}}
  
  # AGENT_EVAL_LLM_BASE_URL / _API_KEY / _MODEL set
  {"result":{"composite":0.83,"dimensions":{"buyer_quality":0.9,"voice":0.8,"signal":0.7},
             "rubricVersion":"anti-slop@59605f13","model":"fake-judge-model","durationMs":16}}
  

  Note "model":"fake-judge-model" — the id the endpoint echoed, not the id requested. Served-model identity survives the new boundary.

• The digest gates moved only their live constants. ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 and ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 are updated by name; ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256 and ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256 are historical facts about published evidence and are untouched. src/analyst/benchmark-reference-result.test.ts 5/5.

Closes #539

[tangletools]

✅ Auto-approved drewstone PR — 10602dcd

This PR was opened by the trusted drewstone account.

This approval is provisional and was applied by the local stand-in because the pr-reviewer webhook host is unreachable (2026-08-21). CI on this head is fully green. The full PR reviewer audit re-runs via the resweep when the service returns and will publish findings if it detects issues.