From 6b1c3ab140b8a9676c663d76453cade05b7ec31e Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Thu, 17 Sep 2026 20:26:47 -0700 Subject: [PATCH] docs(evals): metering, provider validation and on_removed for host adapters The adapter guide told hosts they own usage accounting, which stopped being true when the engine began metering each adapted result. A host following it now double-counts every replay. It also predates provider validation, so it left adapters re-checking what the runner already rejects. Documents replace_scenarios!'s on_removed: option alongside them, since a host syncing a suite from its own catalog is the caller that needs :disable. Co-Authored-By: Claude Opus 5 (1M context) --- docs/framework/evaluations.md | 43 +++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/docs/framework/evaluations.md b/docs/framework/evaluations.md index 24d1f37c..a1b222d0 100644 --- a/docs/framework/evaluations.md +++ b/docs/framework/evaluations.md @@ -246,6 +246,26 @@ document, group display names, or environment flags. Re-import the source document to change that selection. Invalid YAML or a selection containing no scenarios returns an import error without creating a sampled evaluation. +### Refreshing a suite + +`Evaluation#replace_scenarios!` writes a catalog's questions onto a +persisted evaluation. A scenario whose key survives keeps its record, so +results from earlier runs still resolve through it. + +What happens to a scenario the new attributes no longer name is the +`on_removed:` option: + +```ruby +evaluation.replace_scenarios!(attributes) # :destroy (default) +evaluation.replace_scenarios!(attributes, on_removed: :disable) # keep, enabled: false +``` + +`:destroy` drops the row. `:disable` keeps it with `enabled: false`, which +takes it out of future runs — the runner selects only enabled scenarios — +while leaving the results that scored it readable. Choose `:disable` for a +suite synced from an evolving catalog, where a question can come back or an +old report still needs its rows. + ## Running a host application's agent from the mounted dashboard The engine normally replays scenarios with `ActionAgent::Agent#test_execute`. @@ -277,13 +297,22 @@ or duplicate results fail the run instead of leaving a completed report with missing rows. Exceptions also mark the run failed and retain results already written. -The host owns provider execution, usage accounting, tool definitions, and -judge configuration. Use the supplied owner rather than a global current -user in background jobs; honor `evaluation.judge_kind`, -`evaluation.judge_model`, and the evaluation's tenant/role configuration. -The adapter path does not build the engine's judge or execute its agent. -It still passes through dashboard authentication, execution enablement, -and quota checks. +The host owns provider execution, tool definitions, and judge +configuration. Use the supplied owner rather than a global current user in +background jobs; honor `evaluation.judge_kind`, `evaluation.judge_model`, +and the evaluation's tenant/role configuration. The adapter path does not +build the engine's judge or execute its agent. It still passes through +dashboard authentication, execution enablement, and quota checks. + +**Usage accounting is the engine's.** Each result the adapter reports is +metered as one `:execution` against the supplied owner — the same unit the +default replay path records, one per scenario × model. An adapter should +not call `ActionAgent.record_usage` itself, or every replay is counted +twice. + +Models reach the adapter already validated: a spec naming a provider the +agent cannot serve fails the run with `ArgumentError` before the adapter is +called, so an adapter does not need to re-check `models.map(&:provider)`. An observed agent can run a persisted evaluation only when its resolver returns an adapter. Direct agent execution remains read-only. A host