docs(evals): metering, provider validation and on_removed for host adapters - #466
Merged
Merged
Conversation
…apters The adapter guide told hosts they own usage accounting, which stopped being true when the engine began metering each adapted result. A host following it now double-counts every replay. It also predates provider validation, so it left adapters re-checking what the runner already rejects. Documents replace_scenarios!'s on_removed: option alongside them, since a host syncing a suite from its own catalog is the caller that needs :disable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TonsOfFun
marked this pull request as ready for review
September 18, 2026 16:49
TonsOfFun
added a commit
that referenced
this pull request
Sep 18, 2026
chore(release): activeagent and actionagent 1.6.3 Brings main up to the 1.6.3 release line: context-meter token attribution (#458), host-adapter metering, provider validation and on_removed (#460), evaluation trace correlation (#465), the judge call kind (#468, closes #462), Agent#generations (#469, closes #464), and the host-adapter docs (#466).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three documentation gaps for hosts that integrate their own agent runtime with the dashboard's scenario evaluations. All three follow from #460 (merged) and were found by writing an integration against these docs.
The adapter guide's usage-accounting claim is now wrong
It says:
Since #460 the engine meters each adapted result as one
:execution, one per scenario × model — the same unit the default replay path records. A host following the current text callsActionAgent.record_usageitself and double-counts every replay. Corrected, with the unit stated so a host knows what it is getting.Provider validation is undocumented
#460 also made the runner reject a spec naming a provider the agent cannot serve, before the adapter is called. Worth saying explicitly, because adapters written against the old behaviour carry their own
models.all? { |m| m.provider.in?(...) }check that is now dead code.replace_scenarios!has no proseThe method is how a host syncs a suite from its own catalog, and its
on_removed:option (also #460) is the difference between an old report still resolving its rows and losing them. Adds a short Refreshing a suite section under Suites covering both values and when to choose:disable.Notes
main(which has fix(dashboard): meter adapted replays, validate providers, keep dropped scenarios #460) and targeted atrelease/1.6.3, since the release branch does not carry that commit yet. Retarget tomainif that is the preferred flow.enabled-scope claim is checked againstScenarioEvaluationRunner#selected_scenarios, which selects@evaluation.scenarios.enabled.ordered.docs-host-adapter-meteringrather thandocs/...because a baredocsbranch already exists on the remote and blocks that namespace.