RFD for agent interaction testing - #279
Conversation
Co-authored-by: Codex<codex@openai.com>
Scope the proposal to a tracer implementation and define isolation, evidence, reliability, token budget, and follow-on work. Co-authored-by: Codex <codex@openai.com>
|
|
||
| The tracer's provisional guard permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens across base input, cache reads, and cache writes, and 1,000 cumulative output tokens. Initial manual runs record actual usage so these limits can be reduced. If the pinned runtime cannot produce the nonce witness within the guard, the prompt, tools, fixture, and context are reduced before any limit is raised. | ||
|
|
||
| At Sonnet 5's standard price after its introductory period, $3 per million input tokens and $15 per million output tokens, the provisional base-token ceiling is approximately $0.09 per run. The runner reports a conservative allowance of $0.20 per run for cache-price differences while the provider key caps the month at $5. The estimate is recalculated when the model pin or [provider pricing](https://www.anthropic.com/research/claude-sonnet-5) changes; tokens remain the primary limit and dollars are derived reporting. |
There was a problem hiding this comment.
I think we can use an even cheaper model like Haiku, it's enough for what we need, right?
There was a problem hiding this comment.
yeah of course. We are interested in the outcome not necessarily the agent(/model) that does the job.
In the end we will support other agents with an ACP adapter, and they should work the same too.
| cargo xtask agent-test [OPTIONS] | ||
| ``` | ||
|
|
||
| Initial options are: |
There was a problem hiding this comment.
I think one --free-only is worth adding. would be all the tests that check for instance that an mcp or skill is loaded, more kind of structure related ones
There was a problem hiding this comment.
By loaded you mean? Discovery or recommendation, or installation or ..?
There was a problem hiding this comment.
Probably not using the right words, I meant to say that there're some stuff we don't really need to spend tokens or interact with the actual underlying model to see if they work correctly or not. For instance, checking if the an MCP you installed is correctly loaded and recognized by the agent, doesn't need any tokens, only a check. same for checking if the MCP is authenticated correctly (in case it's needed), same for skills.
Now if you want to actually test if an skills is triggered correctly, then you have to use the model and wait for it to invoke, that's not free
There was a problem hiding this comment.
Aah, I see. That makes sense.
I'm wondering how do we do it that, we know that the agent have recognized it has a skill or mcp or ...etc without having to wait for it to use it which would need tokens.
There was a problem hiding this comment.
depends on the agent, some of them have commands that allows you to do that, like claude mcp list. I would recommend you to run claude --help to see options
There was a problem hiding this comment.
I'll take a look.
Now that I think about it, I've used claude for some time and I've not run even a single cli command that Claude provides even once 😆
2 participants
What does this PR do?
This RFD proposes an experimental integration-testing harness for exercising Symposium through real CLI processes, scripted user interactions, isolated environment, and selected real coding agents.
Proposed approach
In a nutshell the first tracer will:
cargo agents initandcargo agents syncas real processesScope
This RFD commits only to the tracer. Broader registry scenarios, additional consent branches, persistent conversations, hook and MCP delivery witnesses, ACP conformance, native platform expansion, and release gating are follow-up work informed by what we learn.
Disclosure questions
AI disclosure.
Questions for reviewers.