Skip to content

Agent test UI: pick tools and agents rather than type them - #481

Open
yuyixg wants to merge 1 commit into
SciSharp:mainfrom
yuyixg:feature/agent-test-set-p1-ui
Open

Agent test UI: pick tools and agents rather than type them#481
yuyixg wants to merge 1 commit into
SciSharp:mainfrom
yuyixg:feature/agent-test-set-p1-ui

Conversation

@yuyixg

@yuyixg yuyixg commented Aug 27, 2026

Copy link
Copy Markdown

A tool name and an agent name are both things an author has in front of them and cannot spell from memory, and both fail silently when mistyped: a toolCalled on a function that does not exist can never pass, and a routedToAgent on a misspelt agent fails every run while reading like a real regression. The datalist on the tool fields only suggested -- it never stopped a typo.

  • toolCalled / toolNotCalled target, and a mock's function name, are now searchable pickers over the tool catalogue.
  • routedToAgent expected is an agent picker. It stores the name, not the id: both are accepted (AgentChainHop.Matches), and the name is what the run report prints in the expected and actual columns.
  • agentChain expected is an ordered chip list -- add from a picker, remove, move. Ordered rather than a multi-select because in ordered and exact mode the sequence IS the assertion, and a multi-select hands its options back in list order rather than click order. The move controls appear only under those two modes.

Two things the pickers needed to be usable at all:

The tool list is no longer just the suite agent's. A case can enter on a different agent, and the negative form of a routing assertion names the signature tool of the agent the conversation must stay away from -- by definition someone else's. Tools are now cached per agent id and unioned across the suite agent, the entry agent and the involved agents, kept current by one effect rather than a call at each of the five places those fields can change.

Every picker keeps a manual escape hatch, and a stored value the picker cannot offer switches its own row to the text box instead of being quietly dropped on the next save. Neither catalogue is the whole truth: MCP tools are discovered at run time, and a stored case can name a function its agent has since lost.

The manual-entry flags are editor-only -- buildPayload lists the fields the backend has, so they never reach the API.

A tool name and an agent name are both things an author has in front of them and
cannot spell from memory, and both fail silently when mistyped: a toolCalled on a
function that does not exist can never pass, and a routedToAgent on a misspelt agent
fails every run while reading like a real regression. The datalist on the tool fields
only suggested -- it never stopped a typo.

- toolCalled / toolNotCalled target, and a mock's function name, are now searchable
  pickers over the tool catalogue.
- routedToAgent expected is an agent picker. It stores the name, not the id: both are
  accepted (AgentChainHop.Matches), and the name is what the run report prints in the
  expected and actual columns.
- agentChain expected is an ordered chip list -- add from a picker, remove, move.
  Ordered rather than a multi-select because in `ordered` and `exact` mode the
  sequence IS the assertion, and a multi-select hands its options back in list order
  rather than click order. The move controls appear only under those two modes.

Two things the pickers needed to be usable at all:

The tool list is no longer just the suite agent's. A case can enter on a different
agent, and the negative form of a routing assertion names the signature tool of the
agent the conversation must stay away from -- by definition someone else's. Tools are
now cached per agent id and unioned across the suite agent, the entry agent and the
involved agents, kept current by one effect rather than a call at each of the five
places those fields can change.

Every picker keeps a manual escape hatch, and a stored value the picker cannot offer
switches its own row to the text box instead of being quietly dropped on the next
save. Neither catalogue is the whole truth: MCP tools are discovered at run time, and
a stored case can name a function its agent has since lost.

The manual-entry flags are editor-only -- buildPayload lists the fields the backend
has, so they never reach the API.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants