Repository navigation
feat: bundle, training, task run and agent picks understand always_on and user sims (FD-5371) - #565
Open
sarah-chitty-1 wants to merge 6 commits into
Open
sarah-chitty-1 wants to merge 6 commits into
sarah-chitty-1 wants to merge 6 commits into
Conversation
…t in episodes The always_on step (FD-5364) runs the agent under test like prompt_agent and names its user sim through a user_sim step. Code that looks only for prompt_agent found no solver in such a task. Every change keys on the type name, so it needs none of the new classes. - bundle run --agent and training: solver_steps counts always_on steps and leaves out the agent a user_sim step names (its agent_name, else its name); played_deploys and training's first prompt count always_on too. - task run prints an always_on step's prompt response as it does prompt_agent's. - Episodes: always_on and user_sim refuse an agent played from outside for now, and the default played agent can be an always_on step's, so the refusal names the step. The registry test allows the two types before their classes are registered. Nothing changes for tasks without these step types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loys a user sim The run-wide a2a_agent_id override swapped every deploy_agent and install_agent step, the user sim's too. When a deploy step deploys a user sim (a prompt_agent's user_agent_name, or the agent a user_sim step names), task run now moves the pick onto the tested agent's deploy steps as their step_params, which those steps already prefer. A step's own step_params pick still wins. Tasks without a user sim keep the run-wide pick. user_sim_agents, in bundle/plan.py, names a task's user sims for solver_steps and the CLI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… user sim's; one rule for CLI and hub A run-wide user_overrides.a2a_agent_id (task run --a2a-agent-id, a hub run's agent pick) was applied by every deploy step, including the one that deploys the user sim's agent, so a hub run with an agent pick played the user with the tested agent. The CLI narrowing added in 010b041 (_narrow_agent_pick, moving the pick into the tested agent's step_params) didn't cover the hub, and it also changed the version a picked agent ran at: a pick in step_params runs the agent's latest version, not the deploy step's stored a2a_agent_version. The rule now lives in deploy_agent (_resolve_agent_ref), and install_agent follows it: the run-wide pick is not applied to the agent the run's task names as its user simulator (bundle.plan.user_sim_agents: a user_sim step's agent_name, else its name; a prompt_agent's user_agent_name), when the task also deploys a tested agent (solver_steps). That agent keeps its stored a2a_agent_id and version; its own step_params still win. The task is read from the run's recorded instance (task id and version); a run with no recorded instance, or whose task can't be read (logged), keeps the pick everywhere, as before. Where the pick applies, today's version rule holds (the run's a2a_agent_version, else the step's stored version). The CLI narrowing is removed, so --a2a-agent-id stays run-wide in user_overrides and deploy_agent decides. Behaviour change for prompt_agent tasks: a CLI or hub run with an agent pick, of a task whose deploy_agent (or install_agent) step deploys the agent a prompt_agent step names as user_agent_name (default "human_agent"), no longer swaps that agent for the picked one; it runs the agent the task stores. Tasks that deploy no such agent are unchanged. On the CLI path the tested agent now runs at its deploy step's stored version again, as before 010b041. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…user agent name keeps its own agent Task._run replaces seed placeholders before steps run, so a deploy step named <user> runs as, say, linda. The stored task was read without the seed, so linda never matched the prompt_agent's <user> and the run-wide pick swapped the user's agent too. Apply the run's seed the same way before comparing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pick; prompt_agent tasks are unchanged A prompt_agent task's user_agent_name agent takes a run-wide agent pick exactly as on main, so other teams' runs don't change. The pick is kept off the user's agent only in tasks with a user_sim step (the always-on step's user). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by step class, not type strings Per review: the user sim's deploy becomes its own step (deploy_user_sim_agent, #567), whose class says it plays the user and ignores a run-wide agent pick itself. So deploy_agent and install_agent no longer re-read the stored task mid-run to keep the user sim's agent out of a pick; their agent ref is back to main's. - task_step/roles.py: prompts_agent_under_test (prompt_agent, always_on) and plays_user (the step class's plays_user, looked up through the registry for a config; False until #567 adds it). - solver_steps / user_sim_agents, played_deploys, the episode's trainer agents and refusals, training's prompt pick and task run's response printing use them. The user_sim type special cases are gone. - prompt_agent tasks behave as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Tools that look for "the agent under test" only knew
prompt_agent. This makes them find the agent analways_onstep runs. It also stops a run-wide agent pick from swapping the agent that plays the user.Change
bundle run --agentand training find the agent analways_onstep runs, as they do forprompt_agent. They never pick the user sim's agent.prompt_agentoralways_on. It "plays the user" if its class says so (plays_user, added with the user sim's deploy step in feat(user_sim): the built-in user sim for always-on tasks #567). Both checks live in one small module,task_step/roles.py.task runprints analways_onstep's response, as it does forprompt_agent.always_onstep, refuses an agent played from outside, for now.deploy_agentandinstall_agentare unchanged from main. The user sim's deploy step (feat(user_sim): the built-in user sim for always-on tasks #567) keeps its own agent itself.Behaviour change for existing users
None.
prompt_agenttasks work exactly as on main, including a run-wide agent pick, which still swaps every agent they deploy.Tests
run_wide_agent_pick_test.py: a run-wide pick still reaches everydeploy_agentandinstall_agent, as on main;plays_user_step.py), used to check the bundle and training solver lookup,played_deploysand the episode rules never treat it as the agent under test;prompt_agenttasks give the same solver and user agents as on main;Affected folders pass. The 2 known
test_mcp_cli_builderfailures are on main too.Linear: FD-5371.
🤖 Generated with Claude Code
The PR appears safe to merge; no new blocking issue was found.Summary
The PR adds shared step-role checks so bundle and training lookups recognize
always_onand exclude agents deployed by classes markedplays_user.task runshows responses from always_on steps.Reviews (12) · Last reviewed commit: "always_on plumbing: tell the agent under..." · Reviewed by Greptile