[kaihackney-scale]

What

prompt_agent gets an opt-in user_sim_gate. When the user side is an LLM user-sim, each of its messages is
checked before the agent receives it:

• patterns (free): phrases and shapes that belong to the assistant's side, such as a report of work
  done, a formatted deliverable, offers to do the work, or service phrases.
• judge_model (optional): one model call over the last turns and the candidate. It decides only whose
  voice the message is in. On its own it fails only messages of judge_min_chars or more; shorter ones
  also need a pattern.

A message that fails isn't sent. The user-sim is told why and asked to rewrite, up to max_rewrites
times. If it still fails, on_fail decides what happens:

• fail (the default) ends the conversation and fails the step, after the PromptResponse is recorded;
• end ends the conversation only;
• send records the verdicts and sends the message anyway.

Every verdict is recorded in context.metadata["user_sim_gate"][step_id], and an ended run is recorded in
context.metadata["user_sim_invalid"].

A step type that subclasses PromptAgentTaskStep but runs its own conversation would otherwise accept user_sim_gate and ignore it. Instead it refuses the setting at construction, unless it applies the gate and sets USER_SIM_GATE_SUPPORTED = True. For that, the check-and-rewrite loop is user_sim_gate.run_gate(gate, text, history, rewrite): rewrite(note) returns the user-sim's new message, or None if the user side ended. prompt_agent uses the same function.

{"user_sim_gate": {"judge_model": "<model>", "max_rewrites": 2, "on_fail": "fail"}}

Why

An LLM playing the user can slip into the assistant's role. It answers its own question, reports work as
done, or offers to do the agent's job. After that, the rest of the run no longer tests the agent. Checking
each message, with a rewrite, keeps those messages out of the conversation, whether or not the user-sim
cooperates.

How it was tested

• Unit tests:
  • tst/unit/task_step/test_user_sim_gate.py: the patterns, judge parsing, the rendered history and the
    rewrite note.
  • tst/unit/task_step/test_prompt_agent_user_side.py: 8 new tests over the real _UserSide request flow:
    • pass, rewrite and recover;
    • fail after the rewrites, both fail and end;
    • the judge failing a message the patterns pass;
    • a judge error falling back to the patterns;
    • a short message not failed by the judge alone;
    • dict round-trip and validation;
    • run_gate on its own (rewrite, give up, send, stop);
    • a subclass with its own loop refusing the gate unless it supports it.
• Offline, against labels of real user-sim messages (two independent model labelers, with an adjudicator for
  disagreements, checked against known cases):
  • every labeled out-of-role message is caught;
  • under 0.5% of in-role messages are failed;
  • on held-out messages the patterns alone flag 0.22% of in-role messages.
• Rewrite recovery: at real out-of-role points, asked to rewrite, the user-sim recovered every time,
  with the next scripted request.
• E2E: gated and ungated runs of the same multi-turn tasks, through the full task (deploy, conversation,
  snapshot).
  • Every run completed.
  • The gate recorded a verdict for every user-sim message.
  • No out-of-role message reached the agent, judged separately by a grader with the scenario.
  • The rewrite path was not triggered live, because nothing failed; the unit tests and the offline
    replay cover it.

ruff check is clean.

No screenshots: the change has no UI.

Off by default: a step without user_sim_gate behaves exactly as before, and to_dict leaves the field out.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

Adds an optional gate that checks simulated-user messages before the agent receives them.

• Prompt-agent user-sim replies can be checked before they reach the agent.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Candidate[Simulated-user message] --> Check[Check patterns and optional judge]
  Check -->|Pass| Send[Send message to agent]
  Check -->|Fail, rewrites remain| Rewrite[Ask user simulator to rewrite]
  Rewrite -->|New message| Check
  Rewrite -->|User ends| End[End conversation]
  Check -->|Fail, no rewrites remain| Policy{on_fail}
  Policy -->|send| Send
  Policy -->|end| End
  Policy -->|fail| Fail[Record response and fail step]

Loading

_{Reviews (3) · Last reviewed commit: "feat(prompt_agent): run_gate for any con..." · Reviewed by Greptile}

[earakely-scale]

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from outside this diff, in case it's useful: there's a second open PR adding a step type that subclasses PromptAgentTaskStep and filters its inherited fields by an explicit list, so user_sim_gate would land on it as a silently-inert field.

• This PR: src/agent_env/task_step/task_steps/prompt_agent.py:181 adds user_sim_gate as a constructor kwarg on PromptAgentTaskStep, serialized at :309-310 / :348, and enforced inside _UserSide (:1063, :1130).
• feat(always_on): a step that runs an assistant over days of task time, with a simulated user (FD-5364) #566 (feat(always_on): a step that runs an assistant over days of task time, with a simulated user, head 974a6aba): src/agent_env/task_step/task_steps/always_on.py:201 is class AlwaysOnTaskStep(PromptAgentTaskStep). Its __init__ takes **prompt_fields and pops a fixed drop list at :217 (max_conversation_turns, user_agent_name, user_a2a_url, post_turn_hooks, user_output_format, user_model, snapshot_config), then forwards the rest to super().__init__. Its to_dict at :303-304 starts from TaskStep.to_dict(self) — not the parent's — and copies only _PROMPT_FIELDS (:252) plus _OWN.

So in either merge order, an always_on step authored with user_sim_gate is accepted at construction (it isn't in the drop list), dropped on to_dict (it isn't in _PROMPT_FIELDS), and never consulted: always_on.py has its own _Run loop and contains no reference to _UserSide, _converse or user_sim_gate. The :588 warning only fires on the prompt_agent path, so there'd be no signal either. Deciding where it goes — a third list entry, the drop list, or out of scope — is probably cheaper now than after both land.

Related, same direction: #567 addresses the same failure mode from the prompt side — user_sim_versions.py builds a system prompt around "You are not the AI assistant … Don't do your assistant's work: never look things up for it or hand over facts it should find", and user_sim.md reports a measured effect (done collapsed into the same message 9/10 → 1/10). Worth a look if you want the gate's pattern list and that prompt to agree on what counts as the assistant's voice.

---

Generated by Claude Code