Skip to content

Report a surface-owned tool call to the surface - #77

Open
rhlsthrm wants to merge 1 commit into
CopilotKit:mainfrom
rhlsthrm:tell-the-surface-about-a-tool-call
Open

Report a surface-owned tool call to the surface#77
rhlsthrm wants to merge 1 commit into
CopilotKit:mainfrom
rhlsthrm:tell-the-surface-about-a-tool-call

Conversation

@rhlsthrm

Copy link
Copy Markdown

What this changes

agent-langgraph ends a run when the model calls a tool the surface owns, which is the right shape and is documented as such in the conditional edge: the run finishes, the browser acts, and the surface starts the next run with the result in hand.

The TOOL_CALL_* events were only sent from the branch watching on_chain_end for the tools node — the node that executes this deployment's own tools, and precisely the node an ending run skips. So the assembled call sat in pending and was never emitted. The run was a clean RUN_STARTED/RUN_FINISHED pair carrying nothing at all: the person's message with no answer under it, no tool call for the surface to execute, and no explanation, because an empty run is not an error by the protocol and the stopped-turn banner only fires on RUN_ERROR.

Every browser action was affected, since all of them are useFrontendTool registrations: opening a page, filling a form, asking for help at a sign-in. As the default /bot agent is risk-analyst, which points at this Bot, that is the first thing a fresh clone tries.

The fix drains pending before RUN_FINISHED, emitting TOOL_CALL_START/ARGS/END and no result — producing the result is the surface's half, and it begins the next run holding it. finalMessage is removed with it: it was written twice and never read, which is what was left behind when the events it was accumulated for were not sent.

Where it runs

  • New state that outlives a request? None. pending is per-run, inside the stream's own closure, and is now emptied before the run ends rather than being left populated.
  • What happens on the second replica? No change. A run is one request against one Bot process, and the tool call now leaves in that response instead of being dropped in it. Nothing is read back later.
  • Anything serialised? Nothing new.
  • Anything fanned out to a browser? Only the AG-UI stream that already answers this request, over events the protocol already defines.
  • New listener, port, or schedule? None.

Boundary and audit

  • Every acting call still goes through the gateway. This adds no execution path: it reports a call the surface then executes through the gateway exactly as it does for a built-in agent, which is where resolve/decide/audit happen. Before this, the browser action never reached the gateway at all, so nothing was recorded because nothing was attempted.
  • New refusals and new failures each write a row. No new refusal or failure class; a refused action now reaches the point where it can be refused and recorded.
  • Nothing new is trusted from the client. The events carry the call the model made, assembled by the framework in this process.

Changelog

  • A line under UnreleasedFixed.

Proof

Local stack on this commit, COMPUTER_SUPERVISOR_URL set so each Bot gets its own container.

Before, three times over, on /bot (default agent, so risk-analyst): "Open news.ycombinator.com and tell me the top story", "Use your browser to open https://example.com and tell me the exact h1 text", and a third variation. Each produced no assistant message, no computer.* row in audit_events, and no computer container. Plain prose turns on the same Bot answered normally, so the transport was fine.

After: same Bot, "Open https://example.com on your computer and quote its h1 exactly" → the live screen renders the page and the answer is "Example Domain". audit_events has the matching row:

07:58:44 | computer.action_allowed | risk-analyst | computer_navigate | https://example.com

and no failure row beside it. openbot-computer-risk-analyst was deleted before the run, so that container was also built from nothing during it.

agent-langgraph ends a run when the model calls a tool the surface owns,
rather than inventing a result for it. That is the right shape: the run
finishes, the browser acts, and the next run carries the result back.

But the TOOL_CALL_* events were only sent from the branch watching the
node that executes this deployment's own tools, and an ending run never
enters that node. The assembled call sat in `pending` and was dropped, so
the run was a clean RUN_STARTED/RUN_FINISHED pair carrying nothing: no
answer under the person's message, no call for the surface to execute,
and no error, because an empty run is not one.

Drained before the run finishes now, without a result. Producing that is
the surface's half, and it begins the next run holding it.

`finalMessage` goes with it: it was written twice and never read, which
is what was left after the events it was collected for were not sent.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant