Skip to content

Ask for the wheel by calling for it, not by asking in words - #80

Open
rhlsthrm wants to merge 1 commit into
CopilotKit:mainfrom
rhlsthrm:ask-for-the-wheel-by-calling-for-it
Open

Ask for the wheel by calling for it, not by asking in words#80
rhlsthrm wants to merge 1 commit into
CopilotKit:mainfrom
rhlsthrm:ask-for-the-wheel-by-calling-for-it

Conversation

@rhlsthrm

Copy link
Copy Markdown

What this changes

Handing the browser over is a tool call. A sentence in the transcript is not one: nobody is offered the wheel, the person has no way to help, and the page stays exactly where it was.

The guidance already said to call computer_request_help at a sign-in, which is correct but was not emphatic enough to beat the obvious-looking alternative. Observed on a running deployment, asked to file an issue on GitHub while signed out, three attempts in a row:

  1. "You need to be signed in to GitHub to file an issue. Please sign in, and I'll continue... Would you like to proceed with signing in now?"
  2. On "Yes, please proceed": "Please enter your GitHub username or email address, and password on the sign-in page. Once signed in, let me know."
  3. Only when told "call computer_request_help" explicitly did the takeover appear.

The second one is the serious one. The prompt already forbids asking to be told a secret, and the model obeyed that narrowly while asking the person to type their password into a page they cannot see and are not looking at — which is both useless and exactly the habit this product exists to remove.

So the guidance now says calling it is what asking means, names the sentences that are not it, and gives the reason rather than only the rule: the person is not looking at this page and cannot type into it until control is handed over. The tool description says the same, since that is what a model weighs when it chooses between talking and calling.

No code paths change — the prompt and one tool description.

Where it runs

  • New state that outlives a request? None. Prompt text and a tool description.
  • What happens on the second replica? Identical: both are static strings compiled into the Bot and the app.
  • Anything serialised? No.
  • Anything fanned out to a browser? No. The takeover this makes reachable uses the existing control endpoints and the existing ComputerView prompt.
  • New listener, port, or schedule? None.

Boundary and audit

  • Every acting call still goes through the gateway. Unchanged. computer_request_help was always registered and always went through the same path; this only makes a Bot actually reach for it.
  • New refusals and new failures each write a row. No new class. computer.help_requested, computer.control_taken and computer.control_released are existing events that now get written where previously a Bot wrote a sentence and stopped.
  • Nothing new is trusted from the client.

Changelog

  • A line under UnreleasedFixed.

Proof

Biome lint and format clean on both files.

On a running deployment, fresh thread, one plain request and no coaching — "File an issue on github.com/CopilotKit/openbot titled 'Test from OpenBot' with a one-line body." The Bot navigated, snapshotted, hit the sign-in wall and called the tool on its own:

The assistant needs you. I need you to sign in to GitHub so I can file an issue on the CopilotKit/openbot repository. [Take control]

09:40:05 | computer.action_allowed | general-assistant | computer_navigate
09:40:15 | computer.help_requested | general-assistant | I need you to sign in to GitHub so I can file an issue on the CopilotKit/openbot repository.

Ten seconds from the page load to the wheel being offered, where three exchanges previously ended in a Bot asking for a password.

Composes with #79 but does not depend on it: on main this reaches the two shipped Bots, which already import the guidance. #79 is what puts the same guidance in front of built-in agents, and the run above is a built-in agent with both applied.

Handing the browser over is a tool call. A sentence in the transcript is
not one: nothing is offered, nobody can help, and the page stays where it
is. The guidance said to call computer_request_help at a sign-in, which
is true but not emphatic enough to beat the obvious-looking alternative,
so Bots wrote "please sign in and let me know" and stopped.

Worse, one went on to ask for a username and password to be typed into a
sign-in page the person cannot see, which the prompt already forbids in
the narrow form of telling the Bot a secret.

So: calling it IS how you ask, the sentences that are not it are named,
and the reason is given — the person is not looking at this page and
cannot type into it until control is handed over. The tool description
says the same thing, since that is what the model weighs when choosing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant