Skip to content

Wait before rebuilding a computer that just lost a race - #78

Open
rhlsthrm wants to merge 1 commit into
CopilotKit:mainfrom
rhlsthrm:pause-before-rebuilding-a-computer
Open

Wait before rebuilding a computer that just lost a race#78
rhlsthrm wants to merge 1 commit into
CopilotKit:mainfrom
rhlsthrm:pause-before-rebuilding-a-computer

Conversation

@rhlsthrm

Copy link
Copy Markdown

What this changes

Creating a computer and starting it are two calls to Docker. A container name the daemon has not published yet answers the second with a 404, and ensure already reads that correctly as a lost race whose answer is to rebuild.

It retried immediately, though, and with ATTEMPTS = 2 the retry is the entire budget: it arrived within a millisecond, inspectOwned still saw nothing, the create came back 409 because the name was in fact taken, and start returned 404 again with no attempt left. The whole request then failed as DockerUnavailableError, which is surfaced to the person as the computer not being startable and recorded as computer.action_failed.

The symptom is that the first browser action a Bot is ever asked for fails, and the same request seconds later works, so it reads as a flaky Bot rather than as a race. On this machine (OrbStack) it hit two of three Bots on their first ever action.

One poll interval before rebuilding fixes it — 250 ms, which is what waitUntilAnswering already uses for the same question about the same container. Both sleeps now share a pause helper.

Where it runs

  • New state that outlives a request? None. The change is a delay inside one ensure call.
  • What happens on the second replica? Unchanged, and slightly better. Two replicas asking for one Bot's computer concurrently is the race this retry path exists for: the loser now looks again after the winner's create has settled instead of before it, which is the difference between rebuilding successfully and reporting Docker as unreachable. Idempotency still rests on the container name and the ownership label, not on timing.
  • Anything serialised? Nothing new. Concurrent ensure calls are still resolved by Docker refusing a duplicate name with 409, which this path already treats as success.
  • Anything fanned out to a browser? No.
  • New listener, port, or schedule? None. No new timer either: one bounded await on an existing retry path.

Boundary and audit

  • Every acting call still goes through the gateway. Untouched: this is below it, in provisioning.
  • New refusals and new failures each write a row. No new class. The failure this removes was already written as computer.action_failed with the Docker message, which is how it was diagnosed.
  • Nothing new is trusted from the client. Nothing here reads a request.

Changelog

  • A line under UnreleasedFixed.

Proof

Before, on a first-ever action for a Bot, from audit_events:

computer.action_allowed  general-assistant  computer_navigate  https://example.com
computer.action_failed   general-assistant  computer_navigate  The supervisor could not reach Docker
  (Error: (HTTP code 404) no such container - No such container: openbot-computer-general-assistant).
  A computer cannot be started without it.

Then the same request a minute later succeeded, with the container present. The same pair appeared for risk-analyst on its first action.

After, on this commit: rebuilt the supervisor, deleted openbot-computer-risk-analyst outright, and asked that Bot to open a page. The container was created and started on the first attempt, the page rendered in the live screen, and audit_events has computer.action_allowed with no failure row beside it. A second Bot (knowledge) also provisioned first time.

Creating a computer and starting it are two calls to Docker. A name the
daemon has not published yet answers the second with a 404, which the
supervisor already reads correctly as a lost race and answers by
rebuilding.

It went straight back round, though, and the retry is the whole budget:
it arrived a millisecond later, saw the same unpublished name, and spent
the second attempt on the state that failed the first. The request then
failed as Docker being unreachable, so the first browser action a Bot was
ever asked for was refused with "no such container", and the next message
seconds later worked.

One poll interval first, which is what waitUntilAnswering already uses
for the same question. The two sleeps share a helper.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant