You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds an opt-in COMPUTER_BACKEND=cua-driver implementation behind the existing computer-service contract. Playwright remains the default.
The Cua Driver backend:
uses semantic browser snapshots and refs for read, click, type, and key actions;
preserves the existing HTTP, WebSocket, token, takeover, shell, and per-Bot APIs;
keeps isolated browser profiles under PROFILES_DIR/cua-driver, separate from Playwright profiles;
uses native window screenshots and native input for takeover actions; and
runs in both the standalone agent-computer image and the all-in-one production image.
The production packaging starts Xvfb and Openbox only for the Cua backend. It keeps Chromium and the display under pwuser, waits for graceful browser/profile shutdown before stopping X11, and builds both container paths in CI.
Where it runs
New state that outlives a request? No new distributed request state. The backend uses the existing per-Bot computer lifecycle. Durable browser data lives in the mounted profile root under PROFILES_DIR/cua-driver; live Cua sessions remain process-local, like the existing live Playwright browser.
What happens on the second replica? Multi-replica behavior is unchanged. Each agent-computer replica owns the browsers and profiles assigned to that deployment instance; the supervisor's per-Bot container model remains the isolation path.
Anything serialised? No new cross-process serialization or database writes. The manager deduplicates concurrent starts for one Bot inside its process, and each isolated named profile remains owned by that computer instance.
Anything fanned out to a browser? The existing gateway and WebSocket screen path are unchanged. Frames and takeover input still use the current computer-service endpoints.
New listener, port, or schedule? None. The computer service remains on port 4100 inside the deployment boundary, and the all-in-one image still exposes only port 3001.
Boundary and audit
Every acting call still goes through the existing gateway: resolve, decide, audit, then act.
Backend refusals and failures return through the existing computer endpoints, so the gateway retains the same audit path.
The backend does not add client-trusted identity or policy inputs. Direct computer-service requests still require COMPUTER_TOKEN.
Changelog
No changelog entry: deployment behavior is unchanged unless an operator explicitly sets COMPUTER_BACKEND=cua-driver. The new configuration and operational differences are documented in the README and deployment, configuration, and architecture guides.
Known limitations
Linux semantic clicks use a background-safe synthetic DOM event. Controls that require a trusted browser gesture may reject it.
Cua profiles do not apply OpenBot's per-Bot egress proxy settings yet; the API reports egress: null for this backend.
Ref-scoped key actions support Enter and printable characters. Arrows and editing keys use the native window route without a ref.
Standalone default smoke: Playwright navigation passed, with no Xvfb or Openbox process started.
All-in-one Cua smoke: Xvfb, Openbox, Bun, and Chromium ran as pwuser; navigation and native screenshot passed.
Graceful shutdown smoke: docker stop --timeout 20 logged SIGTERM: closing the browser so its profile is flushed before X11 exited.
Claude Code Fable reviewed the implementation and final integration delta. All must-fix findings were closed, and its final verdict was READY TO SUBMIT.
Not a review of the Cua backend itself — a note on the CI hunk, which closes something I had found
separately and was about to file. Worth saying so that it does not get filed as a duplicate, and that
the part this does not reach does not disappear with it.
The two lines that matter:
- run: bun install --frozen-lockfile && bun run typecheckworking-directory: agent-computer
agent-computer ships "typecheck": "tsc --noEmit" (agent-computer/package.json:9), and root typecheck is bun run --filter '*' typecheck over workspaces: ["app", "server", "worker"], so that
script had never run in CI — the package holding the only spawn in the deployment. Same for the
lockfile: agent-computer/bun.lock is committed and every install of it was plain bun install, which Dockerfile:24-25 argues against in its own words about Bun's version ("an image built next month is
not the image built today") twenty-three lines above the install it did not apply to. Your comment
gets the reason right too: outside the workspaces deliberately, because it is built as its own image.
I checked it is free rather than assuming — bunx tsc --noEmit in agent-computer is clean on main
today, so nothing was being hidden, and bun install --frozen-lockfile there resolves 63 packages, so
the lockfile is valid as committed.
What the same reasoning still leaves open, after this lands:
typecheck
--frozen-lockfile
agent-computer
this PR
this PR (Dockerfile:49, agent-computer/Dockerfile:15)
supervisor
still nothing — supervisor appears nowhere in ci.yml
still bun install (supervisor/Dockerfile:14)
agent-bot
no script to run
still bun install (agent-bot/Dockerfile:9)
agent-langgraph
no script to run
still bun install (agent-langgraph/Dockerfile:9)
supervisor is the closer parallel: it has the identical "typecheck": "tsc --noEmit"
(supervisor/package.json:9), it is equally outside workspaces, and it is the only thing in the
deployment holding a Docker socket (supervisor/Dockerfile:1). Both of its gates are also free today — bunx tsc --noEmit clean, --frozen-lockfile resolves 73 packages. #40 touches agent-bot/Dockerfile but only adds USER bun, so that install line is untouched by anything open.
I am not suggesting you widen this PR — the Cua backend is the change, and supervisor has nothing to
do with it. Happy to send the remainder as its own small PR once this is in, so it rebases onto your ci.yml hunk rather than racing it. If you would rather carry it here, that is fine too; I would just
rather it be decided than dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Adds an opt-in
COMPUTER_BACKEND=cua-driverimplementation behind the existing computer-service contract. Playwright remains the default.The Cua Driver backend:
PROFILES_DIR/cua-driver, separate from Playwright profiles;agent-computerimage and the all-in-one production image.The production packaging starts Xvfb and Openbox only for the Cua backend. It keeps Chromium and the display under
pwuser, waits for graceful browser/profile shutdown before stopping X11, and builds both container paths in CI.Where it runs
PROFILES_DIR/cua-driver; live Cua sessions remain process-local, like the existing live Playwright browser.agent-computerreplica owns the browsers and profiles assigned to that deployment instance; the supervisor's per-Bot container model remains the isolation path.Boundary and audit
COMPUTER_TOKEN.Changelog
COMPUTER_BACKEND=cua-driver. The new configuration and operational differences are documented in the README and deployment, configuration, and architecture guides.Known limitations
egress: nullfor this backend.Proof
Candidate:
53e5a8868ee85ea5131ea89d6353c1685b849eabagent-computertypechecks passed.agent-computer: 99 tests passed.agent-computer/Dockerfileand the root productionDockerfilebuilt successfully from the candidate.pwuser; navigation and native screenshot passed.docker stop --timeout 20loggedSIGTERM: closing the browser so its profile is flushedbefore X11 exited.READY TO SUBMIT.