diff --git a/AGENTS.md b/AGENTS.md index 8d99e82cf..2188ae130 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -139,7 +139,7 @@ and deserialized through a registry. | `agents/` | The `agents` bundle: the agents agent-env ships, one folder each under `agents/agents/`, with a smoke task each under `agents/tasks/`. Its Claude Code, `@local/agentenv-framework/agents/claude-code`, is the default agent. An agent's Python packages are its folder's `requirements.txt`, installed in its image, never in agent-env's own environment. A run that pulls one of its agents reads the whole bundle, so `tst/unit/bundle/shipped_bundles_test.py` parses and checks every shipped bundle. | | `computers/` | The `computers` bundle: the base computers agent-env ships as `computer` artifacts, `ubuntu-24.04` (XFCE on Xvfb, supervisord as the session user, settings as `AGENTENV_SETTING_`), and `omarchy-4.5`, a disk alone built on the composing machine from Omarchy's own VM image (its `[forms.disk]` `source`) by `disk/omarchy/`'s recipe, its services systemd user units of the graphical session that start through `with-seed`, booted by the launcher as a `linux-desktop` guest (`env/envs/computer/utils/omarchy_disk.py`, on `thin_disk.py`, which Windows' disks use too), aliased through `[artifacts]`. It has no tasks either. `disk/linux/` is the Linux disk form's recipe: `boot/`, whose Dockerfile makes a disk form (a GPT disk of an EFI system partition holding one UKI, Ubuntu's kernel with a busybox initramfs whose init reads a NoCloud seed and `switch_root`s into the image's entrypoint, and an ext4 root filesystem that is a container image's, plus `/etc/agentenv/vm/image.env` from its OCI config), `tools/`, the disk tools that make it unprivileged on the builder's own platform (`mkfs.ext4 -d`), and `launcher/`, the QEMU launcher that boots one under KVM. A base's `[forms.disk]` with no image is made from its container form; the composer makes each arch's disk from its composed container, keeping the base disk's EFI system partition, keyed by the container's key, the base disk and the recipe; gate 3 probes it where a provider boots disks natively (`disk_image_platforms()`), and a deploy's first boot with KVM probes one left `unverified`; a deploy naming `form = "disk"` boots it through `create_vm(image=..., env=...)`, its settings in the seed. | | `browsers/` | The `browsers` bundle: the website browsers agent-env ships, today `chromium`, a `computer_app` env (Chrome for Testing at the build Playwright pins, for amd64 and arm64, set up for unattended use by managed policies in `/etc/opt/chrome_for_testing/policies/managed`, opened at boot on its `start_urls`, its profile its data). All of it, its pin, install step, policies and command, is in `envs/chromium/chromium.py`; `tst/integration/computer/chromium_app_test.py` runs it. Like `drivers`, it has no tasks. | -| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. | +| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, the page driver aliased `page`, a browser computer's, whose screen is the browser's active tab and which serves Playwright MCP's `browser_*` tools beside `computer_*` from a Playwright MCP it runs attached to the browser over a CDP address the relay issues it (`browser_evaluate` and `browser_run_code_unsafe` hidden by default), and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. | | `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker, and `desktop`, a computer env of `ubuntu-24.04` and `reference` with a probe task and an agent task). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, ` = ""` shipping `src///`. | ### Training helpers diff --git a/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py b/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py index 4f5f0f70a..90d50fd51 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py @@ -245,12 +245,14 @@ def deco(fn: Callable) -> Callable: class ForwardedTool(NamedTuple): """A tool an environment serves on another server's behalf, as that server lists it: its descriptor, its own schema included, its MCP ``annotations`` and its ``output_schema``; ``call`` takes the call's arguments as keywords - and returns the other server's ``CallToolResult``, which reaches the caller as it is.""" + and returns the other server's ``CallToolResult``, which reaches the caller as it is. ``hidden_by_default`` hides it + as ``@tool(hidden_by_default=True)`` hides a tool of the environment's own.""" tool: EnvironmentTool call: Callable[..., Any] annotations: dict | None = None output_schema: dict | None = None + hidden_by_default: bool = False class _ResourceMarker(NamedTuple): @@ -450,7 +452,8 @@ def _register_forwarded(self, app: Any, descriptor: EnvironmentTool, fn: Callabl fn_metadata=FuncMetadata(arg_model=_passed_through(), output_schema=schema, output_model=_as_it_came() if schema is not None else None), is_async=True, - annotations=_mcp_tool_annotations(forwarded.annotations) if forwarded.annotations is not None else None) + annotations=_mcp_tool_annotations(forwarded.annotations) if forwarded.annotations is not None else None, + meta={HIDDEN_BY_DEFAULT_META_KEY: True} if forwarded.hidden_by_default else None) def _register_resource(self, app: Any, resource: _ServedResource) -> None: """A resource as it is, its description absent when it has none, or a template through FastMCP's own @@ -1283,6 +1286,8 @@ async def call(**arguments: Any) -> Any: return await forwarded.call(**arguments) setattr(call, _FORWARDED_ATTR, forwarded) + if forwarded.hidden_by_default: + setattr(call, _HIDDEN_BY_DEFAULT_ATTR, True) return call diff --git a/packages/agentenv-protocol/tests/test_tool_access.py b/packages/agentenv-protocol/tests/test_tool_access.py index c2d520e27..04ef1a378 100644 --- a/packages/agentenv-protocol/tests/test_tool_access.py +++ b/packages/agentenv-protocol/tests/test_tool_access.py @@ -17,6 +17,7 @@ from mcp.types import CallToolResult, TextContent # noqa: E402 from agentenv_protocol import ( # noqa: E402 + HIDDEN_BY_DEFAULT_META_KEY, ROLE_HEADER, ROLE_META_KEY, AgentEnvEnvironment, @@ -140,6 +141,34 @@ async def click(**arguments): assert agent_tools == ["read", "write"] and refused.isError +@pytest.mark.asyncio +async def test_a_forwarded_tool_hidden_by_default_stays_hidden_through_every_tool_rule_until_a_rule_names_it(): + class Forwarding(_Env): + def forwarded_tools(self): + async def evaluate(**arguments): + return CallToolResult(content=[TextContent(type="text", text="evaluated")]) + return [ForwardedTool(EnvironmentTool(name="evaluate", inputSchema={"type": "object"}), evaluate, + hidden_by_default=True)] + + with anyio.fail_after(30): + async with _served(Forwarding()) as (rest, session): + async with session("agent") as agent: + listed, refused = await _names(agent), await agent.call_tool("evaluate", {}) + await rest.post("/tools/enable", json={"role": "agent", "tools": "*"}) + async with session("agent") as agent: + after_every_tool = await _names(agent) + await rest.post("/tools/enable", json={"role": "agent", "tools": ["evaluate"]}) + async with session("agent") as agent: + shown = {t.name: t for t in (await agent.list_tools()).tools} + called = await agent.call_tool("evaluate", {}) + async with session() as anyone: + anyone_tools = await _names(anyone) + assert listed == after_every_tool == anyone_tools == ["read", "write"] + assert refused.isError and refused.structuredContent["code"] == "permission_denied" + assert sorted(shown) == ["evaluate", "read", "write"] and not called.isError + assert shown["evaluate"].meta == {HIDDEN_BY_DEFAULT_META_KEY: True} and called.content[0].text == "evaluated" + + @pytest.mark.asyncio async def test_everything_off_then_one_tool_on_for_one_role(): with anyio.fail_after(30): diff --git a/src/agent_env/drivers/README.md b/src/agent_env/drivers/README.md index bae467356..0d30899f6 100644 --- a/src/agent_env/drivers/README.md +++ b/src/agent_env/drivers/README.md @@ -21,6 +21,17 @@ a tab that paints nothing shows white. It serves the page group too: `computer_n it. The browser ending on its own fails the computer for good: its ready check is `final`, and its tools answer with why. Name it as `page`, or by its id, `@local/agentenv-framework/drivers/page`. +Beside those it serves Playwright MCP's `browser_*` tools (0.0.84, `--caps vision`, on Node 24), as +`playwright-mcp-tools.json` lists them. Its install lays Node and Playwright MCP's three npm packages out from pinned +downloads, and it runs Playwright MCP itself, attached to the browser over a CDP address the browser's relay issues to +the user `playwright-mcp`, so the guard applies to it as to any CDP client; it never starts a browser of its own. Each +caller gets a session of its own there. It starts on the first call, again once it has exited, and again on a fresh +address after the browser's data is reset; it ends with the driver, gets none of the driver's environment, and its file +access is `~/Uploads`. `browser_evaluate` and `browser_run_code_unsafe`, which run the agent's own code, are hidden by +default: a task's tool access shows them by naming them. To move to another version, update the downloads in +`env.toml` and regenerate the manifest from that version's `tools/list`; the browser computer's integration test checks +the two agree. + A run that names it writes it: its files, what its class declares for each platform it supports, read without `mss`, `python-xlib` or PyAV installed, and how a computer installs it. Its `env.toml`'s `[install.linux]` has a computer env's composer put the folder in `/opt/agentenv/drivers/reference/`, install `requirements.txt`, resolved and diff --git a/src/agent_env/drivers/envs/page/driver.py b/src/agent_env/drivers/envs/page/driver.py index fdbc8ae68..9b140492f 100644 --- a/src/agent_env/drivers/envs/page/driver.py +++ b/src/agent_env/drivers/envs/page/driver.py @@ -1,20 +1,31 @@ """The page driver, for a computer with no display whose one website browser runs alone: its screen is the browser's active tab. A screenshot is the page, and input reaches the page as CDP input events, both through the browser's own server on its socket (``browser/v1``'s ``page/*`` operations), since only that server holds the browser's debugging pipe. -The browser ending on its own is an infrastructure failure, since nothing here launches it again. Needs -``agentenv-framework-protocol[computer]`` where it runs.""" +The browser ending on its own is an infrastructure failure, since nothing here launches it again. Beside its own tools it +serves Playwright MCP's ``browser_*``, from a Playwright MCP it runs attached to the browser over a CDP address the +browser's relay issues to it. Needs ``agentenv-framework-protocol[computer]`` where it runs.""" from __future__ import annotations import asyncio import base64 +import contextlib +import functools +import json +import logging import math +import os +import socket import string +import tempfile from pathlib import Path -from typing import Any, Optional +from typing import Any, Awaitable, Callable, Optional import httpx +from mcp import ClientSession +from mcp.client.streamable_http import streamable_http_client +from mcp.types import CallToolResult, TextContent -from agentenv_protocol import client +from agentenv_protocol import EnvironmentTool, ForwardedTool, ToolContext, client from agentenv_protocol.computer import ( ComputerDriverEnvironment, DeviceUnavailable, @@ -80,6 +91,26 @@ BLANK = b"\xff\xff\xff" #: How long one call to the browser's server may take. CALL_TIMEOUT_S = 30.0 +#: Playwright MCP, as its install lays it out beside the driver, and its tools as 0.0.84 lists them with ``--caps +#: vision``, which an integration test checks against the one installed. +HERE = Path(__file__).resolve().parent +NODE = HERE / "node" / "bin" / "node" +PLAYWRIGHT_MCP = HERE / "node_modules" / "@playwright" / "mcp" / "cli.js" +PLAYWRIGHT_MCP_TOOLS = HERE / "playwright-mcp-tools.json" +#: Playwright MCP's tools that run the agent's own code, in the page and in Playwright MCP's own process: hidden until +#: a task's tool access names them. +HIDDEN_TOOLS = frozenset({"browser_evaluate", "browser_run_code_unsafe"}) +#: Whom the browser issues Playwright MCP's CDP address to, as its record names the connections. +PLAYWRIGHT_MCP_USER = "playwright-mcp" +#: What Playwright MCP's process keeps of the driver's environment, which holds the deploy's tokens and settings. +KEPT_ENVIRONMENT = ("PATH", "HOME", "USER", "LOGNAME", "LANG", "LC_ALL", "TZ", "TMPDIR") +#: Where Playwright MCP writes what it names itself, and its log. +PLAYWRIGHT_MCP_OUTPUT = Path(tempfile.gettempdir()) / "agentenv-playwright-mcp" +#: How long Playwright MCP may take to listen, and one of its tools to answer. +PLAYWRIGHT_MCP_START_S = 30.0 +PLAYWRIGHT_MCP_CALL_TIMEOUT_S = 300.0 + +logger = logging.getLogger(__name__) @computer_driver(name="page", supports=[{"os": {"type": "linux", "display": ["none"]}, "arch": ["amd64", "arm64"]}]) @@ -94,7 +125,8 @@ class PageDriver(ComputerDriverEnvironment): "around it. Its screen is the page's viewport. Go to a URL with computer_navigate, back, forward or " "reload with computer_history, and list, open, switch or close tabs with computer_tabs. Each screenshot " "names the page, and an open dialog, which computer_dialog answers, or file chooser, which " - "computer_upload gives files in ~/Uploads.") + "computer_upload gives files in ~/Uploads. Its browser_* tools, Playwright MCP's, act on the same " + "browser by the page's accessibility snapshot; browser_file_upload takes files in ~/Uploads too.") def __init__(self) -> None: super().__init__() @@ -102,6 +134,25 @@ def __init__(self) -> None: self._pointer = (0.0, 0.0) self._buttons = 0 self._held: list[str] = [] + self._playwright = _PlaywrightMcp(self._playwright_endpoint) + + def forwarded_tools(self) -> list[ForwardedTool]: + """Playwright MCP's tools, as its manifest lists them, each called in the caller's own session there; those + that run the agent's own code hidden until a task's tool access names them.""" + return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]), + functools.partial(self._playwright_call, t["name"]), t.get("annotations"), + t.get("outputSchema"), hidden_by_default=t["name"] in HIDDEN_TOOLS) + for t in json.loads(PLAYWRIGHT_MCP_TOOLS.read_text(encoding="utf-8"))["tools"]] + + async def _playwright_call(self, tool: str, /, **arguments: Any) -> CallToolResult: + return await self._playwright.call(tool, **arguments) + + async def _playwright_endpoint(self) -> str: + """The CDP address the browser issues Playwright MCP, at this driver's own loopback address, where its relay is + reached as any agent reaches it: the same one until the browser's data is reset, which ends it.""" + folder, _ = await self._found() + origin = f"http://127.0.0.1:{os.environ.get('MCP_PORT', '18765')}/svc/{folder.name}" + return (await self._call("cdp", {"user": PLAYWRIGHT_MCP_USER, "origin": origin}))["ws_url"] @display async def viewport(self) -> Display: @@ -383,5 +434,144 @@ def _steps(path: list) -> list[tuple[float, float]]: return steps +class _PlaywrightMcp: + """Playwright MCP, attached to the browser over the CDP address ``endpoint`` gives, and its sessions there, one per + caller of the driver's tools, so each agent keeps its own snapshots and tab. It starts on the first call, again once + it has exited, and again on a fresh address once the browser's data reset has ended the last; it ends with the + driver, so a driver started again finds none of its predecessor's.""" + + def __init__(self, endpoint: Callable[[], Awaitable[str]]) -> None: + self._endpoint = endpoint + self._lock = asyncio.Lock() + self._process: Optional[asyncio.subprocess.Process] = None + self._running_on: Optional[str] = None + self._url = "" + self._sessions: dict[Optional[str], _Session] = {} + + async def call(self, tool: str, /, **arguments: Any) -> CallToolResult: + """``tool`` called on Playwright MCP as it answers, or refused saying why it couldn't be.""" + try: + session = await self._session(ToolContext.current().caller.session) + return await session.call_tool(tool, arguments) + except DeviceUnavailable as e: + return _refused(e.message) + except Exception as e: + return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") + + async def _session(self, caller: Optional[str]) -> ClientSession: + async with self._lock: + url = await self._running() + held = self._sessions.get(caller) + if held is None or held.url != url or held.owner.done(): + if held is not None: + held.close.set() + held = self._sessions[caller] = await _Session.open(url) + return held.session + + async def _running(self) -> str: + """Playwright MCP's MCP URL, started on the browser's current address for it unless it runs there already.""" + endpoint = await self._endpoint() + if self._process is not None and self._process.returncode is None and endpoint == self._running_on: + return self._url + await self._stop() + port = _free_port() + PLAYWRIGHT_MCP_OUTPUT.mkdir(parents=True, exist_ok=True) + uploads = Path.home() / "Uploads" + uploads.mkdir(parents=True, exist_ok=True) + with (PLAYWRIGHT_MCP_OUTPUT / "playwright-mcp.log").open("ab") as log: + self._process = await asyncio.create_subprocess_exec( + *_command(endpoint, port), cwd=uploads, env=_environment(), stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.DEVNULL, stderr=log) + self._running_on, self._url = endpoint, f"http://127.0.0.1:{port}/mcp" + await _listening(port, self._process) + return self._url + + async def _stop(self) -> None: + for held in self._sessions.values(): + held.close.set() + self._sessions.clear() + if self._process is not None and self._process.returncode is None: + self._process.terminate() + with contextlib.suppress(asyncio.TimeoutError): + await asyncio.wait_for(self._process.wait(), 10) + if self._process.returncode is None: + self._process.kill() + await self._process.wait() + self._process = None + + +class _Session: + """One MCP session on Playwright MCP, held open by a task of its own until ``close`` is set, since mcp's client + contexts are anyio task groups, which only the task that entered one can leave.""" + + def __init__(self, url: str) -> None: + self.url = url + self.close = asyncio.Event() + self.session: Optional[ClientSession] = None + self.owner: Optional[asyncio.Task] = None + + @classmethod + async def open(cls, url: str) -> _Session: + held = cls(url) + ready: asyncio.Future = asyncio.get_running_loop().create_future() + held.owner = asyncio.create_task(held._own(ready)) + held.session = await ready + return held + + async def _own(self, ready: asyncio.Future) -> None: + try: + timeout = httpx.Timeout(PLAYWRIGHT_MCP_CALL_TIMEOUT_S, connect=PLAYWRIGHT_MCP_START_S) + async with httpx.AsyncClient(timeout=timeout, trust_env=False) as http, \ + streamable_http_client(self.url, http_client=http) as (read, write, _), \ + ClientSession(read, write) as session: + await session.initialize() + ready.set_result(session) + await self.close.wait() + except BaseException as e: + if not ready.done(): + ready.set_exception(e if isinstance(e, Exception) else RuntimeError(repr(e))) + elif not isinstance(e, asyncio.CancelledError): + logger.warning(f"a session on Playwright MCP ended: {type(e).__name__}: {e}") + if not isinstance(e, Exception): + raise + + +def _command(endpoint: str, port: int) -> list[str]: + """Playwright MCP on loopback ``port``, attached to the browser at ``endpoint``, never a browser of its own, with + its file access confined to its working folder, the session user's ~/Uploads; ``setpriv`` ends it with the driver.""" + return ["setpriv", "--pdeathsig", "TERM", "--", str(NODE), str(PLAYWRIGHT_MCP), "--cdp-endpoint", endpoint, + "--host", "127.0.0.1", "--port", str(port), "--caps", "vision", "--ignore-roots", + "--output-dir", str(PLAYWRIGHT_MCP_OUTPUT)] + + +def _environment() -> dict[str, str]: + return {name: os.environ[name] for name in KEPT_ENVIRONMENT if name in os.environ} + + +def _free_port() -> int: + with socket.socket() as probe: + probe.bind(("127.0.0.1", 0)) + return probe.getsockname()[1] + + +async def _listening(port: int, process: asyncio.subprocess.Process) -> None: + """Once Playwright MCP listens on ``port``; DeviceUnavailable when it exits first or takes too long.""" + deadline = asyncio.get_running_loop().time() + PLAYWRIGHT_MCP_START_S + while asyncio.get_running_loop().time() < deadline: + if process.returncode is not None: + raise DeviceUnavailable(f"Playwright MCP exited at start with {process.returncode}; its log is in " + f"{PLAYWRIGHT_MCP_OUTPUT}") + with contextlib.suppress(OSError): + _, writer = await asyncio.open_connection("127.0.0.1", port) + writer.close() + return + await asyncio.sleep(0.1) + raise DeviceUnavailable(f"Playwright MCP didn't listen within {PLAYWRIGHT_MCP_START_S:.0f} s") + + +def _refused(message: str) -> CallToolResult: + return CallToolResult(isError=True, content=[TextContent(type="text", text=message)]) + + if __name__ == "__main__": PageDriver().serve() diff --git a/src/agent_env/drivers/envs/page/env.toml b/src/agent_env/drivers/envs/page/env.toml index 6af38616c..9fc8c6514 100644 --- a/src/agent_env/drivers/envs/page/env.toml +++ b/src/agent_env/drivers/envs/page/env.toml @@ -4,4 +4,14 @@ port = 18765 [install.linux] python = "requirements.txt" +# Node and Playwright MCP 0.0.84, whose browser_* tools the driver serves beside its own: Node 24.22.2's Linux tarball +# (sha256 from nodejs.org's SHASUMS256.txt) and Playwright MCP's three npm packages (checked against npm's integrity). +downloads = [ + { url = "https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-x64.tar.gz", sha256 = "361718c444e49af2ca5a110c83ba6a8901143f23ab285ebbef0341f391f482a9", to = "node.tar.gz", arch = "amd64" }, + { url = "https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-arm64.tar.gz", sha256 = "95c264150aceb7f3437a29fd6984924f5f8d1511ea3cbd6c508b83f6c555dae2", to = "node.tar.gz", arch = "arm64" }, + { url = "https://registry.npmjs.org/@playwright/mcp/-/mcp-0.0.84.tgz", sha256 = "4e749bdc5a19138fed51cf0f51e43dca265c1bfc0fc388220fc70756aac60c26", to = "playwright-mcp.tgz" }, + { url = "https://registry.npmjs.org/playwright/-/playwright-1.65.0-alpha-2026-10-30.tgz", sha256 = "0d140b937c384637321d1c90b63794b1f1739b82966b66038077c1493ab9bc2f", to = "playwright.tgz" }, + { url = "https://registry.npmjs.org/playwright-core/-/playwright-core-1.65.0-alpha-2026-10-30.tgz", sha256 = "6f46fa60a6ee52221ebe612a03329f8b233d20158ad37f6b240dd087f4a0f4f9", to = "playwright-core.tgz" }, +] +run = [{ script = "install-playwright-mcp.sh" }] services = [{ name = "driver", command = "venv/bin/python -m agentenv_protocol.computer.serve driver.py:PageDriver" }] diff --git a/src/agent_env/drivers/envs/page/install-playwright-mcp.sh b/src/agent_env/drivers/envs/page/install-playwright-mcp.sh new file mode 100755 index 000000000..aba7eb83f --- /dev/null +++ b/src/agent_env/drivers/envs/page/install-playwright-mcp.sh @@ -0,0 +1,19 @@ +#!/bin/sh +# Node and Playwright MCP, from the tarballs the install downloaded and checked, with no npm install, since an install +# has no network: Node's binary and licence in node/, and Playwright MCP's three npm packages in node_modules/, each +# where npm would put it. +set -eu +work=$(mktemp -d) +tar -xzf node.tar.gz -C "$work" +mkdir -p node/bin +mv "$work"/node-*/bin/node node/bin/node +mv "$work"/node-*/LICENSE node/LICENSE +rm -rf "$work" node.tar.gz +unpack() { + mkdir -p "node_modules/$2" + tar -xzf "$1" -C "node_modules/$2" --strip-components=1 + rm -f "$1" +} +unpack playwright-mcp.tgz @playwright/mcp +unpack playwright.tgz playwright +unpack playwright-core.tgz playwright-core diff --git a/src/agent_env/drivers/envs/page/playwright-mcp-tools.json b/src/agent_env/drivers/envs/page/playwright-mcp-tools.json new file mode 100644 index 000000000..c5799bc70 --- /dev/null +++ b/src/agent_env/drivers/envs/page/playwright-mcp-tools.json @@ -0,0 +1,1131 @@ +{ + "version": "0.0.84", + "caps": [ + "vision" + ], + "tools": [ + { + "name": "browser_close", + "description": "Close the page", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": {}, + "additionalProperties": false + }, + "annotations": { + "title": "Close browser", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_resize", + "description": "Resize the browser window", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "width": { + "type": "number", + "description": "Width of the browser window" + }, + "height": { + "type": "number", + "description": "Height of the browser window" + } + }, + "required": [ + "width", + "height" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Resize browser window", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_console_messages", + "description": "Returns all console messages", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "level": { + "default": "info", + "description": "Level of the console messages to return. Each level includes the messages of more severe levels. Defaults to \"info\".", + "type": "string", + "enum": [ + "error", + "warning", + "info", + "debug" + ] + }, + "all": { + "description": "Return all console messages since the beginning of the session, not just since the last navigation. Defaults to false.", + "type": "boolean" + }, + "filename": { + "description": "File name to save the console messages to. Relative file names are resolved against the workspace root. If not provided, messages are returned as text.", + "type": "string" + } + }, + "required": [ + "level" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Get console messages", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_handle_dialog", + "description": "Handle a dialog", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "accept": { + "type": "boolean", + "description": "Whether to accept the dialog." + }, + "promptText": { + "description": "The text of the prompt in case of a prompt dialog.", + "type": "string" + } + }, + "required": [ + "accept" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Handle a dialog", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_emulate_media", + "description": "Emulate CSS media features for the page, for example switch between the light and dark color scheme. Omitted parameters are left unchanged; null clears an override.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "colorScheme": { + "description": "Emulates the prefers-color-scheme media feature", + "anyOf": [ + { + "type": "string", + "enum": [ + "light", + "dark" + ] + }, + { + "type": "null" + } + ] + }, + "reducedMotion": { + "description": "Emulates the prefers-reduced-motion media feature", + "anyOf": [ + { + "type": "string", + "enum": [ + "reduce", + "no-preference" + ] + }, + { + "type": "null" + } + ] + }, + "forcedColors": { + "description": "Emulates the forced-colors media feature", + "anyOf": [ + { + "type": "string", + "enum": [ + "active", + "none" + ] + }, + { + "type": "null" + } + ] + }, + "contrast": { + "description": "Emulates the prefers-contrast media feature", + "anyOf": [ + { + "type": "string", + "enum": [ + "more", + "no-preference" + ] + }, + { + "type": "null" + } + ] + }, + "media": { + "description": "Changes the CSS media type of the page", + "anyOf": [ + { + "type": "string", + "enum": [ + "screen", + "print" + ] + }, + { + "type": "null" + } + ] + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Emulate media features", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_evaluate", + "description": "Evaluate JavaScript expression on page or element", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "description": "Exact target element reference from the page snapshot, or a unique element selector", + "type": "string" + }, + "function": { + "type": "string", + "description": "() => { /* code */ } or (element) => { /* code */ } when element is provided" + }, + "filename": { + "description": "File name to save the result to. Relative file names are resolved against the workspace root. If not provided, result is returned as text.", + "type": "string" + } + }, + "required": [ + "function" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Evaluate JavaScript", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_file_upload", + "description": "Upload one or multiple files", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "paths": { + "description": "The absolute paths to the files to upload. Can be single file or multiple files. If omitted, file chooser is cancelled.", + "type": "array", + "items": { + "type": "string" + } + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Upload files", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_drop", + "description": "Drop files or MIME-typed data onto an element, as if dragged from outside the page. At least one of \"paths\" or \"data\" must be provided.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "paths": { + "description": "Absolute paths to files to drop onto the element.", + "type": "array", + "items": { + "type": "string" + } + }, + "data": { + "description": "Data to drop, as a map of MIME type to string value (e.g. {\"text/plain\": \"hello\", \"text/uri-list\": \"https://example.com\"}).", + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": { + "type": "string" + } + } + }, + "required": [ + "target" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Drop files or data onto an element", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_find", + "description": "Search the accessibility snapshot of the current page for text or a regular expression. Returns matching snapshot nodes with a few lines of surrounding context (like search snippets), each shown under its path from the root of the tree, which is cheaper than capturing the whole snapshot when you only need to locate an element and its ref.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "text": { + "description": "Plain text to search for in the page snapshot (case-insensitive substring match). Provide either text or regex, not both.", + "type": "string" + }, + "regex": { + "description": "Regular expression to search for in the page snapshot. Matching is case-sensitive by default; wrap the pattern in slashes to add flags, e.g. \"/error/i\" for case-insensitive. Provide either text or regex, not both.", + "type": "string" + }, + "maxResults": { + "description": "Maximum number of matching items to return. Defaults to returning all matches.", + "type": "number" + }, + "filename": { + "description": "Save results to a file instead of returning them in the response. Relative file names are resolved against the workspace root.", + "type": "string" + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Find in page snapshot", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_fill_form", + "description": "Fill multiple form fields", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "name": { + "type": "string", + "description": "Human-readable field name" + }, + "type": { + "type": "string", + "enum": [ + "textbox", + "checkbox", + "radio", + "combobox", + "slider" + ], + "description": "Type of the field" + }, + "value": { + "type": "string", + "description": "Value to fill in the field. If the field is a checkbox, the value should be `true` or `false`. If the field is a combobox, the value should be the text of the option." + } + }, + "required": [ + "target", + "name", + "type", + "value" + ], + "additionalProperties": false + }, + "description": "Fields to fill in" + } + }, + "required": [ + "fields" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Fill form", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_press_key", + "description": "Press a key on the keyboard", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "key": { + "type": "string", + "description": "Name of the key to press or a character to generate, such as `ArrowLeft` or `a`" + } + }, + "required": [ + "key" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Press a key", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_type", + "description": "Type text into editable element", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "text": { + "type": "string", + "description": "Text to type into the element" + }, + "submit": { + "description": "Whether to submit entered text (press Enter after)", + "type": "boolean" + }, + "slowly": { + "description": "Whether to type one character at a time. Useful for triggering key handlers in the page. By default entire text is filled in at once.", + "type": "boolean" + } + }, + "required": [ + "target", + "text" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Type text", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_move_xy", + "description": "Move mouse to a given position", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "x": { + "type": "number", + "description": "X coordinate" + }, + "y": { + "type": "number", + "description": "Y coordinate" + } + }, + "required": [ + "x", + "y" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Move mouse", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_click_xy", + "description": "Click mouse button at a given position", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "x": { + "type": "number", + "description": "X coordinate" + }, + "y": { + "type": "number", + "description": "Y coordinate" + }, + "button": { + "description": "Button to click, defaults to left", + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + }, + "clickCount": { + "description": "Number of clicks, defaults to 1", + "type": "number" + }, + "delay": { + "description": "Time to wait between mouse down and mouse up in milliseconds, defaults to 0", + "type": "number" + } + }, + "required": [ + "x", + "y" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Click", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_drag_xy", + "description": "Drag left mouse button to a given position", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "startX": { + "type": "number", + "description": "Start X coordinate" + }, + "startY": { + "type": "number", + "description": "Start Y coordinate" + }, + "endX": { + "type": "number", + "description": "End X coordinate" + }, + "endY": { + "type": "number", + "description": "End Y coordinate" + } + }, + "required": [ + "startX", + "startY", + "endX", + "endY" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Drag mouse", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_down", + "description": "Press mouse down", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "button": { + "description": "Button to press, defaults to left", + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Press mouse down", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_up", + "description": "Press mouse up", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "button": { + "description": "Button to press, defaults to left", + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Press mouse up", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_wheel", + "description": "Scroll mouse wheel", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "deltaX": { + "default": 0, + "description": "X delta", + "type": "number" + }, + "deltaY": { + "default": 0, + "description": "Y delta", + "type": "number" + } + }, + "required": [ + "deltaX", + "deltaY" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Scroll mouse wheel", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_navigate", + "description": "Navigate to a URL", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "The URL to navigate to" + } + }, + "required": [ + "url" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Navigate to a URL", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_navigate_back", + "description": "Go back to the previous page in the history", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": {}, + "additionalProperties": false + }, + "annotations": { + "title": "Go back", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_network_requests", + "description": "Returns a numbered list of network requests since loading the page. Use browser_network_request with the number to get full details.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "static": { + "default": false, + "description": "Whether to include successful static resources like images, fonts, scripts, etc. Defaults to false.", + "type": "boolean" + }, + "filter": { + "description": "Only return requests whose URL matches this regexp (e.g. \"/api/.*user\").", + "type": "string" + }, + "all": { + "description": "Return all network requests since the beginning of the session, not just since the last navigation. Defaults to false.", + "type": "boolean" + }, + "filename": { + "description": "File name to save the network requests to. Relative file names are resolved against the workspace root. If not provided, requests are returned as text.", + "type": "string" + } + }, + "required": [ + "static" + ], + "additionalProperties": false + }, + "annotations": { + "title": "List network requests", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_network_request", + "description": "Returns full details (headers and body) of a single network request, or a single part if `part` is set. Use the number from browser_network_requests.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "index": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991, + "description": "1-based index of the request, as printed by browser_network_requests." + }, + "part": { + "description": "Return only this part of the request. Omit to return full details.", + "type": "string", + "enum": [ + "request-headers", + "request-body", + "response-headers", + "response-body" + ] + }, + "filename": { + "description": "File name to save the result to. Relative file names are resolved against the workspace root. If not provided, output is returned as text.", + "type": "string" + } + }, + "required": [ + "index" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Show network request details", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_run_code_unsafe", + "description": "Run a Playwright code snippet. Unsafe: executes arbitrary JavaScript in the Playwright server process and is RCE-equivalent.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "code": { + "description": "A JavaScript function containing Playwright code to execute. It will be invoked with a single argument, page, which you can use for any page interaction. For example: `async (page) => { await page.getByRole('button', { name: 'Submit' }).click(); return await page.title(); }`", + "type": "string" + }, + "filename": { + "description": "Load code from the specified file. Relative file names are resolved against the workspace root. If both code and filename are provided, code will be ignored.", + "type": "string" + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Run Playwright code (unsafe)", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_take_screenshot", + "description": "Take a screenshot of the current page. You can't perform actions based on the screenshot, use browser_snapshot for actions.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "description": "Exact target element reference from the page snapshot, or a unique element selector", + "type": "string" + }, + "type": { + "description": "Image format for the screenshot. If unset, inferred from the filename extension, otherwise png.", + "type": "string", + "enum": [ + "png", + "jpeg", + "webp" + ] + }, + "filename": { + "description": "File name to save the screenshot to. Relative file names are resolved against the workspace root. If not specified, the screenshot is saved into the output directory as `page-{timestamp}.{png|jpeg|webp}`.", + "type": "string" + }, + "fullPage": { + "description": "When true, takes a screenshot of the full scrollable page, instead of the currently visible viewport. Cannot be used with element screenshots.", + "type": "boolean" + }, + "scale": { + "default": "css", + "description": "Image resolution scale. \"css\" produces a screenshot sized in CSS pixels (smaller, consistent across devices). \"device\" produces a high-resolution screenshot using device pixels (larger, accounts for the device pixel ratio). Default is css.", + "type": "string", + "enum": [ + "css", + "device" + ] + } + }, + "required": [ + "scale" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Take a screenshot", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_snapshot", + "description": "Capture accessibility snapshot of the current page, this is better than screenshot", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "target": { + "description": "Exact target element reference from the page snapshot, or a unique element selector", + "type": "string" + }, + "filename": { + "description": "Save snapshot to a file instead of returning it in the response. Relative file names are resolved against the workspace root.", + "type": "string" + }, + "depth": { + "description": "Limit the depth of the snapshot tree", + "type": "number" + }, + "boxes": { + "description": "Include each element's bounding box as [box=x,y,width,height] in the snapshot. Coordinates are viewport-relative, in CSS pixels (Element.getBoundingClientRect)", + "type": "boolean" + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Page snapshot", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_click", + "description": "Perform click on a web page", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "doubleClick": { + "description": "Whether to perform a double click instead of a single click", + "type": "boolean" + }, + "button": { + "description": "Button to click, defaults to left", + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + }, + "modifiers": { + "description": "Modifier keys to press", + "type": "array", + "items": { + "type": "string", + "enum": [ + "Alt", + "Control", + "ControlOrMeta", + "Meta", + "Shift" + ] + } + } + }, + "required": [ + "target" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Click", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_drag", + "description": "Perform drag and drop between two elements", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "startElement": { + "description": "Human-readable source element description used to obtain the permission to interact with the element", + "type": "string" + }, + "startTarget": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "endElement": { + "description": "Human-readable target element description used to obtain the permission to interact with the element", + "type": "string" + }, + "endTarget": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + } + }, + "required": [ + "startTarget", + "endTarget" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Drag mouse", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_hover", + "description": "Hover over element on page", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + } + }, + "required": [ + "target" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Hover mouse", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_select_option", + "description": "Select an option in a dropdown", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "values": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Array of values to select in the dropdown. This can be a single value or multiple values." + } + }, + "required": [ + "target", + "values" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Select option", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_tabs", + "description": "List, create, close, or select a browser tab.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": [ + "list", + "new", + "close", + "select" + ], + "description": "Operation to perform" + }, + "index": { + "description": "Tab index, used for close/select. If omitted for close, current tab is closed.", + "type": "number" + }, + "url": { + "description": "URL to navigate to in the new tab, used for new.", + "type": "string" + }, + "isolatedContext": { + "description": "Name of an isolated browser context to create the new tab in, used for new. Tabs in the same isolated context share cookies and storage, tabs in different contexts are fully isolated. If omitted or \"default\", the tab is created in the default context.", + "type": "string" + } + }, + "required": [ + "action" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Manage tabs", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_wait_for", + "description": "Wait for text to appear or disappear or a specified time to pass. When both text and textGone are provided, waits for the first one to happen", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "time": { + "description": "The time to wait in seconds, at most 30. When combined with text or textGone, serves as a timeout for them instead of the default action timeout", + "type": "number" + }, + "text": { + "description": "The text to wait for", + "type": "string" + }, + "textGone": { + "description": "The text to wait for to disappear", + "type": "string" + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Wait for", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + } + ] +} diff --git a/tst/integration/computer/browser_computer_test.py b/tst/integration/computer/browser_computer_test.py index 5e3656cca..14154b31b 100644 --- a/tst/integration/computer/browser_computer_test.py +++ b/tst/integration/computer/browser_computer_test.py @@ -1,7 +1,9 @@ """The browser as a computer: agentenv's chromium deployed alone, on the ubuntu-24.04-browser base with no desktop and the page driver, whose screen is chromium's active tab, headful on the X server only the browser uses. Gate 3 makes this machine's arch ready with it, its input checks and page tools verified on the HTML desk; deployed through the computer provider like any computer, -its computer_* tools see and drive the page, and its page tools navigate, move in history, answer a dialog, use tabs +its computer_* tools see and drive the page, Playwright MCP's browser_* tools beside them type into it and answer its +dialog, the two that run code hidden until tool access names one, as the installed Playwright MCP lists them, and +back after Playwright MCP or the driver is killed, and its page tools navigate, move in history, answer a dialog, use tabs and upload; the browser's X server runs but no desktop, and nothing listens but the driver and the site proxy; a killed app server or driver is started again by supervisord and the computer is ready again; chromium killed fails it for good, computer-ready/v1 saying why and the tools answering with it, and so does its X server killed, which takes @@ -59,8 +61,10 @@ from agentenv_protocol.client import find_extension_method, invoke_extension from agentenv_protocol.computer import COMPUTER_PAGE, COMPUTER_READY from agentenv_protocol.computer.conformance.client import image +from agentenv_protocol.tool_access import ENABLE, ENABLE_TOOL_EXTENSION_URI from agentenv_protocol.website_browser import BROWSER +import agent_env from agent_env.a2a_agent import conversation_store from agent_env.artifact import EnvironmentUniverseArtifact from agent_env.artifact.artifacts.environment import EnvironmentArtifact @@ -161,6 +165,16 @@ def log_message(self, *args): time.sleep(float(sys.argv[1])) """ SERVE_START_S = 3 +#: Playwright MCP's own tools/list in the computer, from the node and cli.js it's given, as JSON. +LISTED_TOOLS = """import json, subprocess, sys +messages = [{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2025-06-18", + "capabilities": {}, "clientInfo": {"name": "test", "version": "0"}}}, + {"jsonrpc": "2.0", "method": "notifications/initialized"}, {"jsonrpc": "2.0", "id": 2, "method": "tools/list"}] +out = subprocess.run(sys.argv[1:] + ["--cdp-endpoint", "http://127.0.0.1:9/", "--caps", "vision"], capture_output=True, + input="\\n".join(map(json.dumps, messages)) + "\\n", text=True, timeout=60).stdout +print(json.dumps(next(r["result"]["tools"] for r in map(json.loads, out.splitlines()) if r.get("id") == 2))) +""" +PLAYWRIGHT_MCP_TOOLS = Path(agent_env.__file__).parent / "drivers" / "envs" / "page" / "playwright-mcp-tools.json" #: A page that needs no server, so killing the driver, whose child a served site is, leaves it to open. DATA_PAGE = "data:text/html,Home

Home

" #: A page a task's tab steps open, bring to the front and close by its URL. @@ -882,6 +896,89 @@ async def test_playwright_and_puppeteer_drive_it_through_its_cdp_address_which_a assert fresh != ada and not report.failed +@pytest.mark.asyncio +async def test_playwright_mcps_tools_drive_its_page_hide_the_code_tools_and_come_back_after_a_kill(browser_computer): + with namespace_routing(): + deployed = await Env.get(browser_computer.id).deploy( + env_params={"app_settings": {"chromium": {"start_urls": [PAGE_URL]}}}) + context = TaskStepContext(deployed_envs=[deployed]) + serving = None + try: + viewport = (await _screenshot(deployed)).size + listed = await _tool_names(deployed) + typed = await _mcp(deployed, _type_hello) + state = await _state(deployed, lambda s: s["v"] == "hello") + after_typing = (await _screenshot(deployed)).size + serving = asyncio.ensure_future(inside.run(deployed, ["python3", "-c", SERVE_SITE, "900"], timeout_s=960)) + await asyncio.sleep(SERVE_START_S) + confirmed = await _mcp(deployed, _confirm_ask) + await deployed.invoke(ENABLE_TOOL_EXTENSION_URI, ENABLE, {"role": "default", "tools": ["browser_evaluate"]}) + shown, evaluated = await _tool_names(deployed), await _mcp(deployed, _evaluate_title) + running = (await inside.run(deployed, ["sh", "-c", "ps -eo args | grep -m1 '[@]playwright/mcp/cli.js'"], + timeout_s=30))["stdout"].split() + installed = json.loads((await inside.run(deployed, ["python3", "-c", LISTED_TOOLS, *running[:2]], + timeout_s=120))["stdout"]) + await _kill(deployed, "[@]playwright/mcp/cli.js") + after_its_kill = await _mcp(deployed, _snapshot) + await _kill(deployed, "[c]omputer.serve driver.py:PageDriver") + await _ready(deployed) + after_driver_kill = await _mcp(deployed, _snapshot) + count = (await inside.run(deployed, ["sh", "-c", "pgrep -fc '[@]playwright/mcp/cli.js'"], + timeout_s=30))["stdout"].strip() + record = await _record(deployed) + finally: + if serving is not None: + serving.cancel() + report = await teardown_run(context) + + playwright = {t["name"] for t in json.loads(PLAYWRIGHT_MCP_TOOLS.read_text())["tools"]} + assert playwright - {"browser_evaluate", "browser_run_code_unsafe"} <= set(listed), listed + assert not {"browser_evaluate", "browser_run_code_unsafe"} & set(listed) and "computer_screenshot" in listed + assert "hello" in typed and state["v"] == "hello" and after_typing == viewport + assert "confirmed true" in confirmed, confirmed + assert "browser_evaluate" in shown and "browser_run_code_unsafe" not in shown and "confirmed true" in evaluated + assert installed == json.loads(PLAYWRIGHT_MCP_TOOLS.read_text())["tools"] + assert "Page URL" in after_its_kill and "Page URL" in after_driver_kill and count == "1" + assert "playwright-mcp" in {connection["user"] for connection in record["cdp_connections"]} + assert not report.failed + + +async def _mcp(deployed, work): + """What ``work`` makes of one MCP session on the computer, as a role-less agent opens one.""" + async with streamablehttp_client(deployed.mcp_url) as (read, write, _): + async with ClientSession(read, write) as session: + await session.initialize() + return await work(session) + + +async def _called(session: ClientSession, tool: str, arguments: dict) -> str: + result = await session.call_tool(tool, arguments) + text = "\n".join(part.text for part in result.content if part.type == "text") + assert not result.isError, f"{tool}: {text}" + return text + + +async def _snapshot(session: ClientSession) -> str: + return await _called(session, "browser_snapshot", {}) + + +async def _type_hello(session: ClientSession) -> str: + ref = re.search(r"textbox \[ref=(\w+)\]", await _snapshot(session)).group(1) + return await _called(session, "browser_type", {"target": ref, "text": "hello"}) + + +async def _confirm_ask(session: ClientSession) -> str: + await _called(session, "browser_navigate", {"url": SITE}) + ref = re.search(r'button "ask" \[ref=(\w+)\]', await _snapshot(session)).group(1) + await session.call_tool("browser_click", {"target": ref}) + await _called(session, "browser_handle_dialog", {"accept": True}) + return await _snapshot(session) + + +async def _evaluate_title(session: ClientSession) -> str: + return await _called(session, "browser_evaluate", {"function": "() => document.title"}) + + @pytest.mark.asyncio async def test_behind_a_front_that_ends_tls_an_address_under_the_origin_named_is_https_and_wss_and_clients_connect( browser_computer, cdp_clients): diff --git a/tst/unit/drivers/page_driver_test.py b/tst/unit/drivers/page_driver_test.py index f6ee8f1e4..13a97d4c8 100644 --- a/tst/unit/drivers/page_driver_test.py +++ b/tst/unit/drivers/page_driver_test.py @@ -1,31 +1,55 @@ """The page driver against a fake browser server on the app's socket: it passes the definition checks with a page screen, turns tool calls into the CDP input they mean, shows a tab that paints nothing as white, and once the browser -has ended on its own, its ready check fails for good and its calls say why.""" +has ended on its own, its ready check fails for good and its calls say why. Beside its own tools it serves Playwright +MCP's from their manifest, the two that run the agent's code hidden, on a Playwright MCP it runs on the CDP address the +browser issues it under the driver's loopback origin: here a stand-in, which starts on the first call, keeps a session +per caller, starts again on a new address or once killed, and never gets the driver's environment.""" from __future__ import annotations +import asyncio import base64 import importlib.util import json +import os +import signal +import sys from pathlib import Path import httpx import pytest +from pytest_socket import enable_socket import agent_env from agentenv_protocol.computer import (DeviceUnavailable, FileChooser, InvalidArgument, PageDialog, PageState, Platform, Tab) from agentenv_protocol.computer import driver as computer_driver from agentenv_protocol.computer.app_runtime import LAUNCHES +from agentenv_protocol.tool_context import ToolContext, bound from agentenv_protocol.website_browser import BROWSER DRIVER = Path(agent_env.__file__).parent / "drivers" / "envs" / "page" / "driver.py" NO_DISPLAY = Platform(os="linux", distribution="ubuntu", version="24.04", arch="amd64", display="none") PAGE_OPS = ("page/viewport", "page/capture", "page/input", "page/state", "navigate", "history", "tabs/list", - "tabs/open", "tabs/activate", "tabs/close", "dialog/answer", "files/choose") + "tabs/open", "tabs/activate", "tabs/close", "dialog/answer", "files/choose", "cdp") GETS = ("page/viewport", "page/capture", "page/state", "tabs/list") CARD = {"name": "chromium", "capabilities": {"extensions": [{"uri": BROWSER, "params": {"methods": { op: {"method": "GET" if op in GETS else "POST", "endpoint": f"/agentenv/ext/browser/{op}"} for op in PAGE_OPS}}}]}} +#: A stand-in for Playwright MCP on the port and CDP address it's started with, whose one tool says what it was started +#: with and which session called it; started on an address ending /exits, it exits at once. +FAKE_PLAYWRIGHT_MCP = """import json, os, sys +from mcp.server.fastmcp import Context, FastMCP +args = sys.argv[1:] +endpoint, port = args[args.index("--cdp-endpoint") + 1], int(args[args.index("--port") + 1]) +if endpoint.endswith("/exits"): + sys.exit(3) +app = FastMCP("fake", host="127.0.0.1", port=port) +@app.tool() +def browser_snapshot(ctx: Context) -> str: + return json.dumps({"pid": os.getpid(), "endpoint": endpoint, "session": id(ctx.session), "cwd": os.getcwd(), + "env": sorted(os.environ)}) +app.run("streamable-http") +""" #: A 2×1 PNG's first 24 bytes: its signature, then its IHDR's length, type, width and height. PNG = b"\x89PNG\r\n\x1a\n" + b"\x00\x00\x00\rIHDR" + (2).to_bytes(4, "big") + (1).to_bytes(4, "big") @@ -220,3 +244,81 @@ async def test_a_call_the_browser_refuses_as_given_wrong_is_an_invalid_argument_ with pytest.raises(InvalidArgument, match="there's no file cv.pdf in ~/Uploads"): await page.choose_files(["cv.pdf"]) + +def test_it_serves_playwright_mcps_tools_from_their_manifest_and_hides_the_two_that_run_the_agents_code(page): + manifest = json.loads((DRIVER.parent / "playwright-mcp-tools.json").read_text()) + forwarded = page.forwarded_tools() + + assert manifest["version"] == "0.0.84" and manifest["caps"] == ["vision"] + assert [f.tool.name for f in forwarded] == [t["name"] for t in manifest["tools"]] + assert [f.tool.inputSchema for f in forwarded] == [t["inputSchema"] for t in manifest["tools"]] + assert {f.tool.name for f in forwarded if f.hidden_by_default} == {"browser_evaluate", "browser_run_code_unsafe"} + + +@pytest.mark.asyncio +async def test_it_asks_the_browser_for_playwright_mcps_address_under_its_own_loopback_origin(page, monkeypatch): + monkeypatch.setenv("MCP_PORT", "18765") + page.served["cdp"] = {"http_url": "http://127.0.0.1:18765/svc/chromium/cdp/t/", + "ws_url": "ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser"} + + endpoint = await page._playwright_endpoint() + + assert endpoint == "ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser" + assert page.sent[-1] == ("cdp", {"user": "playwright-mcp", "origin": f"http://127.0.0.1:18765/svc/{page._browser[0].name}"}) + + +def test_playwright_mcp_attaches_to_the_browser_never_launches_one_and_ends_with_the_driver(): + command = _load()._command("ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser", 40123) + + assert command[:4] == ["setpriv", "--pdeathsig", "TERM", "--"] + assert command[command.index("--cdp-endpoint") + 1] == "ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser" + assert {"--caps", "vision", "--ignore-roots", "127.0.0.1", "40123"} <= set(command) + assert not {"--isolated", "--headless", "--browser", "--allow-unrestricted-file-access"} & set(command) + + +@pytest.mark.asyncio +async def test_playwright_mcp_starts_on_the_first_call_keeps_a_session_per_caller_and_starts_again_when_it_must( + monkeypatch, tmp_path): + enable_socket() + module = _load() + fake = tmp_path / "fake_playwright_mcp.py" + fake.write_text(FAKE_PLAYWRIGHT_MCP) + monkeypatch.setattr(module, "_command", lambda endpoint, port: [ + sys.executable, str(fake), "--cdp-endpoint", endpoint, "--port", str(port)]) + monkeypatch.setattr(module, "PLAYWRIGHT_MCP_OUTPUT", tmp_path / "out") + monkeypatch.setenv("HOME", str(tmp_path / "home")) + monkeypatch.setenv("AGENTENV_CONTROL_TOKEN", "secret") + endpoints = ["ws://127.0.0.1:18765/svc/chromium/cdp/one/devtools/browser"] + + async def endpoint() -> str: + return endpoints[-1] + + playwright = module._PlaywrightMcp(endpoint) + + async def called(caller: str): + with bound(ToolContext.as_caller(session=caller)): + return await playwright.call("browser_snapshot") + + async def snapshot(caller: str) -> dict: + result = await called(caller) + assert not result.isError, result.content + return json.loads(result.content[0].text) + + try: + first, again, other = await snapshot("ada"), await snapshot("ada"), await snapshot("bob") + endpoints.append("ws://127.0.0.1:18765/svc/chromium/cdp/two/devtools/browser") + moved = await snapshot("ada") + os.kill(moved["pid"], signal.SIGKILL) + await asyncio.sleep(1) + revived = await snapshot("ada") + endpoints.append("ws://127.0.0.1:18765/svc/chromium/cdp/exits") + refused = await called("ada") + finally: + await playwright._stop() + + assert first["pid"] == again["pid"] == other["pid"] and first["session"] == again["session"] != other["session"] + assert first["endpoint"].endswith("/cdp/one/devtools/browser") and first["cwd"] == str(tmp_path / "home" / "Uploads") + assert "AGENTENV_CONTROL_TOKEN" not in first["env"] and "HOME" in first["env"] + assert moved["pid"] != first["pid"] and moved["endpoint"].endswith("/cdp/two/devtools/browser") + assert revived["pid"] != moved["pid"] + assert refused.isError and "exited at start with 3" in refused.content[0].text