Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion packages/agentenv-protocol/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1000,7 +1000,35 @@ are the profile's template. The base class adds:
and, completed, its size and sha256; and `dialogs`, each with its type, message, default text, answer
and who gave it.
`hidden_page()` gives agentenv a page of its own for reading state, in no window and never in the record.
- **`browser/v1`** on its card: `chromium` (its `version`), `clock: "real"` and, alone, `window`; and
- **Its log**, `self.log`, a `BrowserLog` (`agentenv_protocol.website_browser.log`): one entry for each action agents take on it
and an event each time a tab opens, navigates or closes. Its server writes it, the one writer, to
`/var/log/agentenv/browser-<name>.jsonl` (its own `.agentenv/` folder when that can't be written), one JSON object a
line with its `seq` and `at` and one of `action`, `tab`, `outcome` or `reset`. A line's number is taken when what it
records begins and the line written when it ends, so `log {after}` returns lines in order up to the last whose every
earlier line is written (`last`), and `last` never moves back. An action still going after 60 s is written by the
next read as `unfinished`, so it holds back nothing after it, and once it ends an `outcome` line names it (`entry`)
with what it ended with. A relayed connection's commands reach the browser in the order they came. The file lasts the
computer's boot, a restarted server carrying on the sequence and the
browser's own reset (`data/reset`) being a line. A computer's `reset_env` boots it afresh, with a new log, so a task
that needs the one before takes it first, with `snapshot_env`.
- **An action** is a CDP command through the relay (`via: "cdp"`, `who` the connection's user, its `connection`
number, `method`, and the tab its session is on), an op a driver makes for an agent's MCP tool call (`via: "mcp"`,
`who` the caller's role, `tool`, `call`, `op`), or an acting op agentenv makes itself (`via: "agentenv"`). Each
carries `tab`, `url`, `ok` and, failed, `error`, and is marked `acts` or not by its method or op
(`log.acts`, `ACTING_OPS`): input, navigation, tab commands, `Runtime.evaluate` and `callFunctionOn`, DOM
writes, faked responses, and viewport and cookie changes act; the rest read. An acting entry keeps its `params`
(and a tool call's `args`) with what could carry a secret taken out: text typed while a password field has focus,
as its length; cookie and header values; the values of URL query keys that name a credential; a request's body;
and each string past 2 KB cut, with its length and sha256. A reading entry keeps none.
- **A tab event** (`opened`, `navigated`, `closed`) says what caused it: `by: "command"` with the `entry`, `via` and
`who` of the command that asked for it, matched exactly by the target a new tab's command answers with and the
loader a navigation's does; `by: "agentenv"`; `exit` or `stop`; or `by: "page"`, with the `after` of the latest
acting entry on its tab (its opener's, for a popup), for a click that follows a link, a redirect or a page script.
- A driver passes an agent's tool call to the browser in the `AgentEnv-Tool-Call` header (`tool_call_header`), with
the caller's `AgentEnv-Role`; the page driver sends it on every call it makes for a tool. `data/get` writes the log
beside `browser-state.json` as `browser-log.jsonl`, which `data/add` passes over.
- **`browser/v1`** on its card: `chromium` (its `version`), `clock: "real"` and, alone, `window`; `log {after}`
(GET), framework-only, its log's lines after `after`; and
`record` (GET), framework-only, which returns that record with `launches`, the app server's log of each
launch (with its cause: `boot`, `server_restart`, `launch`, `data/add` or `data/reset`), each stop it made
and each exit it didn't, with its exit `code` or the `signal` that ended it, since the computer booted.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,10 @@ async def _computer_app_launch(self) -> dict:
async def _computer_app_ready(self) -> dict:
return (await self._computer_app_runtime().readiness()).model_dump()

def served_asgi(self, asgi: Any) -> Any:
"""The application its server serves, ``asgi`` as built from its card: a subclass may wrap it."""
return asgi

def _computer_app_runtime(self) -> AppRuntime:
if self._computer_app is None:
raise TypeError(f"{type(self).__qualname__}.__init__ must call super().__init__()")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,13 @@ async def lifespan(scope):
started.cancel()

asgi.router.lifespan_context = lifespan
served = app.served_asgi(asgi)
if LOOPBACK:
listening, token = bind(Path.cwd())
server = uvicorn.Server(uvicorn.Config(guarded(asgi, token), log_level="warning"))
server = uvicorn.Server(uvicorn.Config(guarded(served, token), log_level="warning"))
server.run(sockets=[listening])
else:
server = uvicorn.Server(uvicorn.Config(asgi, uds=str(SOCKET), log_level="warning"))
server = uvicorn.Server(uvicorn.Config(served, uds=str(SOCKET), log_level="warning"))
SOCKET.unlink(missing_ok=True)
server.run()
if failed:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ class ToolContext:
tool: str = ""
arguments: Mapping[str, Any] = dataclasses.field(default_factory=lambda: types.MappingProxyType({}), hash=False)
call_id: str = ""
#: Whether the call is an MCP tool call, rather than a REST or extension call.
mcp: bool = False

@property
def role(self) -> Optional[str]:
Expand Down Expand Up @@ -189,7 +191,7 @@ def context_from_mcp(context: Any, tool: str, arguments: Mapping[str, Any]) -> T
request = getattr(request_context, "request", None)
headers = getattr(request, "headers", None) if request is not None else None
return ToolContext(caller=caller_from_request(extra, headers), tool=tool, arguments=_frozen(arguments),
call_id=uuid.uuid4().hex)
call_id=uuid.uuid4().hex, mcp=True)


def _meta_entries(meta: Any) -> Optional[Mapping[str, Any]]:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,11 @@
from starlette.requests import Request
from starlette.websockets import WebSocket, WebSocketDisconnect

from .log import ERROR_CHARS, KEY_FIELDS, TYPING_METHODS, BrowserLog, acts, redact_text, redacted_params

if TYPE_CHECKING:
from .website_browser import DevToolsPipe
from .control import BrowserControl
from .devtools import DevToolsPipe

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -55,6 +58,10 @@
GUARDED_KEPT = 100
#: What the record says the guard did, by its action.
GUARDED = {"refuse": "refused", "send": "changed", "answer": "answered", "close": "closed"}
#: The commands that navigate a tab without naming the loader they navigate with, whose next navigation is theirs.
NAVIGATES_NEXT = frozenset({"Page.reload", "Page.navigateToHistoryEntry"})
#: What a command's entry says when its connection ended before the browser answered it.
ENDED_UNANSWERED = "the connection ended before the browser answered"


@dataclass
Expand Down Expand Up @@ -192,9 +199,14 @@ class _Connection:
"""One client's WebSocket, its sessions over the pipe (its browser-level one and those it attached), its unanswered
commands, what waits to be sent to it, and its entry in the record."""

def __init__(self, websocket: WebSocket, user: str, session: str, record: dict) -> None:
self.websocket, self.user, self.session, self.record = websocket, user, session, record
def __init__(self, websocket: WebSocket, user: str, session: str, record: dict, number: int) -> None:
self.websocket, self.user, self.session, self.record, self.number = websocket, user, session, record, number
self.sessions = {session}
#: The tab each session it attached is on.
self.targets: dict[str, str] = {}
#: Held by each of its commands from when it arrives until it's sent, so they're sent in the order they came,
#: whatever one waits on before it's sent.
self.order = asyncio.Lock()
self.commands: set[asyncio.Future] = set()
self.outbox: asyncio.Queue[str | tuple[int, str]] = asyncio.Queue()
self.waiting_bytes = 0
Expand All @@ -205,10 +217,13 @@ class Relay:
"""Each WebSocket connection given its own browser-level session over ``devtools``, its commands sent on it past the
guard and the answers and events of its sessions sent back, so several clients share one browser; and a record of
who connected when, and what the guard did to their commands. ``dialogs_answered`` says agentenv answers the
browser's dialogs itself."""
browser's dialogs itself. With a ``log``, each command is an entry in it, on the tab its session is on, and
``control``, agentenv's own connection, tells its tab's URL, whether a password field has focus there, and what
causes the tab events a command brings."""

def __init__(self, devtools: DevToolsPipe, *, dialogs_answered: bool = False) -> None:
self.devtools, self.dialogs_answered = devtools, dialogs_answered
def __init__(self, devtools: DevToolsPipe, *, dialogs_answered: bool = False, log: Optional[BrowserLog] = None,
control: Optional[BrowserControl] = None) -> None:
self.devtools, self.dialogs_answered, self.log, self.control = devtools, dialogs_answered, log, control
self.connections: list[dict] = []
self._by_session: dict[str, _Connection] = {}
self._open: set[_Connection] = set()
Expand All @@ -222,7 +237,7 @@ async def serve(self, websocket: WebSocket, user: str) -> None:
await websocket.close(code=1013, reason=f"the browser isn't answering: {type(e).__name__}")
return
record = {"user": user, "opened_at": _now(), "guarded": []}
connection = _Connection(websocket, user, session, record)
connection = _Connection(websocket, user, session, record, len(self.connections) + 1)
self._by_session[session] = connection
self._open.add(connection)
self.connections.append(record)
Expand Down Expand Up @@ -263,25 +278,56 @@ async def _command(self, connection: _Connection, message: Any) -> None:
method, params = message["method"], message.get("params") or {}
session = message.get("sessionId") or connection.session
verdict = guard(method, params, dialogs_answered=self.dialogs_answered)
if session not in connection.sessions:
verdict, answer = None, {"error": NO_SESSION}
elif verdict.action == "refuse":
answer = {"error": {"code": SERVER_ERROR, "message": f"agentenv refuses {method}: {verdict.reason}"}}
elif verdict.action != "send":
answer = {"result": {}}
else:
try:
answer = await self.devtools.request(method, verdict.params, session_id=session)
except ConnectionError as e:
answer = {"error": {"code": SERVER_ERROR, "message": str(e)}}
if verdict is not None and (verdict.action != "send" or verdict.changed):
self._guarded(connection, method, verdict)
reply = {"id": message.get("id"), **{key: answer[key] for key in ("result", "error") if key in answer}}
if message.get("sessionId"):
reply["sessionId"] = message["sessionId"]
self._put(connection, reply)
if verdict is not None and verdict.action == "close":
self._close(connection, CLOSED, dropping=False)
logged = self._begin(connection, session, method, params)
answer, guarded = {"error": {"code": SERVER_ERROR, "message": ENDED_UNANSWERED}}, None
sending: Optional[asyncio.Future] = None
try:
async with connection.order:
if logged is not None:
await logged.check_password()
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved
if session not in connection.sessions:
verdict, answer = None, {"error": NO_SESSION}
elif verdict.action == "refuse":
answer = {"error": {"code": SERVER_ERROR,
"message": f"agentenv refuses {method}: {verdict.reason}"}}
elif verdict.action != "send":
answer = {"result": {}}
else:
sending = asyncio.ensure_future(self.devtools.request(method, verdict.params, session_id=session))
await asyncio.sleep(0)
if sending is not None:
try:
answer = await sending
except ConnectionError as e:
answer = {"error": {"code": SERVER_ERROR, "message": str(e)}}
if verdict is not None and (verdict.action != "send" or verdict.changed):
guarded = GUARDED[verdict.action]
self._guarded(connection, method, verdict)
reply = {"id": message.get("id"), **{key: answer[key] for key in ("result", "error") if key in answer}}
if message.get("sessionId"):
reply["sessionId"] = message["sessionId"]
self._put(connection, reply)
if verdict is not None and verdict.action == "close":
self._close(connection, CLOSED, dropping=False)
finally:
if sending is not None and not sending.done():
sending.cancel()
if logged is not None:
logged.finish(answer, guarded)

def _begin(self, connection: _Connection, session: str, method: str, params: dict) -> Optional[_Logged]:
"""``connection``'s command, begun in the log, if there is one, on the tab ``session`` is on, or for a browser
command the tab it names; what it causes made known to the control."""
if self.log is None:
return None
target = connection.targets.get(session) if session != connection.session else None
if target is None and isinstance(params.get("targetId"), str):
target = params["targetId"]
acting = acts(method)
url = self.control.url_of(target) if self.control is not None and target else None
begun = self.log.begin({"via": "cdp", "who": connection.user, "connection": connection.number,
"method": method, "acts": acting, "tab": target, "url": url})
return _Logged(self, begun, method, params, target, acting)

@staticmethod
def _guarded(connection: _Connection, method: str, verdict: Verdict) -> None:
Expand All @@ -297,8 +343,11 @@ def _event(self, message: dict) -> None:
if message.get("method") == "Target.attachedToTarget" and params.get("sessionId"):
connection.sessions.add(params["sessionId"])
self._by_session[params["sessionId"]] = connection
if isinstance(target := (params.get("targetInfo") or {}).get("targetId"), str):
connection.targets[params["sessionId"]] = target
elif message.get("method") == "Target.detachedFromTarget" and params.get("sessionId"):
connection.sessions.discard(params["sessionId"])
connection.targets.pop(params["sessionId"], None)
self._by_session.pop(params["sessionId"], None)
if message["sessionId"] == connection.session:
message = {key: value for key, value in message.items() if key != "sessionId"}
Expand Down Expand Up @@ -343,6 +392,70 @@ async def _send(self, connection: _Connection) -> None:
await connection.websocket.send_text(message)


class _Logged:
"""A relayed command's entry in the log, from when it arrives until it's answered or its connection ends: what it
causes made known to the control as it's sent, whether a password field has focus on its tab before it types,
and its answer."""

def __init__(self, relay: Relay, begun: dict, method: str, params: dict, target: Optional[str],
acting: bool) -> None:
self.relay, self.begun, self.method, self.params, self.target, self.acting = \
relay, begun, method, params, target, acting
control = relay.control
self._cause = cause = {"seq": begun["seq"], "via": "cdp", "who": begun["action"]["who"]}
self._landed: Optional[asyncio.Future] = None
self._checks_password = self.password = False
if control is None or not acting:
return
if method == "Target.createTarget":
self._landed = asyncio.get_running_loop().create_future()
control.causes.creating(self._landed, cause)
elif method == "Target.closeTarget" and isinstance(params.get("targetId"), str):
control.causes.closing(params["targetId"], cause)
elif method == "Page.navigate" and target:
self._landed = asyncio.get_running_loop().create_future()
control.causes.navigating(target, self._landed, cause)
elif method in NAVIGATES_NEXT and target:
control.causes.navigates_next(target, cause)
self._checks_password = method in TYPING_METHODS and bool(target) and any(field in params
for field in KEY_FIELDS)
#: What a command that types types is withheld until its check says no password field has focus, so one
#: that never gets checked, its connection gone first, keeps it withheld.
self.password = self._checks_password

async def check_password(self) -> None:
"""Before a command that types is sent: whether a password field has focus on its tab, true when that can't
be told."""
if self._checks_password:
self.password = await self.relay.control.password_focused(self.target)

def finish(self, answer: dict, guarded: Optional[str]) -> None:
"""The entry written with its answer, or with why there is none: whether it worked, why not, what the guard
did, and, acting, its parameters as the log keeps them; what it caused made known."""
result = answer.get("result") or {}
if self._landed is not None and not self._landed.done():
if "error" in answer or result.get("errorText"):
self._landed.cancel()
else:
self._landed.set_result(result.get("targetId") if self.method == "Target.createTarget"
else result.get("loaderId"))
fields: dict[str, Any] = {"ok": "error" not in answer}
if "error" in answer:
fields["error"] = redact_text(str((answer["error"] or {}).get("message", answer["error"])))[:ERROR_CHARS]
if self.method in NAVIGATES_NEXT and self.target and self.relay.control is not None:
self.relay.control.causes.navigated_nowhere(self.target, self._cause)
if guarded is not None:
fields["guard"] = guarded
if self.acting:
fields["params"] = redacted_params(self.method, self.params, password=self.password)
opened = result.get("targetId") if self.method == "Target.createTarget" else None
if isinstance(opened, str):
fields["tab"] = self.target = opened
if self.target and self.relay.control is not None and self.begun["action"].get("url") is None:
fields["url"] = self.relay.control.url_of(self.target)
self.relay.log.finish(self.begun, **fields)


def _well_formed(message: Any) -> bool:
"""Whether ``message`` is a command the relay can read: an object with a string ``method``, and object ``params``
and a string ``sessionId`` if any."""
Expand Down
Loading
Loading