Add Jev browser agent loop example

.github/workflows/ci.yml

@@ -21,6 +21,12 @@ jobs:
           cache: npm
       - run: npm run check:lockfile
       - run: npm ci
+      - name: Install Jev example dependencies
+        working-directory: packages/browser-loop/examples/jev-system-one
+        run: npm ci
+      - name: Jev example tests
+        working-directory: packages/browser-loop/examples/jev-system-one
+        run: npm run typecheck && npm test
       # The pi print/RPC test loads the extension the way pi does, through the
       # package's own entry points, so dist has to exist before the unit run.
       - run: npm run build --workspace @onkernel/browser-loop

packages/browser-loop/examples/jev-system-one/README.md

@@ -0,0 +1,126 @@
+# Jev browser agent loop
+
+This example runs a custom browser-agent loop with TypeSafe AI's Jev. It does not register Jev as a chat-model provider or expose Browser Loop tools to Jev. Code observes viewport-visible controls, enumerates a bounded candidate space, asks Jev to choose an operation and target, and executes that target through `BrowserExecutor`.
+
+The loop uses:
+
+- page-specific `CLICK`, `TYPE_TEXT`, `SELECT`, `SCROLL`, and `WAIT` candidates;
+- speculative operation and target questions in one System One request;
+- a small text-model escape hatch only after Jev selects a field or navigation operation;
+- code-owned target guards, step limits, and repeated-no-change detection;
+- `DONE` and `BLOCKED` as explicit Jev choices.
+
+Navigation is part of the loop. The runner opens a blank tab before starting; startup pages expose only navigation and terminal candidates. Jev sees the goal plus the current URL, title, text, elements, values, and recent actions, then chooses `NAVIGATE`. Literal URLs in the task become bounded candidates. Otherwise the text resolver produces the destination URL.
+
+## Data flow
+
+```mermaid
+flowchart LR
+  O[Browser observation] --> C[Build candidate space]
+  C --> J[Jev operation and target]
+  J --> R{Needs text?}
+  R -->|no| L[Lower candidate]
+  R -->|yes| T[Text resolver]
+  T --> L
+  L --> E[BrowserExecutor.execute]
+  E --> O
+```
+
+The two action layers have different responsibilities:
+
+| Layer | Responsibility |
+| --- | --- |
+| `JevCandidateSpace` | Dynamic semantic choices that make sense on the current page |
+| `BrowserAction` / `BrowserActStep` | Fixed Browser Loop execution protocol |
+
+Examples:
+
+| Jev candidate | Browser Loop execution |
+| --- | --- |
+| Click Search | target guard, then `browser_click` at its current viewport point |
+| Type in From | text resolver, guarded click, `CTRL+A`, then `browser_type` |
+| Select Business | guarded native-select update |
+| Navigate | `browser_navigate` |
+| Done / blocked | no browser action |
+
+## Run
+
+Requirements:
+
+- Node.js 22+
+- `KERNEL_API_KEY`
+- `TYPESAFE_API_KEY`
+- `TEXT_MODEL_API_KEY` for tasks that require navigation inference or text entry
+
+The text helper uses an OpenAI-compatible `/chat/completions` endpoint. It limits responses to 1,024 tokens and retries once when a provider returns malformed JSON, before any browser mutation:
+
+```bash
+export TEXT_MODEL_API_KEY="$OPENAI_API_KEY"
+export TEXT_MODEL_BASE_URL="https://api.openai.com/v1"
+export TEXT_MODEL="gpt-5.4-nano"
+# Defaults to none. Use low, medium, or high; provider omits the setting.
+export TEXT_MODEL_REASONING="none"
+```
+
+Install the repository dependencies, then the example's isolated Jev dependency:
+
+```bash
+# Repository root
+npm ci
+
+cd packages/browser-loop/examples/jev-system-one
+npm ci
+npm run typecheck
+npm test
+
+npm run run -- \
+  --task "Open https://news.ycombinator.com, then open the newest submissions page using the new link"
+```
+
+There is intentionally no `--url` argument. The runner opens a blank tab, then initial navigation is selected and executed by the agent loop. The command prints the browser's live-view URL, step timings, and a compact final result to stderr. On macOS, interactive terminal runs also open the live view in the default browser.
+
+```text
+live view: https://...
+[step 1] jev=184ms model=jev-1.13.0 tokens=812/34 operation=99% NAVIGATE "Navigate to https://example.com/"
+[step 1] freshness=91ms changed=false
+[step 1] action=927ms resolve=0ms execute=701ms observe=226ms NAVIGATE "Navigate to https://example.com/" changed=true url=https://example.com/
+[step 2] jev=156ms model=jev-1.13.0 tokens=1041/41 operation=96% target=91% CLICK "Click link More information"
+[result] status=completed elapsed=1487ms steps=2 url=https://example.com/more reason="Jev found visible completion evidence"
+```
+
+Jev timing covers only the System One request. Freshness timing is a target-specific identity and state check rather than another complete observation. Action timing is split into optional text resolution, browser execution, and the single successor observation. Single-step interactions use direct browser primitives with a 10-second deadline; a timeout stops the loop with an unknown execution outcome. `WAIT` retains navigation-safe `browser_act` execution, whose passive-wait path uses one baseline and one successor observation.
+
+## Jev request
+
+Jev receives more than the candidate labels. Every question is conditioned on structured state:
+
+```json
+{
+  "goal": "Open Google Flights and search from SFO to JFK",
+  "page": {
+    "url": "about:blank",
+    "title": "",
+    "text": ""
+  },
+  "elements": [],
+  "recent_actions": []
+}
+```
+
+The operation question contains only currently available operations. Target questions are added for operations with multiple candidates. Jev answers those questions speculatively in the same request; the loop consumes only the target for the selected operation.
+
+## Files
+
+- `agent.ts`: observe/choose/lower/execute loop and safety bounds
+- `actions.ts`: page-specific candidate construction
+- `browser.ts`: `BrowserExecutor` adapter, target validation, and execution
+- `snapshot.ts`: viewport control and text observation
+- `models.ts`: Jev System One operation and target policy
+- `text.ts`: optional OpenAI-compatible string resolver
+- `run.ts`: Kernel browser setup and CLI
+
+## Current boundaries
+
+This is deliberately a custom example rather than a generalized policy API. Its observation pass includes only controls whose center is inside the current viewport, records each control's executable operations from its underlying DOM element, and assigns a stable identity for the life of the document. When a visible cross-origin frame is present, it supplements that state with the frame controls from Browser Loop's stitched accessibility observation. Before input, the runtime validates only the selected control's identity and state. A stale target causes a fresh observation and policy decision; snapshot-scoped references are not remapped.
+
+Editable controls expose separate `TYPE_TEXT` and `Open …` click candidates so Jev can distinguish entering a literal from opening an autocomplete or picker. The example does not generate prose answers, handle CAPTCHA, upload files, or enter passwords. The viewport candidate list is bounded to 250 grounded actions. Page text is treated as untrusted data, and the text resolver returns `null` when required information is absent.

packages/browser-loop/examples/jev-system-one/accessibility.ts

@@ -0,0 +1,175 @@
+import type { ObservationElement } from "./types";
+
+const CLICKABLE_ROLES = new Set(["button", "link", "checkbox", "radio", "switch", "tab", "menuitem", "menuitemcheckbox", "menuitemradio", "treeitem", "option"]);
+
+interface ParsedLine {
+	depth: number;
+	role: string;
+	name: string;
+	ref?: string;
+	states: ReadonlyMap<string, string | boolean | number>;
+}
+
+export function elementsFromAccessibilitySnapshot(snapshot: string, visibleFrameLabels: readonly string[]): ObservationElement[] {
+	const lines = frameDescendants(
+		snapshot.split("\n").map(parseLine).filter((line): line is ParsedLine => line !== undefined),
+		new Set(visibleFrameLabels),
+	);
+	const consumedOptions = new Set<string>();
+	const additions: ObservationElement[] = [];
+	for (let index = 0; index < lines.length; index++) {
+		const line = lines[index]!;
+		if (!line.ref || consumedOptions.has(line.ref)) continue;
+		const states = elementState(line.states);
+		let operations: ObservationElement["operations"] = [];
+		let options: ObservationElement["options"] = [];
+		if (line.role === "combobox") {
+			const descendants = descendantOptions(lines, index);
+			if (descendants.length && states.expanded !== true) {
+				operations = ["SELECT"];
+				options = descendants.map((option) => ({ label: option.name, value: option.name, selected: option.states.get("selected") === true }));
+				for (const option of descendants) if (option.ref) consumedOptions.add(option.ref);
+			} else operations = ["CLICK"];
+		} else if (["textbox", "searchbox", "spinbutton"].includes(line.role)) {
+			operations = ["TYPE_TEXT", "CLICK"];
+		} else if (CLICKABLE_ROLES.has(line.role)) operations = ["CLICK"];
+		if (!operations.length) continue;
+		additions.push({
+			id: `ax:${line.ref}`,
+			node: -Number(line.ref.slice(1)),
+			role: line.role,
+			name: line.name || line.role,
+			value: states.value ?? "",
+			operations,
+			options,
+			...(states.checked === undefined ? {} : { checked: states.checked }),
+			...(states.selected === undefined ? {} : { selected: states.selected }),
+			...(states.expanded === undefined ? {} : { expanded: states.expanded }),
+			...(states.disabled === undefined ? {} : { disabled: states.disabled }),
+			guard: "",
+			ref: line.ref,
+			rect: { x: 0, y: 0, width: 0, height: 0 },
+		});
+	}
+	return additions;
+}
+
+function frameDescendants(lines: readonly ParsedLine[], visibleFrameLabels: ReadonlySet<string>): ParsedLine[] {
+	const descendants: ParsedLine[] = [];
+	const frames: Array<{ depth: number; included: boolean }> = [];
+	for (const line of lines) {
+		while (frames.length && line.depth <= frames[frames.length - 1]!.depth) frames.pop();
+		if (line.role === "Iframe" || line.role === "IframePresentational") {
+			const parent = frames[frames.length - 1];
+			const included = parent?.included ?? visibleFrameLabels.has(line.name);
+			frames.push({ depth: line.depth, included });
+			continue;
+		}
+		if (frames.some((frame) => frame.included)) descendants.push(line);
+	}
+	return descendants;
+}
+
+function parseLine(source: string): ParsedLine | undefined {
+	if (!source.trim() || source.startsWith("… truncated") || source === "(empty accessibility tree)") return undefined;
+	const leading = source.match(/^\s*/)?.[0].length ?? 0;
+	const body = source.slice(leading);
+	const roleEnd = body.indexOf(" ");
+	const role = roleEnd === -1 ? body : body.slice(0, roleEnd);
+	let rest = roleEnd === -1 ? "" : body.slice(roleEnd + 1);
+	let name = "";
+	if (rest.startsWith('"')) {
+		const end = quotedStringEnd(rest);
+		if (end === -1) return undefined;
+		name = JSON.parse(rest.slice(0, end + 1)) as string;
+		rest = rest.slice(end + 1).trimStart();
+	}
+	const groups = [...rest.matchAll(/\[([^\]]*)\]/g)].map((match) => match[1] ?? "");
+	const ref = groups.find((group) => /^e\d+$/.test(group));
+	return {
+		depth: Math.floor(leading / 2),
+		role,
+		name,
+		...(ref ? { ref } : {}),
+		states: parseStates(groups.find((group) => group !== ref) ?? ""),
+	};
+}
+
+function descendantOptions(lines: readonly ParsedLine[], parentIndex: number): ParsedLine[] {
+	const parent = lines[parentIndex]!;
+	const options: ParsedLine[] = [];
+	for (let index = parentIndex + 1; index < lines.length; index++) {
+		const candidate = lines[index]!;
+		if (candidate.depth <= parent.depth) break;
+		if (candidate.role === "option" && candidate.states.get("disabled") !== true) options.push(candidate);
+	}
+	return options;
+}
+
+function parseStates(source: string): ReadonlyMap<string, string | boolean | number> {
+	const states = new Map<string, string | boolean | number>();
+	for (const token of splitStateTokens(source)) {
+		const equals = token.indexOf("=");
+		if (equals === -1) {
+			states.set(token, true);
+			continue;
+		}
+		const key = token.slice(0, equals);
+		const raw = token.slice(equals + 1);
+		try {
+			const value = JSON.parse(raw) as unknown;
+			states.set(key, typeof value === "string" || typeof value === "boolean" || typeof value === "number" ? value : raw);
+		} catch {
+			states.set(key, raw);
+		}
+	}
+	return states;
+}
+
+function splitStateTokens(source: string): string[] {
+	const tokens: string[] = [];
+	let start = 0;
+	let quoted = false;
+	let escaped = false;
+	for (let index = 0; index < source.length; index++) {
+		const character = source[index]!;
+		if (escaped) escaped = false;
+		else if (character === "\\") escaped = true;
+		else if (character === '"') quoted = !quoted;
+		else if (character === "," && !quoted) {
+			tokens.push(source.slice(start, index).trim());
+			start = index + 1;
+		}
+	}
+	const final = source.slice(start).trim();
+	if (final) tokens.push(final);
+	return tokens.filter(Boolean);
+}
+
+function elementState(states: ReadonlyMap<string, string | boolean | number>): {
+	value?: string;
+	checked?: boolean | "mixed";
+	selected?: boolean;
+	expanded?: boolean;
+	disabled?: boolean;
+} {
+	const checked = states.get("checked");
+	return {
+		...(states.has("value") ? { value: String(states.get("value")) } : {}),
+		...(checked === true || checked === false || checked === "mixed" ? { checked } : {}),
+		...(states.has("selected") ? { selected: states.get("selected") === true } : {}),
+		...(states.has("expanded") ? { expanded: states.get("expanded") === true } : {}),
+		...(states.get("disabled") === true ? { disabled: true } : {}),
+	};
+}
+
+function quotedStringEnd(value: string): number {
+	let escaped = false;
+	for (let index = 1; index < value.length; index++) {
+		const character = value[index]!;
+		if (escaped) escaped = false;
+		else if (character === "\\") escaped = true;
+		else if (character === '"') return index;
+	}
+	return -1;
+}