From 0848cf9c468f0ac6aa1a8990492a41cfd1d0e752 Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Wed, 19 Aug 2026 21:14:55 -0400 Subject: [PATCH] docs: correct stale recorder facts and mark the extension a prototype Correct documentation facts that no longer describe the shipped package, and separate the Chrome extension prototype from the supported paths. - CLAUDE.md: the `[audio]` extra does not exist; the transcription extra is `transcribe-fast`. `video.py` drives an external FFmpeg process and does not use PyAV. The recording pipeline uses native input observers, not pynput. Privacy is an optional integration, not automatic scrubbing. - README.md: native capture covers Windows, macOS, and Linux, and window-scoped capture covers an RDP or Citrix client window. Flow's Playwright recorder can now launch Chromium or attach to one existing signed-in local Chromium tab, and its browser recording guide exists, so link it. The Flow adapter compiles right clicks, left-button drags, modifier chords, and scrolling; the old limitation text was wrong. - Mark the Chrome extension, its bridge transport, and its event schemas as a development prototype, and record the promotion contract it must meet. - Mark docs/DESIGN.md and docs/VISUALIZATION_DESIGN.md as historical where they no longer describe the current API. This change carries no lifecycle label movement. The package stays Experimental in README.md and stays `Development Status :: 2 - Pre-Alpha` in pyproject.toml. Capture moves from Experimental to Production in one later visible change, once its qualification evidence exists. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 14 +++++---- README.md | 37 ++++++++++++++++------- chrome_extension/README.md | 47 +++++++++++++++++++++++++++++ chrome_extension/manifest.json | 6 ++-- docs/DESIGN.md | 9 ++++++ docs/VISUALIZATION_DESIGN.md | 5 +++ openadapt_capture/browser_bridge.py | 8 ++++- openadapt_capture/browser_events.py | 6 +++- openadapt_capture/cli.py | 9 ++++-- 9 files changed, 116 insertions(+), 25 deletions(-) create mode 100644 chrome_extension/README.md diff --git a/CLAUDE.md b/CLAUDE.md index 1f1f6bb..3ab167e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,10 +5,12 @@ **openadapt-capture** is the data collection component of the OpenAdapt GUI automation ecosystem. It captures platform-agnostic GUI interaction streams (mouse, keyboard, screen) with time-aligned media for training ML models or replaying workflows. Key responsibilities: -- Record human demonstrations with mouse, keyboard, and screen capture +- Record human demonstrations with native mouse, keyboard, and screen capture - Time-align all events and media (video, audio) - Process raw events into structured actions (clicks, drags, typing) -- Support privacy scrubbing of sensitive data +- Retain optional action-time structural evidence where a provider exists +- Keep raw captures local unless a caller invokes an explicit transfer path +- Expose an optional privacy integration without claiming automatic scrubbing **Always use PRs, never push directly to main** @@ -18,8 +20,8 @@ Key responsibilities: # Install the package uv add openadapt-capture -# Install with audio support (large download) -uv add "openadapt-capture[audio]" +# Install with on-device transcription support (large download) +uv add "openadapt-capture[transcribe-fast]" # Run tests (exclude browser bridge tests which need websockets fixtures) uv run pytest tests/ -v --ignore=tests/test_browser_bridge.py @@ -60,7 +62,7 @@ openadapt_capture/ extensions/ # SynchronizedQueue (multiprocessing.Queue wrapper) utils.py # Timestamps, screenshots, monitor dims config.py # Recording config (RECORD_VIDEO, RECORD_AUDIO, etc.) - video.py # Video encoding (av/ffmpeg) + video.py # External FFmpeg process boundary and frame extraction audio.py # Audio recording + transcription visualize/ # Demo GIF and HTML viewer generation share.py # Magic Wormhole sharing @@ -101,7 +103,7 @@ SQLAlchemy-based per-capture databases: # Fast tests (unit + integration, no recording) uv run pytest tests/ -v --ignore=tests/test_browser_bridge.py -m "not slow" -# Slow tests (full recording pipeline with pynput synthetic input) +# Slow tests (full native recording pipeline with synthetic input) uv run pytest tests/ -v -m slow # All tests diff --git a/README.md b/README.md index 77468c3..a0ea812 100644 --- a/README.md +++ b/README.md @@ -65,14 +65,17 @@ Documentation for the whole stack lives at | Recording path | Current implementation | | --- | --- | -| Windows and RDP demonstrations | `openadapt-capture` records native input and action-gated screen video; `openadapt-flow` converts the session into compiler input. | -| Browser demonstrations | `openadapt-flow` records its Playwright browser directly. It does not require this package. | -| Chrome extension in this repository | Experimental DOM-capture code for development; it is not the supported web recorder or governed replay path. | +| Windows, macOS, and Linux demonstrations | `openadapt-capture` records native input and action-gated screen video; Windows can also retain action-time UI Automation evidence. `openadapt-flow` converts the session into compiler input. | +| RDP and Citrix/VDI demonstrations | `openadapt-capture` records the selected client window in its own pixel space. The remote application remains externally black-box, and `openadapt-flow` converts the session into compiler input. | +| Browser demonstrations | `openadapt-flow` uses its Playwright recorder. It can launch Chromium or attach to one existing signed-in local Chromium tab. It does not require this package. | +| Chrome extension in this repository | Prototype alternate acquisition transport. It is not the supported recorder and must not perform direct replay. | The browser path stays inside `openadapt-flow` because the compiler needs -ordered before/after frames, page state, secret-field redaction, and events in -its own recording format. The extension captures useful DOM context, but it -does not provide that end-to-end contract. +ordered before/after frames, page state, secret-field redaction, and one bound +event schema. Flow now supports an existing authenticated browser session +through its local-loopback CDP attach mode. The extension can become another +acquisition transport after it emits that same evidence contract. It must not +create a second compiler format or bypass governed replay. ## Use it with OpenAdapt @@ -283,9 +286,17 @@ boundary. `openadapt-flow` still refuses desktop `--secret` authoring until its source-time field-redaction contract can prove that sensitive values were not retained. Review the desktop guide before recording sensitive workflows. -The experimental Chrome extension can observe pages across its configured host +The Chrome extension prototype can observe pages across its configured host permissions and can emit DOM text and keyboard events to a local WebSocket. -Treat it as development code; do not deploy it in a sensitive browser profile. +It does not yet provide source-time secret exclusion, authenticated +profile/tab/document/session binding, acknowledged ordered delivery, or exact +frame-to-event evidence. Its direct DOM replay does not use Flow's identity, +policy, fresh-frame, and effect checks. Treat it as development code. Do not +deploy it in a sensitive browser profile. + +Use Flow's supported attach recorder when an existing SSO or 2FA browser +session is required. See the +[`openadapt-flow` browser recording guide](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/docs/BROWSER_RECORDING.md). ## Current limitations @@ -293,10 +304,14 @@ Treat it as development code; do not deploy it in a sensitive browser profile. screen-recording and input-monitoring permissions. - Native Windows capture retains UIA evidence when the application exposes it; opaque remote applications still require Flow's visual and OCR bindings. -- The Flow adapter rejects unsupported input such as drag, non-left-click, and - modifier-chord actions instead of silently compiling an incomplete workflow. +- The Flow adapter compiles left and right clicks, left-button drags, typed + text, named keys, modifier chords, and scrolling. It rejects unsupported + input such as middle clicks, non-left-button drags, malformed shortcuts, and + unmapped keys instead of silently compiling an incomplete workflow. - Browser-extension installation, security hardening, and compiler integration - are not part of the current product path. + are not part of the current product path. Promotion requires the shared Flow + schema, source-time secret exclusion, authenticated and sequenced delivery, + exact frame binding, compiler integration, and removal of direct replay. See the organization-wide [repository lifecycle registry](https://github.com/OpenAdaptAI/.github/blob/main/REPOSITORY_LIFECYCLE.md) diff --git a/chrome_extension/README.md b/chrome_extension/README.md new file mode 100644 index 0000000..1d0e59b --- /dev/null +++ b/chrome_extension/README.md @@ -0,0 +1,47 @@ +# OpenAdapt Capture Chrome extension prototype + +This directory is a development prototype. It is not the supported OpenAdapt +browser recorder or a governed replay path. + +Use the Playwright recorder in `openadapt-flow` for browser workflows. It can +launch a clean Chromium browser or attach to one existing signed-in local +Chromium tab. Both modes retain the compiler's event, DOM identity, exact +before/after frame, field geometry, and source-time secret-redaction contract. +See the +[`openadapt-flow` browser recording guide](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/docs/BROWSER_RECORDING.md). + +## Current prototype behavior + +The extension can collect DOM events and visible HTML and send them to the +Capture WebSocket bridge on `localhost:8765`. It also contains legacy direct +DOM replay code. + +Do not use it in a sensitive browser profile. The current implementation does +not provide these supported-path controls: + +- source-time password and declared-secret exclusion; +- authenticated profile, tab, document, run, and recording-session binding; +- an acknowledged, ordered event sequence with reconnect recovery; +- exact retained frame-to-event binding in one coordinate system; +- compiler integration through the shared Flow recording schema; or +- governed replay with identity, policy, fresh-frame, and effect checks. + +## Promotion contract + +This extension can become a supported alternate acquisition transport. It +must first: + +1. Emit the shared Flow event and evidence schema. +2. Remove direct replay. All execution must use the governed Flow runtime. +3. Exclude secrets before an event leaves the content script. +4. Authenticate and bind the browser profile, tab, document, run, session, and + monotonic event sequence. +5. Acknowledge or safely resume every event after a reconnect. It must not drop + an event and report success. +6. Bind each action to exact before/after frames and viewport metadata. +7. Pass at least three trials for record and compile, secret exclusion, + ambiguity refusal, reconnect behavior, and browser lifecycle preservation. + +Until these items pass the same acceptance gate as the Playwright recorder, +this directory stays a prototype component. This status applies only to this +directory. It does not apply to the `openadapt-capture` package. diff --git a/chrome_extension/manifest.json b/chrome_extension/manifest.json index 289ae80..fe31f6c 100644 --- a/chrome_extension/manifest.json +++ b/chrome_extension/manifest.json @@ -1,8 +1,8 @@ { "manifest_version": 3, - "name": "OpenAdapt Capture", + "name": "OpenAdapt Capture (Prototype)", "version": "1.0.0", - "description": "DOM-level event capture for browser interactions with coordinate tracking and semantic element references", + "description": "Development prototype for DOM event capture; use openadapt-flow for supported browser recording", "permissions": [ "activeTab", "tabs", @@ -34,6 +34,6 @@ "48": "icons/icon48.png", "128": "icons/icon128.png" }, - "default_title": "OpenAdapt Capture" + "default_title": "OpenAdapt Capture prototype" } } diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 3cb762c..383284d 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -1,5 +1,14 @@ # openadapt-capture Design +> **Historical design record.** This file retains early goals and proposed +> formats. It is not the current package contract. The current recorder +> writes `recording.db` plus verified time-aligned MP4 media, uses native input +> observers on Windows, macOS, and Linux, retains action-time Windows UIA +> evidence, and supports window-scoped capture on Windows and macOS. See +> [`README.md`](../README.md) and the public API for the current behavior. Items +> such as `capture.db`, chunked continuous capture, and structural observers on +> every operating system remain historical proposals where the code differs. + ## Problem Statement We need a platform-agnostic representation of GUI interactions that: diff --git a/docs/VISUALIZATION_DESIGN.md b/docs/VISUALIZATION_DESIGN.md index ab5f65e..ed83730 100644 --- a/docs/VISUALIZATION_DESIGN.md +++ b/docs/VISUALIZATION_DESIGN.md @@ -1,5 +1,10 @@ # Visualization Design +> **Historical design record.** This file contains early visualization +> proposals. The current API and privacy rules are in [`README.md`](../README.md) +> and `openadapt_capture.visualize`. In particular, the current HTML viewer does +> not embed recorded audio by default. + ## Goals 1. **README Demo**: Automated GIF/video generation showing a recording with visual event overlays diff --git a/openadapt_capture/browser_bridge.py b/openadapt_capture/browser_bridge.py index 6a8dff9..729c26b 100644 --- a/openadapt_capture/browser_bridge.py +++ b/openadapt_capture/browser_bridge.py @@ -1,4 +1,10 @@ -"""WebSocket server for Chrome extension communication. +"""Prototype WebSocket transport for Chrome extension development. + +This module is not the supported OpenAdapt browser recorder or a governed +replay path. Use the Playwright launch or attach recorder in ``openadapt-flow`` +for compile-ready browser demonstrations. The transport remains available for +development while it gains authenticated session binding, acknowledged +ordering, source-time secret exclusion, and exact event/frame evidence. This module provides the BrowserBridge WebSocket server that connects to the Chrome extension for capturing browser DOM events. It handles: diff --git a/openadapt_capture/browser_events.py b/openadapt_capture/browser_events.py index c03630f..441252b 100644 --- a/openadapt_capture/browser_events.py +++ b/openadapt_capture/browser_events.py @@ -1,4 +1,8 @@ -"""Browser event schemas for Chrome extension communication. +"""Event schemas for the Chrome extension development prototype. + +These passive schemas do not make the extension a supported recorder or its +legacy direct DOM replay a governed execution path. The supported browser +recorder is the Playwright launch or attach path in ``openadapt-flow``. This module defines Pydantic models for all browser event types captured by the Chrome extension. These events include DOM-level interactions with rich diff --git a/openadapt_capture/cli.py b/openadapt_capture/cli.py index 70c7aa3..3490d6e 100644 --- a/openadapt_capture/cli.py +++ b/openadapt_capture/cli.py @@ -34,9 +34,10 @@ def record( unless RECORD_AUDIO_RETAIN_WAVEFORM is explicitly enabled. Requires an on-device transcription backend to be installed. images: Also save screenshots as PNGs (default: False). - browser_events: Capture browser DOM events via Chrome extension (default: False). - Requires the openadapt-capture Chrome extension to be installed and - connects via WebSocket on localhost:8765. + browser_events: Enable the development Chrome extension prototype + (default: False). It is not the supported browser recorder. Use + openadapt-flow browser launch or attach mode for a compile-ready + recording. The prototype connects to localhost:8765. send_profile: Send profiling data via wormhole after recording (default: False). window_owner: Owner-app substring for window-scoped recording — capture ONE window in its own pixel space (e.g. --window-owner Parallels). @@ -86,6 +87,8 @@ def record( if window: print(f"Window-scoped: owner={window_owner!r} title={window_title!r}") if browser_events: + print("PROTOTYPE: Capture Chrome-extension events are not a supported path.") + print("Use openadapt-flow browser launch or attach mode for real workflows.") print("Browser event capture enabled (WebSocket on localhost:8765)") print("Make sure the openadapt-capture Chrome extension is installed.") print("Press Ctrl+C or type stop sequence to stop recording...")