From c60b5b582b61521f254b1215cb3070a2b4acb441 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Tue, 18 Aug 2026 14:33:22 +0200 Subject: [PATCH 1/2] docs(agents): the HUD lift is a cursor poll now, not a WH_MOUSE_LL hook #388 deleted `{ forward: true }`, and this section still described it: `forward` as `@platform darwin,win32`, "Windows installs a global WH_MOUSE_LL hook, macOS forwards through its own event path". None of that is true any more. The main process polls screen.getCursorScreenPoint() while the HUD is click-through and pushes the window-relative point to the renderer, which hit-tests it with elementFromPoint().closest("[data-hud-interactive='true']") -- one path, no platform branch. The RULE is untouched, and that is the part worth being explicit about: an agent still has to move the real cursor, because the poll reads the OS cursor position and CDP-injected input does not change it. Says so, and says what the mechanism used to be, so the next reader who finds `forward` in the git history knows this page is current rather than stale. Also corrects the window size while in here: 600x160 was wrong before #388 -- createHudOverlayWindow builds 820x560 and the renderer then resizes to fit its content (measured 904x698, bar at the bottom, empty reserve above). --- AGENTS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index cb065a87..6bc4910f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -113,8 +113,8 @@ This section is the *mechanics*. **What to actually run is [`technical-documenta - **It is invisible in screenshots by default.** The HUD (and the Notes window) call `setContentProtection(true)` so the recording controls never end up baked into a recording — the same `SetWindowDisplayAffinity` that WGC honours also hides them from *your* screenshots. The window is there, and clicks land, but you are aiming blind at a rectangle you cannot see. Set **`OPENSCREEN_DISABLE_CONTENT_PROTECTION=1`** in the app's environment to turn it off for a session; every skipped window logs a warning. Unset it before recording anything real, or the HUD ends up in the video. - **On macOS 26+ content protection is auto-disabled, so the HUD *is* visible and screenshottable with no flag.** That OS never displays a content-protected window at all — not just absent from captures, but never painted, leaving a tray icon, a live renderer and nothing on screen (confirmed on macOS 26.5 / Electron 41.2.1). `applyContentProtection` therefore skips the call there and logs a warning per window; the trade-off is that the HUD can appear in recordings on that OS until the ScreenCaptureKit helper excludes our own windows via `SCContentFilter(excludingWindows:)`, which it currently passes as `[]`. `OPENSCREEN_FORCE_CONTENT_PROTECTION=1` re-enables it to re-test against a future Electron. - The HUD is what opens the editor (clapper icon, tooltip *Open Studio*), so without that flag a whole slice of the app is unreachable from automation: killing the app to redeploy a native addon leaves you unable to reopen a project. -- Frameless, transparent, always-on-top, `skipTaskbar`, centered at the **bottom of the primary display** (`createHudOverlayWindow`, 600×160). It is **click-through** (`setIgnoreMouseEvents(true, { forward: true })`): moving the real cursor over an interactive control makes that region clickable and shows its tooltip, so `mouse_move` → screenshot → `left_click` works; a blind click on empty HUD area passes through to the desktop. -- **Only a real OS mouse move reaches the HUD — on macOS as much as on Windows.** `forward` is `@platform darwin,win32` in Electron's own typings, and the renderer asks for click-through on both; **Linux is the exception** (`!enabled && !isLinuxHud` in `LaunchWindow.tsx`, where the call is a no-op), so it is the one platform where a blind click on the HUD simply lands. The implementations differ — Windows installs a global `WH_MOUSE_LL` hook, macOS forwards through its own event path — but the consequence is identical: moving the real cursor onto a control is what lifts the input-transparency. CDP-injected input never does that, on any platform: Playwright's `.click()`, `javascript_tool`-dispatched pointer events, and anything else synthesised into the renderer arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD test IDs and stays green for exactly that reason; it proves renderer wiring, not reachability, and a macOS spec written the same way would prove no more. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. +- Frameless, transparent, always-on-top, `skipTaskbar`, centered at the **bottom of the primary display** (`createHudOverlayWindow`, 820×560 at construction, then resized to fit its content — measured 904×698 with the bar at the bottom and mostly empty reserve above it). It is **click-through** (`setIgnoreMouseEvents(ignore)`): moving the real cursor over an interactive control makes that region clickable and shows its tooltip, so `mouse_move` → screenshot → `left_click` works; a blind click on empty HUD area passes through to the desktop. +- **Only a real OS mouse move reaches the HUD — on macOS as much as on Windows.** While the window is input-transparent Chromium delivers it no pointer events at all, so the main process samples the OS cursor instead: the `hud-overlay-cursor` poll in `electron/windows.ts` reads `screen.getCursorScreenPoint()` while the HUD is click-through and pushes the window-relative point to the renderer, which hit-tests it with `elementFromPoint(…).closest("[data-hud-interactive='true']")`. One path, both platforms — there is no platform branch. **Linux is the exception** (`!enabled && !isLinuxHud` in `LaunchWindow.tsx`, where the call is a no-op), so it is the one platform where a blind click on the HUD simply lands. What lifts the input-transparency is therefore a change in the *OS cursor position*, which is precisely why CDP-injected input never does it, on any platform: Playwright's `.click()`, `javascript_tool`-dispatched pointer events, and anything else synthesised into the renderer arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD test IDs and stays green for exactly that reason; it proves renderer wiring, not reachability, and a macOS spec written the same way would prove no more. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. (Until #385 the lift was Electron's `{ forward: true }` — a global `WH_MOUSE_LL` hook on Windows — which Windows can revoke without telling the app, leaving the HUD painted and permanently dead. The poll replaced it. The rule for you is unchanged, because both mechanisms key off the real cursor.) - Control row (left→right): layout preset, **source** button (`Screen`/`Window` → label becomes the picked source), system-audio toggle, mic toggle, **webcam toggle** (shows the detected camera name), cursor-highlight toggle, **record**, notes, open-editor, language, minimize, close. The record button is disabled until a source is chosen (tooltip: "Please select a source to record"). **The tray icon** (bottom-right notification area) From 5fc3d1fcfdbe1fafe9d2b90b4e9ada1f0cac5d05 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Tue, 18 Aug 2026 14:44:26 +0200 Subject: [PATCH 2/2] docs(agents): the poll keys off the cursor RELATIVE to the window CodeRabbit caught an over-strong premise, and it traces back to a deliberate choice in #388: pollHudCursor dedupes on the window-relative point, not on the cursor, precisely because "hud-overlay-set-size" re-anchors the window on every content change and the bar can arrive under a pointer that never moved. So "what lifts the input-transparency is a change in the OS cursor position" was not true -- a resize or re-anchor produces a fresh sample on its own. Reworded to what the poll actually reads, and the conclusion is now tied to the property that is airtight rather than to the one that is merely usual: synthesised input moves no pointer at all, so it can never put one on a control. That is what makes a passing injected click prove renderer wiring and not reachability, which is the whole reason this paragraph exists. No code change: a re-anchor lifting click-through is correct -- the pointer IS over the bar once the bar has moved under it. --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 6bc4910f..91483965 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -114,7 +114,7 @@ This section is the *mechanics*. **What to actually run is [`technical-documenta - **On macOS 26+ content protection is auto-disabled, so the HUD *is* visible and screenshottable with no flag.** That OS never displays a content-protected window at all — not just absent from captures, but never painted, leaving a tray icon, a live renderer and nothing on screen (confirmed on macOS 26.5 / Electron 41.2.1). `applyContentProtection` therefore skips the call there and logs a warning per window; the trade-off is that the HUD can appear in recordings on that OS until the ScreenCaptureKit helper excludes our own windows via `SCContentFilter(excludingWindows:)`, which it currently passes as `[]`. `OPENSCREEN_FORCE_CONTENT_PROTECTION=1` re-enables it to re-test against a future Electron. - The HUD is what opens the editor (clapper icon, tooltip *Open Studio*), so without that flag a whole slice of the app is unreachable from automation: killing the app to redeploy a native addon leaves you unable to reopen a project. - Frameless, transparent, always-on-top, `skipTaskbar`, centered at the **bottom of the primary display** (`createHudOverlayWindow`, 820×560 at construction, then resized to fit its content — measured 904×698 with the bar at the bottom and mostly empty reserve above it). It is **click-through** (`setIgnoreMouseEvents(ignore)`): moving the real cursor over an interactive control makes that region clickable and shows its tooltip, so `mouse_move` → screenshot → `left_click` works; a blind click on empty HUD area passes through to the desktop. -- **Only a real OS mouse move reaches the HUD — on macOS as much as on Windows.** While the window is input-transparent Chromium delivers it no pointer events at all, so the main process samples the OS cursor instead: the `hud-overlay-cursor` poll in `electron/windows.ts` reads `screen.getCursorScreenPoint()` while the HUD is click-through and pushes the window-relative point to the renderer, which hit-tests it with `elementFromPoint(…).closest("[data-hud-interactive='true']")`. One path, both platforms — there is no platform branch. **Linux is the exception** (`!enabled && !isLinuxHud` in `LaunchWindow.tsx`, where the call is a no-op), so it is the one platform where a blind click on the HUD simply lands. What lifts the input-transparency is therefore a change in the *OS cursor position*, which is precisely why CDP-injected input never does it, on any platform: Playwright's `.click()`, `javascript_tool`-dispatched pointer events, and anything else synthesised into the renderer arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD test IDs and stays green for exactly that reason; it proves renderer wiring, not reachability, and a macOS spec written the same way would prove no more. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. (Until #385 the lift was Electron's `{ forward: true }` — a global `WH_MOUSE_LL` hook on Windows — which Windows can revoke without telling the app, leaving the HUD painted and permanently dead. The poll replaced it. The rule for you is unchanged, because both mechanisms key off the real cursor.) +- **Only a real OS mouse move reaches the HUD — on macOS as much as on Windows.** While the window is input-transparent Chromium delivers it no pointer events at all, so the main process samples the OS cursor instead: the `hud-overlay-cursor` poll in `electron/windows.ts` reads `screen.getCursorScreenPoint()` while the HUD is click-through and pushes the window-relative point to the renderer, which hit-tests it with `elementFromPoint(…).closest("[data-hud-interactive='true']")`. One path, both platforms — there is no platform branch. **Linux is the exception** (`!enabled && !isLinuxHud` in `LaunchWindow.tsx`, where the call is a no-op), so it is the one platform where a blind click on the HUD simply lands. What the poll keys off is the OS cursor's position *relative to the window*, so a resize or re-anchor that slides the bar under a motionless pointer produces a fresh sample too. What it can never key off is synthesised input: Playwright's `.click()`, `javascript_tool`-dispatched pointer events and everything like them move no pointer at all, so they never put one on a control. They arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD test IDs and stays green for exactly that reason; it proves renderer wiring, not reachability, and a macOS spec written the same way would prove no more. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. (Until #385 the lift was Electron's `{ forward: true }` — a global `WH_MOUSE_LL` hook on Windows — which Windows can revoke without telling the app, leaving the HUD painted and permanently dead. The poll replaced it. The rule for you is unchanged, because both mechanisms key off the real cursor.) - Control row (left→right): layout preset, **source** button (`Screen`/`Window` → label becomes the picked source), system-audio toggle, mic toggle, **webcam toggle** (shows the detected camera name), cursor-highlight toggle, **record**, notes, open-editor, language, minimize, close. The record button is disabled until a source is chosen (tooltip: "Please select a source to record"). **The tray icon** (bottom-right notification area)