From 3cf7eacf05b3dd9e2a03a59c300188f31174b63b Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Tue, 21 Jul 2026 11:20:32 +0000 Subject: [PATCH 01/15] First published version of design --- docs/source/capture-api-design.md | 396 ++++++++++++++++++++++++++++++ 1 file changed, 396 insertions(+) create mode 100644 docs/source/capture-api-design.md diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md new file mode 100644 index 00000000..08508b22 --- /dev/null +++ b/docs/source/capture-api-design.md @@ -0,0 +1,396 @@ +--- +orphan: true +--- + +# Source-bound capture API + +Status: design proposal for issues [#470](https://github.com/BoboTiG/python-mss/issues/470) and +[#544](https://github.com/BoboTiG/python-mss/issues/544). + +## Overview + +`MSS` is a platform session. It enumerates sources and creates source-bound `Capture` objects. CPU captures continue +returning `ScreenShot` objects. + +```python +with MSS() as session: + monitor = session.list_monitors()[0] + + with session.create_capture(monitor) as capture: + image = capture.grab() +``` + +The source types are explicit. A region restricts a source; it is not itself a source. + +```python +CaptureSource = Desktop | Monitor | Window + + +@dataclass(frozen=True, slots=True) +class Region: + left: int + top: int + width: int + height: int +``` + +```python +capture = session.create_capture( + session.desktop, + region=Region(left=10, top=20, width=640, height=480), +) +``` + +## Public types + +### Sources + +`Desktop`, `Monitor`, and `Window` are read-only. Enumeration returns a new snapshot on each call. + +```python +session.desktop +session.list_monitors() # tuple[Monitor, ...]; no desktop entry +session.list_windows() # tuple[Window, ...] +``` + +Source coordinate spaces use physical pixels: + +```text +Desktop virtual-desktop coordinates; its origin may be negative +Monitor (0, 0) is the displayed monitor's upper-left pixel +Window (0, 0) is the selected window area's upper-left pixel +``` + +Displayed orientation determines source dimensions. Callers do not apply a rotation transform. + +`Monitor` supports attributes. It may retain string-key access temporarily for migration. + +```python +monitor.width +monitor["width"] # Compatibility access. +``` + +### Windows + +```python +class Window: + id: int + title: str + pid: int | None + exe: str | None + class_name: str | None + bounds: Region + visible: bool + minimized: bool + attributes: Mapping[str, object] +``` + +Core properties describe the enumeration snapshot. Expensive properties such as `exe` may be loaded lazily and return +`None` if the process disappears or access is denied. A `Window` belongs to the session that enumerated it. Window +objects initially use object identity for equality; callers can compare native IDs explicitly. + +`list_windows()` includes top-level application windows and minimized windows. Hidden windows require +`include_hidden=True`. Child controls, shell surfaces, menus, tooltips, and similar transient windows are excluded where +the platform can identify them reliably. + +```python +WindowSelector = Callable[[tuple[Window, ...]], Window | None] + + +def find_window( + self, + selector: WindowSelector, + include_hidden: bool = False, +) -> Window | None: ... + + +def get_window( + self, + selector: WindowSelector, + include_hidden: bool = False, +) -> Window: ... +``` + +MSS supplies selectors for common cases: + +```python +session.find_window(window_by_id(hwnd)) +session.find_window(window_by_title("Game")) +session.find_window(window_by_title(re.compile(r"(^| - )Firefox$"))) +session.find_window(window_by_properties(pid=12345, exe="game.exe")) +session.find_window(lambda windows: choose_window(windows)) +``` + +Built-in selectors return `None` for no match and raise `WindowSelectionError` for multiple matches. String matching is +exact and case-sensitive; regular expressions use `search()`. A custom selector must return one of the supplied windows +or `None`, and its exceptions propagate unchanged. `get_window()` converts a `None` result to `WindowSelectionError`. + +Window identity is native and does not silently follow an application through native window recreation. A destroyed +window remains lost even if another window later has the same title, PID, class, or native ID. + +## Capture creation + +```python +class CaptureCapability(Flag): + NONE = 0 + FRAME_NOTIFICATIONS = auto() + PRESENTATION_TIMESTAMPS = auto() + SOURCE_MISSED_FRAME_COUNT = auto() +``` + +```python +def create_capture( + self, + source: CaptureSource, + *, + region: Region | None = None, + backend: str = "auto", + required_capabilities: CaptureCapability = CaptureCapability.NONE, + with_cursor: bool = False, + client_area_only: bool = False, +) -> Capture[ScreenShot]: ... +``` + +`client_area_only` is valid only for a `Window`. Its default captures the complete native window, including non-client +decorations. Regions are relative to the selected complete-window or client-area extent. + +`with_cursor` is strict. `True` guarantees cursor inclusion and `False` guarantees exclusion. Automatic backend +selection considers only providers that can satisfy the requested value. Cursor pixels are composited only where they +intersect the final clipped output. With cursor inclusion enabled, cursor movement, shape changes, and visibility +changes count as output updates for `frames()`. + +Configuration is separate from capability flags: source type, cursor inclusion, client-area capture, regions, and CPU +output are requirements already expressed by the capture request. + +### Regions + +Region fields are integers. Negative `left` and `top` values are valid. Negative width or height raises `ValueError`; +zero is valid. The effective rectangle is recomputed for each acquisition: + +```python +effective = requested_region.intersection(source.bounds) +``` + +The returned image describes the clipped rectangle: + +```python +image.pos == Pos(effective.left, effective.top) +image.size == Size(effective.width, effective.height) +``` + +A completely clipped region returns an empty `ScreenShot` without a native pixel copy. Continuous capture still +observes source updates because a resized source may make the region nonempty later. For an empty intersection, the +position is the requested origin clamped to the nearest source boundary. + +### Backend selection + +```python +@dataclass(frozen=True, slots=True) +class Backend: + name: str + capabilities: CaptureCapability + + +class BackendFailure(NamedTuple): + name: str + reason: str +``` + +```python +capture.backend.name +capture.backend.capabilities +capture.backend_failures # tuple[BackendFailure, ...] +``` + +For `backend="auto"`, MSS filters a documented platform/source priority list by the capture configuration and required +capabilities, then tries eligible providers in order. `backend_failures` records each higher-priority provider considered +before the successful one and why it could not be used. If none succeeds, `BackendUnavailableError.failures` contains +the same records. + +An explicit backend never falls back. Automatic fallback occurs only during capture creation; recovery never changes a +capture's provider. Priority does not change in patch releases. Minor releases append new providers behind established +eligible defaults; established defaults may be reordered in a major release. Providers may be disabled sooner for +correctness, security, or platform compatibility. + +## CPU results and timing + +The default result remains `ScreenShot`. It contains CPU-addressable, tightly packed, top-to-bottom BGRA/BGRX bytes: + +```text +row stride width * 4 +channels B, G, R, A/X +alpha not guaranteed meaningful +``` + +Backends may acquire another native format but convert as necessary before returning a CPU `ScreenShot`. + +```python +@dataclass(frozen=True, slots=True) +class FrameTiming: + sequence: int + source_generation: int + source_sequence: int | None + presented_at_ns: int | None + acquired_at_ns: int + ready_at_ns: int + delivered_at_ns: int + + +@dataclass(frozen=True, slots=True) +class CaptureStatistics: + acquired_frames: int + delivered_frames: int + dropped_frames: int + source_missed_frames: int | None +``` + +Built-in `ScreenShot` results from the new API always have `image.timing`; directly constructed and legacy screenshots +may have `timing=None`. All times use one documented monotonic nanosecond clock. `presented_at_ns` is `None` unless the +backend supplies a reliable presentation time in that clock domain. Acquisition time is never reported as presentation +time. + +`sequence` is capture-wide and monotonic across generator restarts and native recovery. Gaps reveal MSS-side drops. +`source_sequence` belongs to `source_generation`; the generation increments when recovery changes the native sequence +domain. Repeated `grab()` results receive new MSS sequences but may share a source sequence and presentation time. + +`capture.statistics` returns an immutable, thread-safe snapshot and remains available after loss, failure, or closure. +`acquired_frames` counts backend updates accepted by MSS, including updates later dropped. `delivered_frames` counts +successful returns and yields. `source_missed_frames` is `None` unless the backend can count misses reliably. + +`MSS.cls_image` is snapshotted when a capture is created. The custom image constructor receives: + +```python +image_class(data, region, size=actual_size, timing=timing) +``` + +Custom classes may ignore optional keywords. Changing `session.cls_image` affects future captures, not an existing +capture's result type. + +## Pull and continuous capture + +```python +image = capture.grab(timeout=1.0) + +for image in capture.frames( + buffer_count=1, + overflow="drop_oldest", + timeout=None, +): + process(image) +``` + +`grab()` returns a newly constructed current image whenever the backend can complete the request. Repeated calls may +contain identical pixels. It does not promise a distinct source presentation or a particular rate. + +`frames()` requires `FRAME_NOTIFICATIONS` and yields only distinct backend-reported output updates. Equal pixels may be +yielded for distinct presentations. MSS does not compare images to infer updates and does not synthesize duplicates. +Calling it without the capability raises `UnsupportedCaptureOperationError`. + +`buffer_count` is a positive integer and counts completed images waiting for delivery, excluding native frame pools and +the image held by the consumer. The supported overflow policies are: + +```text +drop_oldest discard the oldest pending image and queue the newest; default +block stop draining native updates until delivery space is available +``` + +Native APIs may coalesce or miss updates while blocked; this is not an MSS-side drop. Pending images discarded when a +generator ends count as dropped. Version one has no target-FPS or duplicate-frame video mode. + +`timeout` limits each wait for an image. `FrameTimeoutError` terminates that generator but leaves the capture reusable. +`SourceLostError` and `CaptureFailedError` terminate it and leave the capture terminal. Closing the capture ends an +active generator normally. + +Only one `frames()` generator may be active. It becomes active on its first iteration. While active, `grab()` or +advancing another generator raises `CaptureBusyError`. Concurrent `grab()` calls are serialized. An individual generator +must not be advanced concurrently from multiple threads. + +```python +stream = capture.frames() +try: + for image in stream: + if process_and_finish(image): + break +finally: + stream.close() # Required when retaining a generator and leaving early. +``` + +Closing or exhausting the generator returns the capture to its open state. A timeout permits another generator on the +same capture; source loss requires selecting a source and creating a new capture. + +## Lifetime and recovery + +`Capture` is an idempotent context manager: + +```python +with session.create_capture(source) as capture: + image = capture.grab() + +capture.close() +capture.close() # No effect. +``` + +The caller owns captures and should close them promptly. The session weakly tracks live captures and closes them before +closing shared platform resources. Returned image storage remains valid after both capture and session closure. + +```text +OPEN +├── first next(frames) ─────────> STREAMING +├── source disappears ──────────> LOST +├── unrecoverable provider error ─> FAILED +└── close() ────────────────────> CLOSED + +STREAMING +├── generator close/timeout ────> OPEN +├── source disappears ──────────> LOST +├── unrecoverable provider error ─> FAILED +└── close() ────────────────────> CLOSED +``` + +MSS recovers transparently while it can prove the same source identity and capture contract remain available. Examples +include device reset, DXGI duplication invalidation, frame-pool recreation, window resizing, and resolution or +orientation changes for the same monitor. Recovery stays within the selected provider and preserves required +capabilities. + +A minimized, hidden, or temporarily unavailable window is not lost. `frames()` waits for another update and may time +out; `grab()` may construct a new result from the latest retained source image. Window destruction and monitor unplug +are terminal source loss. Recreated windows, reconnected monitors, matching titles, matching geometry, and reused list +indices do not silently retarget a capture. `Desktop` persists across monitor-topology changes. + +Pending delivery images are discarded on native-generation replacement, source loss, or terminal failure. Previously +returned images remain valid. Public timeouts include time spent recovering; `timeout=None` uses finite internal waits +so close, loss, and failure remain observable. + +## Exceptions + +```text +MSSError +├── ScreenShotError legacy API +├── SessionClosedError +├── WindowSelectionError +├── BackendUnavailableError +└── CaptureError + ├── FrameTimeoutError capture reusable + ├── SourceLostError terminal LOST + ├── CaptureFailedError terminal FAILED + ├── CaptureClosedError + ├── CaptureBusyError + └── UnsupportedCaptureOperationError +``` + +Invalid argument types use `TypeError`; invalid values use `ValueError`. Window-selector callback exceptions propagate +unchanged. `CaptureFailedError` chains its native cause. + +## Compatibility and deferred work + +The deprecated `MSS.grab()`, `MSS.monitors`, `save()`, and `shot()` retain their current desktop-rectangle semantics and +return types. Their legacy backend is initialized lazily. Constructor-level `backend=` and `with_cursor=` configure only +that path; new capture selection belongs to `create_capture()`. Compatibility helpers do not call deprecated public +methods internally. + +The new API uses `MSSError` and `CaptureError`; `ScreenShotError` remains the legacy error type. + +THis version 1.0 of the new interface deliberately defers GPU result types and more capture-native formats (CPU or GPU). +Internally, capture implementations are generic over their result type and +producer/queue delivery so these improvements can be added later without changes +to the overall design. From 3a8249fb20334a4f43f5f597f7ff29792fb2a994 Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Wed, 22 Jul 2026 15:53:00 +0000 Subject: [PATCH 02/15] Update design --- docs/source/capture-api-design.md | 303 ++++++++++++++---------------- 1 file changed, 140 insertions(+), 163 deletions(-) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index 08508b22..94991948 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -7,10 +7,13 @@ orphan: true Status: design proposal for issues [#470](https://github.com/BoboTiG/python-mss/issues/470) and [#544](https://github.com/BoboTiG/python-mss/issues/544). +Decisions below reflect discussion on #544 through 2026-07-22. Sections marked **Open** or **Deferred** are +not settled. + ## Overview -`MSS` is a platform session. It enumerates sources and creates source-bound `Capture` objects. CPU captures continue -returning `ScreenShot` objects. +`MSS` is a platform session. It enumerates sources and creates source-bound `Capture` objects. Captures return +`ScreenShot` objects (see [Results](#results)). ```python with MSS() as session: @@ -18,9 +21,10 @@ with MSS() as session: with session.create_capture(monitor) as capture: image = capture.grab() + image = capture.grab(region=Region(left=10, top=20, width=640, height=480)) ``` -The source types are explicit. A region restricts a source; it is not itself a source. +The source types are explicit. A region restricts a source at acquisition time; it is not itself a source. ```python CaptureSource = Desktop | Monitor | Window @@ -34,18 +38,11 @@ class Region: height: int ``` -```python -capture = session.create_capture( - session.desktop, - region=Region(left=10, top=20, width=640, height=480), -) -``` - ## Public types ### Sources -`Desktop`, `Monitor`, and `Window` are read-only. Enumeration returns a new snapshot on each call. +`Desktop`, `Monitor`, and `Window` are read-only. Enumeration returns a new immutable snapshot on each call. ```python session.desktop @@ -53,15 +50,26 @@ session.list_monitors() # tuple[Monitor, ...]; no desktop entry session.list_windows() # tuple[Window, ...] ``` -Source coordinate spaces use physical pixels: +Return type is `tuple[...]` so the snapshot cannot be resized or reassigned in place. -```text -Desktop virtual-desktop coordinates; its origin may be negative -Monitor (0, 0) is the displayed monitor's upper-left pixel -Window (0, 0) is the selected window area's upper-left pixel -``` +#### Coordinate spaces — **Open** + +**Inputs** (windows, monitors, desktop, regions) are always in **display-oriented** pixels: a 90°-rotated 1920×1080 +panel is reported and addressed as width 1080, height 1920. Callers do not apply a rotation transform when specifying +what to capture. + +**Pixel space (physical / framebuffer vs logical / nominal)** is unresolved. Today Windows and Linux/X11 use +framebuffer pixels (geometry matches the buffer); macOS defaults to nominal/logical via +`kCGWindowImageNominalResolution`. Because the new API keeps `ScreenShot` as the result type shared with the legacy +path, there is no separate type with which to quarantine a new contract. Options under discussion on #544: -Displayed orientation determines source dimensions. Callers do not apply a rotation transform. +1. Keep the platform inconsistency on `ScreenShot` (document it). +2. Unify on physical/framebuffer everywhere in a major release (breaking on Retina Macs that rely on nominal size). +3. Make the space explicit on each result (attribute / scale factor). + +Until decided, this document does not claim “physical pixels everywhere.” + +### Monitors `Monitor` supports attributes. It may retain string-key access temporarily for migration. @@ -86,8 +94,10 @@ class Window: ``` Core properties describe the enumeration snapshot. Expensive properties such as `exe` may be loaded lazily and return -`None` if the process disappears or access is denied. A `Window` belongs to the session that enumerated it. Window -objects initially use object identity for equality; callers can compare native IDs explicitly. +`None` if the process disappears or access is denied. A `Window` belongs to the session that enumerated it. + +`Window` equality is **object identity**. Native IDs can be recycled after destroy, so `__eq__` is not based on `id`. +Within the same session, callers who need “same OS window” compare `window.id` while that window still exists. `list_windows()` includes top-level application windows and minimized windows. Hidden windows require `include_hidden=True`. Child controls, shell surfaces, menus, tooltips, and similar transient windows are excluded where @@ -102,13 +112,6 @@ def find_window( selector: WindowSelector, include_hidden: bool = False, ) -> Window | None: ... - - -def get_window( - self, - selector: WindowSelector, - include_hidden: bool = False, -) -> Window: ... ``` MSS supplies selectors for common cases: @@ -123,7 +126,7 @@ session.find_window(lambda windows: choose_window(windows)) Built-in selectors return `None` for no match and raise `WindowSelectionError` for multiple matches. String matching is exact and case-sensitive; regular expressions use `search()`. A custom selector must return one of the supplied windows -or `None`, and its exceptions propagate unchanged. `get_window()` converts a `None` result to `WindowSelectionError`. +or `None`, and its exceptions propagate unchanged. Window identity is native and does not silently follow an application through native window recreation. A destroyed window remains lost even if another window later has the same title, PID, class, or native ID. @@ -136,6 +139,11 @@ class CaptureCapability(Flag): FRAME_NOTIFICATIONS = auto() PRESENTATION_TIMESTAMPS = auto() SOURCE_MISSED_FRAME_COUNT = auto() + + +class WindowArea(Enum): + CLIENT = auto() # client area / content rect / client window + FULL = auto() # entire native window, including non-client chrome ``` ```python @@ -143,44 +151,61 @@ def create_capture( self, source: CaptureSource, *, - region: Region | None = None, backend: str = "auto", required_capabilities: CaptureCapability = CaptureCapability.NONE, - with_cursor: bool = False, - client_area_only: bool = False, -) -> Capture[ScreenShot]: ... + with_cursor: bool | None = None, + area: WindowArea | None = None, +) -> Capture: ... ``` -`client_area_only` is valid only for a `Window`. Its default captures the complete native window, including non-client -decorations. Regions are relative to the selected complete-window or client-area extent. +`area` is valid only with a `Window` source. Invalid for `Monitor` / `Desktop`. Region coordinates on later +`grab` calls are relative to the chosen extent: with `CLIENT`, `(0, 0)` is the client/content origin; with +`FULL`, `(0, 0)` is the full native window origin. + +**Open:** default for `area` when capturing a window (`CLIENT` vs `FULL`). Lean `CLIENT` unless discussion settles +otherwise. + +`with_cursor` is tri-state for backend selection: + +```text +True cursor must be included; only backends that can guarantee that are eligible +False cursor must be excluded; same filter the other way +None don't care (default); auto may pick the best otherwise-eligible backend; + cursor presence is unspecified +``` -`with_cursor` is strict. `True` guarantees cursor inclusion and `False` guarantees exclusion. Automatic backend -selection considers only providers that can satisfy the requested value. Cursor pixels are composited only where they -intersect the final clipped output. With cursor inclusion enabled, cursor movement, shape changes, and visibility -changes count as output updates for `frames()`. +When cursor inclusion is required (`True`), cursor pixels are composited only where they intersect the final clipped +output, and cursor movement, shape changes, and visibility changes count as output updates. -Configuration is separate from capability flags: source type, cursor inclusion, client-area capture, regions, and CPU -output are requirements already expressed by the capture request. +Configuration is separate from capability flags: source type, cursor preference, window area, and regions are +requirements already expressed by the capture request. ### Regions -Region fields are integers. Negative `left` and `top` values are valid. Negative width or height raises `ValueError`; -zero is valid. The effective rectangle is recomputed for each acquisition: +`region` is **not** a `create_capture` argument. It is optional on `grab()`: ```python -effective = requested_region.intersection(source.bounds) +capture.grab() +capture.grab(region=Region(left=10, top=20, width=640, height=480)) ``` -The returned image describes the clipped rectangle: +`None` captures the full source extent (subject to `area` for windows). Region may differ across `grab` calls. + +Region fields are integers. Negative `left` and `top` values are valid. Negative width or height raises `ValueError`; +zero is valid. The effective rectangle is recomputed for each acquisition: ```python -image.pos == Pos(effective.left, effective.top) -image.size == Size(effective.width, effective.height) +effective = requested_region.intersection(source_extent) ``` -A completely clipped region returns an empty `ScreenShot` without a native pixel copy. Continuous capture still -observes source updates because a resized source may make the region nonempty later. For an empty intersection, the -position is the requested origin clamped to the nearest source boundary. +The returned image describes the clipped rectangle. A completely clipped region returns an empty `ScreenShot` without a +native pixel copy. Continuous capture still observes source updates because a resized source may make the region +nonempty later. For an empty intersection, the position is the requested origin clamped to the nearest source boundary. + +Insets / negative width-height as a crop-from-edges sugar remain deferred. + +If a backend cannot re-crop without reinit, `create_capture` still succeeds; the first incompatible `grab` +request raises `UnsupportedCaptureOperationError`. Prefer backends that can re-crop. ### Backend selection @@ -198,8 +223,8 @@ class BackendFailure(NamedTuple): ```python capture.backend.name -capture.backend.capabilities -capture.backend_failures # tuple[BackendFailure, ...] +capture.backend.capabilities # introspection / debugging after auto-select +capture.backend_failures # tuple[BackendFailure, ...] ``` For `backend="auto"`, MSS filters a documented platform/source priority list by the capture configuration and required @@ -212,115 +237,57 @@ capture's provider. Priority does not change in patch releases. Minor releases a eligible defaults; established defaults may be reordered in a major release. Providers may be disabled sooner for correctness, security, or platform compatibility. -## CPU results and timing - -The default result remains `ScreenShot`. It contains CPU-addressable, tightly packed, top-to-bottom BGRA/BGRX bytes: +## Results -```text -row stride width * 4 -channels B, G, R, A/X -alpha not guaranteed meaningful -``` +The result type remains **`ScreenShot`**. It is not replaced by a separate `Frame` type. -Backends may acquire another native format but convert as necessary before returning a CPU `ScreenShot`. - -```python -@dataclass(frozen=True, slots=True) -class FrameTiming: - sequence: int - source_generation: int - source_sequence: int | None - presented_at_ns: int | None - acquired_at_ns: int - ready_at_ns: int - delivered_at_ns: int +Future direction (not v1): `ScreenShot` as a base with `ScreenShotCpu` and `ScreenShotGpu` subclasses sharing common +attributes; CPU and GPU results expose different buffers. +### Buffer layout -@dataclass(frozen=True, slots=True) -class CaptureStatistics: - acquired_frames: int - delivered_frames: int - dropped_frames: int - source_missed_frames: int | None -``` +`ScreenShot` does **not** require tightly packed rows. Backends may return a native stride/pitch. Contiguous packed +BGRA is obtained lazily via `.bgra` (may copy). NumPy/PIL/PyTorch and similar consumers can use the native layout +directly when they support strides. -Built-in `ScreenShot` results from the new API always have `image.timing`; directly constructed and legacy screenshots -may have `timing=None`. All times use one documented monotonic nanosecond clock. `presented_at_ns` is `None` unless the -backend supplies a reliable presentation time in that clock domain. Acquisition time is never reported as presentation -time. +This would mean that legacy `MSS.grab()` stops returning packed buffers as today, but they can be obtained via attributes as +described above. -`sequence` is capture-wide and monotonic across generator restarts and native recovery. Gaps reveal MSS-side drops. -`source_sequence` belongs to `source_generation`; the generation increments when recovery changes the native sequence -domain. Repeated `grab()` results receive new MSS sequences but may share a source sequence and presentation time. +Exact pixel-format negotiation (HDR, YUV, etc.) is deferred with GPU work. -`capture.statistics` returns an immutable, thread-safe snapshot and remains available after loss, failure, or closure. -`acquired_frames` counts backend updates accepted by MSS, including updates later dropped. `delivered_frames` counts -successful returns and yields. `source_missed_frames` is `None` unless the backend can count misses reliably. +### Timing and statistics — **Deferred** -`MSS.cls_image` is snapshotted when a capture is created. The custom image constructor receives: +`FrameTiming` and `CaptureStatistics` are deferred past the first cut of the source-bound API. They can be added later +without blocking capture creation, `ScreenShot`, and basic `grab` / `frames`. -```python -image_class(data, region, size=actual_size, timing=timing) -``` +### `cls_image` -Custom classes may ignore optional keywords. Changing `session.cls_image` affects future captures, not an existing -capture's result type. +Dropped from the new API. Legacy `MSS.cls_image` may remain on the deprecated path; source-bound capture does not grow +an equivalent. ## Pull and continuous capture ```python -image = capture.grab(timeout=1.0) - -for image in capture.frames( - buffer_count=1, - overflow="drop_oldest", - timeout=None, -): - process(image) +image = capture.grab() +image = capture.grab(region=...) ``` `grab()` returns a newly constructed current image whenever the backend can complete the request. Repeated calls may -contain identical pixels. It does not promise a distinct source presentation or a particular rate. - -`frames()` requires `FRAME_NOTIFICATIONS` and yields only distinct backend-reported output updates. Equal pixels may be -yielded for distinct presentations. MSS does not compare images to infer updates and does not synthesize duplicates. -Calling it without the capability raises `UnsupportedCaptureOperationError`. - -`buffer_count` is a positive integer and counts completed images waiting for delivery, excluding native frame pools and -the image held by the consumer. The supported overflow policies are: - -```text -drop_oldest discard the oldest pending image and queue the newest; default -block stop draining native updates until delivery space is available -``` +contain identical pixels. It does not promise a distinct source presentation, a particular rate, or no-drop delivery. +**No `timeout` on `grab()`.** -Native APIs may coalesce or miss updates while blocked; this is not an MSS-side drop. Pending images discarded when a -generator ends count as dropped. Version one has no target-FPS or duplicate-frame video mode. +### `frames()` — **Open** -`timeout` limits each wait for an image. `FrameTimeoutError` terminates that generator but leaves the capture reusable. -`SourceLostError` and `CaptureFailedError` terminate it and leave the capture terminal. Closing the capture ends an -active generator normally. +Intent: a streaming path that can deliver consecutive backend updates **without dropping frames** when the consumer +keeps up. `timeout` belongs on this path, not on `grab()`. +Would require us to add a `wait_for_next_frame` or something similar to `grab()` +function. -Only one `frames()` generator may be active. It becomes active on its first iteration. While active, `grab()` or -advancing another generator raises `CaptureBusyError`. Concurrent `grab()` calls are serialized. An individual generator -must not be advanced concurrently from multiple threads. - -```python -stream = capture.frames() -try: - for image in stream: - if process_and_finish(image): - break -finally: - stream.close() # Required when retaining a generator and leaving early. -``` - -Closing or exhausting the generator returns the capture to its open state. A timeout permits another generator on the -same capture; source loss requires selecting a source and creating a new capture. +Only one streaming consumer may be active per capture at a time; details TBD with the `frames()` design. Concurrent `grab()` calls are serialized. ## Lifetime and recovery -`Capture` is an idempotent context manager: +`Capture` is an idempotent context manager: `__exit__` closes; `close()` may be called again with no effect. ```python with session.create_capture(source) as capture: @@ -335,13 +302,6 @@ closing shared platform resources. Returned image storage remains valid after bo ```text OPEN -├── first next(frames) ─────────> STREAMING -├── source disappears ──────────> LOST -├── unrecoverable provider error ─> FAILED -└── close() ────────────────────> CLOSED - -STREAMING -├── generator close/timeout ────> OPEN ├── source disappears ──────────> LOST ├── unrecoverable provider error ─> FAILED └── close() ────────────────────> CLOSED @@ -352,14 +312,9 @@ include device reset, DXGI duplication invalidation, frame-pool recreation, wind orientation changes for the same monitor. Recovery stays within the selected provider and preserves required capabilities. -A minimized, hidden, or temporarily unavailable window is not lost. `frames()` waits for another update and may time -out; `grab()` may construct a new result from the latest retained source image. Window destruction and monitor unplug -are terminal source loss. Recreated windows, reconnected monitors, matching titles, matching geometry, and reused list -indices do not silently retarget a capture. `Desktop` persists across monitor-topology changes. - -Pending delivery images are discarded on native-generation replacement, source loss, or terminal failure. Previously -returned images remain valid. Public timeouts include time spent recovering; `timeout=None` uses finite internal waits -so close, loss, and failure remain observable. +A minimized, hidden, or temporarily unavailable window is not lost. Window destruction and monitor unplug are terminal +source loss. Recreated windows, reconnected monitors, matching titles, matching geometry, and reused list indices do +not silently retarget a capture. `Desktop` persists across monitor-topology changes. ## Exceptions @@ -370,7 +325,6 @@ MSSError ├── WindowSelectionError ├── BackendUnavailableError └── CaptureError - ├── FrameTimeoutError capture reusable ├── SourceLostError terminal LOST ├── CaptureFailedError terminal FAILED ├── CaptureClosedError @@ -378,19 +332,42 @@ MSSError └── UnsupportedCaptureOperationError ``` +`MSSError` is the package top-level exception. `ScreenShotError` remains for the legacy path only; the new API is not +rooted at `ScreenShotError`. + Invalid argument types use `TypeError`; invalid values use `ValueError`. Window-selector callback exceptions propagate unchanged. `CaptureFailedError` chains its native cause. ## Compatibility and deferred work The deprecated `MSS.grab()`, `MSS.monitors`, `save()`, and `shot()` retain their current desktop-rectangle semantics and -return types. Their legacy backend is initialized lazily. Constructor-level `backend=` and `with_cursor=` configure only -that path; new capture selection belongs to `create_capture()`. Compatibility helpers do not call deprecated public -methods internally. - -The new API uses `MSSError` and `CaptureError`; `ScreenShotError` remains the legacy error type. - -THis version 1.0 of the new interface deliberately defers GPU result types and more capture-native formats (CPU or GPU). -Internally, capture implementations are generic over their result type and -producer/queue delivery so these improvements can be added later without changes -to the overall design. +return types (including today's macOS nominal-resolution default and packed buffers). Their legacy backend is +initialized lazily. Constructor-level `backend=` and `with_cursor=` configure only that path; new capture selection +belongs to `create_capture()`. Compatibility helpers do not call deprecated public methods internally. + +### Settled for this revision + +- Session + source-bound `Capture`; region on `grab`, not `create_capture` +- `WindowArea` instead of `client_area_only`; regions relative to the chosen area +- `with_cursor: bool | None` for auto backend selection +- `find_window` only (no `get_window`); window equality by identity +- Enumeration snapshots as `tuple[...]` +- Keep `ScreenShot`; allow strides; lazy packed `.bgra` +- No `cls_image` on the new path +- No `timeout` on `grab()` +- Expose `capture.backend` / `capabilities` / `backend_failures` for introspection +- `MSSError` as the new-API exception root + +### Open + +- Physical vs logical pixel space on macOS vs Windows/Linux (shared `ScreenShot` contract) +- Default `WindowArea` for window captures +- A `frames()` generator design +- OS picker / authorization models (WGC, Wayland ScreenCast, ScreenCaptureKit) vs app-selected sources + +### Deferred + +- `FrameTiming` / `CaptureStatistics` +- `ScreenShotCpu` / `ScreenShotGpu` split and GPU result types +- Richer pixel formats / color spaces +- Region insets (negative width/height syntactic sugar) From 84a91d24ba8dae495668c901a84ec9a0354e8a03 Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Thu, 30 Jul 2026 10:04:38 -0800 Subject: [PATCH 03/15] Added picker support and PixelSpace --- docs/source/capture-api-design.md | 122 +++++++++++++++++++++++++----- 1 file changed, 104 insertions(+), 18 deletions(-) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index 94991948..c93ba422 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -7,13 +7,13 @@ orphan: true Status: design proposal for issues [#470](https://github.com/BoboTiG/python-mss/issues/470) and [#544](https://github.com/BoboTiG/python-mss/issues/544). -Decisions below reflect discussion on #544 through 2026-07-22. Sections marked **Open** or **Deferred** are +Decisions below reflect discussion on #544 through 2026-07-30. Sections marked **Open** or **Deferred** are not settled. ## Overview `MSS` is a platform session. It enumerates sources and creates source-bound `Capture` objects. Captures return -`ScreenShot` objects (see [Results](#results)). +`ScreenShot` objects (see the Results section below). ```python with MSS() as session: @@ -52,22 +52,23 @@ session.list_windows() # tuple[Window, ...] Return type is `tuple[...]` so the snapshot cannot be resized or reassigned in place. -#### Coordinate spaces — **Open** +Inputs are always display-oriented: a 90°-rotated 1920×1080 panel is addressed as width 1080, height 1920. Callers do +not apply a rotation transform. -**Inputs** (windows, monitors, desktop, regions) are always in **display-oriented** pixels: a 90°-rotated 1920×1080 -panel is reported and addressed as width 1080, height 1920. Callers do not apply a rotation transform when specifying -what to capture. +```python +class PixelSpace(Enum): + LOGICAL = auto() # points / DIPs / nominal display units + PHYSICAL = auto() # backing-store / framebuffer pixels +``` -**Pixel space (physical / framebuffer vs logical / nominal)** is unresolved. Today Windows and Linux/X11 use -framebuffer pixels (geometry matches the buffer); macOS defaults to nominal/logical via -`kCGWindowImageNominalResolution`. Because the new API keeps `ScreenShot` as the result type shared with the legacy -path, there is no separate type with which to quarantine a new contract. Options under discussion on #544: +Enumerated source geometry uses the session's platform-default pixel space: -1. Keep the platform inconsistency on `ScreenShot` (document it). -2. Unify on physical/framebuffer everywhere in a major release (breaking on Retina Macs that rely on nominal size). -3. Make the space explicit on each result (attribute / scale factor). +```python +session.default_pixel_space # PixelSpace +source.bounds # Region in session.default_pixel_space +``` -Until decided, this document does not claim “physical pixels everywhere.” +The initial API does not provide source-geometry conversion between pixel spaces. ### Monitors @@ -103,6 +104,16 @@ Within the same session, callers who need “same OS window” compare `window.i `include_hidden=True`. Child controls, shell surfaces, menus, tooltips, and similar transient windows are excluded where the platform can identify them reliably. +On platforms that prohibit application-driven enumeration, these APIs raise rather than returning an empty snapshot: + +```python +session.list_windows() # SourceEnumerationUnsupportedError +session.list_monitors() # SourceEnumerationUnsupportedError +session.desktop # SourceEnumerationUnsupportedError +``` + +An empty tuple means enumeration succeeded and found no sources. Portal-only Wayland uses the system-picker path below. + ```python WindowSelector = Callable[[tuple[Window, ...]], Window | None] @@ -141,6 +152,11 @@ class CaptureCapability(Flag): SOURCE_MISSED_FRAME_COUNT = auto() +class PickerTarget(Flag): + MONITOR = auto() + WINDOW = auto() + + class WindowArea(Enum): CLIENT = auto() # client area / content rect / client window FULL = auto() # entire native window, including non-client chrome @@ -155,9 +171,30 @@ def create_capture( required_capabilities: CaptureCapability = CaptureCapability.NONE, with_cursor: bool | None = None, area: WindowArea | None = None, + pixel_space: PixelSpace | None = None, ) -> Capture: ... ``` +`pixel_space=None` resolves to a documented platform default: + +```text +Windows PHYSICAL +Linux/X11 PHYSICAL +Linux/Wayland PHYSICAL +macOS LOGICAL +``` + +`None` does not let the backend choose. The resolved value is stable and inspectable: + +```python +capture.pixel_space # PixelSpace; never None +capture.source_bounds # Region in capture.pixel_space +session.default_pixel_space +``` + +Portable applications select a space explicitly. Backend selection rejects providers that cannot produce the requested +space; it never substitutes the other one. + `area` is valid only with a `Window` source. Invalid for `Monitor` / `Desktop`. Region coordinates on later `grab` calls are relative to the chosen extent: with `CLIENT`, `(0, 0)` is the client/content origin; with `FULL`, `(0, 0)` is the full native window origin. @@ -177,8 +214,47 @@ None don't care (default); auto may pick the best otherwise-eligible backend; When cursor inclusion is required (`True`), cursor pixels are composited only where they intersect the final clipped output, and cursor movement, shape changes, and visibility changes count as output updates. -Configuration is separate from capability flags: source type, cursor preference, window area, and regions are -requirements already expressed by the capture request. +Configuration is separate from capability flags: source type, cursor preference, window area, pixel space, and regions +are requirements already expressed by the capture request. + +### System-picker creation + +Application-selected sources use `create_capture()`. User-selected surfaces use an asynchronous system picker and are +bound directly to the returned capture: + +```python +async def create_capture_from_picker( + self, + *, + allowed_to_pick: PickerTarget = PickerTarget.MONITOR | PickerTarget.WINDOW, + backend: str = "auto", + required_capabilities: CaptureCapability = CaptureCapability.NONE, + with_cursor: bool | None = None, + area: WindowArea | None = None, + pixel_space: PixelSpace | None = None, + parent_window: object | None = None, +) -> Capture | None: ... +``` + +```python +capture = await session.create_capture_from_picker( + allowed_to_pick=PickerTarget.WINDOW, + with_cursor=True, + pixel_space=PixelSpace.PHYSICAL, +) +if capture is None: + return # User cancelled. +``` + +`allowed_to_pick` controls the categories offered by the picker; one surface is selected. The selected item, portal +session, authorization, and stream setup are not exposed as a public `CaptureSource`. `area` applies only if the user +selects a window. `parent_window` is the platform-specific parent handle for the picker. Automatic backend fallback may +occur before UI is presented, but MSS presents at most one picker and does not reprompt after selection if capture +initialization fails. + +This path is optional on platforms with application-driven selection and required by portal-only Wayland. Although +frame delivery remains synchronous in the first version, picker creation is async because WGC, ScreenCaptureKit, and +the Wayland portal all complete selection asynchronously. ### Regions @@ -190,6 +266,7 @@ capture.grab(region=Region(left=10, top=20, width=640, height=480)) ``` `None` captures the full source extent (subject to `area` for windows). Region may differ across `grab` calls. +Coordinates and clipping use `capture.pixel_space`. Region fields are integers. Negative `left` and `top` values are valid. Negative width or height raises `ValueError`; zero is valid. The effective rectangle is recomputed for each acquisition: @@ -241,6 +318,14 @@ correctness, security, or platform compatibility. The result type remains **`ScreenShot`**. It is not replaced by a separate `Frame` type. +```python +image.pixel_space == capture.pixel_space +``` + +`image.pos`, `image.size`, buffer dimensions, row stride, and array/tensor shapes use that space. Each result contains +one buffer in one space, never logical and physical copies. `capture.source_bounds` gives the captured source extent in +the capture's resolved pixel space. + Future direction (not v1): `ScreenShot` as a base with `ScreenShotCpu` and `ScreenShotGpu` subclasses sharing common attributes; CPU and GPU results expose different buffers. @@ -322,6 +407,7 @@ not silently retarget a capture. `Desktop` persists across monitor-topology chan MSSError ├── ScreenShotError legacy API ├── SessionClosedError +├── SourceEnumerationUnsupportedError ├── WindowSelectionError ├── BackendUnavailableError └── CaptureError @@ -348,6 +434,8 @@ belongs to `create_capture()`. Compatibility helpers do not call deprecated publ ### Settled for this revision - Session + source-bound `Capture`; region on `grab`, not `create_capture` +- Explicit logical/physical capture space with platform-dependent `None` default +- Separate `create_capture_from_picker`; no public intermediate picked-source object - `WindowArea` instead of `client_area_only`; regions relative to the chosen area - `with_cursor: bool | None` for auto backend selection - `find_window` only (no `get_window`); window equality by identity @@ -360,10 +448,8 @@ belongs to `create_capture()`. Compatibility helpers do not call deprecated publ ### Open -- Physical vs logical pixel space on macOS vs Windows/Linux (shared `ScreenShot` contract) - Default `WindowArea` for window captures - A `frames()` generator design -- OS picker / authorization models (WGC, Wayland ScreenCast, ScreenCaptureKit) vs app-selected sources ### Deferred From ca835e980977bd71013d74956b435bf4edb608f6 Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Fri, 31 Jul 2026 12:14:11 +0200 Subject: [PATCH 04/15] Updates to pixel_space and picker behavior --- docs/source/capture-api-design.md | 95 ++++++++++++++----------------- 1 file changed, 44 insertions(+), 51 deletions(-) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index c93ba422..305465cb 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -61,14 +61,29 @@ class PixelSpace(Enum): PHYSICAL = auto() # backing-store / framebuffer pixels ``` -Enumerated source geometry uses the session's platform-default pixel space: +Pixel space is not selected by the caller. It is determined by the platform and any process or +thread DPI configuration established by the application. MSS does not change process DPI awareness implicitly, +but it does offer a utility function on Windows, for the user's convenience. + +Enumerated source geometry uses the session's effective pixel space: ```python -session.default_pixel_space # PixelSpace -source.bounds # Region in session.default_pixel_space +session.pixel_space # PixelSpace +source.bounds # Region in session.pixel_space +``` +The session therefore informs the user of the pixel_space being used. This should be fixed for the +lifetime of an MSS session. Applications must not change their pixel space while the session or its captures are active. MSS may detect such a change and raise an exception. + +Typical behavior is: + +```text +Windows Determined by the application's DPI-awareness context +Linux/X11 PHYSICAL +Linux/Wayland PHYSICAL capture-buffer pixels +macOS LOGICAL ``` -The initial API does not provide source-geometry conversion between pixel spaces. +The initial API does not provide conversion between pixel spaces and does not resample to produce another space. ### Monitors @@ -152,11 +167,6 @@ class CaptureCapability(Flag): SOURCE_MISSED_FRAME_COUNT = auto() -class PickerTarget(Flag): - MONITOR = auto() - WINDOW = auto() - - class WindowArea(Enum): CLIENT = auto() # client area / content rect / client window FULL = auto() # entire native window, including non-client chrome @@ -171,29 +181,11 @@ def create_capture( required_capabilities: CaptureCapability = CaptureCapability.NONE, with_cursor: bool | None = None, area: WindowArea | None = None, - pixel_space: PixelSpace | None = None, ) -> Capture: ... ``` -`pixel_space=None` resolves to a documented platform default: - -```text -Windows PHYSICAL -Linux/X11 PHYSICAL -Linux/Wayland PHYSICAL -macOS LOGICAL -``` - -`None` does not let the backend choose. The resolved value is stable and inspectable: - -```python -capture.pixel_space # PixelSpace; never None -capture.source_bounds # Region in capture.pixel_space -session.default_pixel_space -``` - -Portable applications select a space explicitly. Backend selection rejects providers that cannot produce the requested -space; it never substitutes the other one. +The caller does not choose logical or physical coordinates. The resolved space is inspectable through +`session.pixel_space` and remains stable for the lifetime of the session and therefore the capture. `area` is valid only with a `Window` source. Invalid for `Monitor` / `Desktop`. Region coordinates on later `grab` calls are relative to the chosen extent: with `CLIENT`, `(0, 0)` is the client/content origin; with @@ -214,47 +206,47 @@ None don't care (default); auto may pick the best otherwise-eligible backend; When cursor inclusion is required (`True`), cursor pixels are composited only where they intersect the final clipped output, and cursor movement, shape changes, and visibility changes count as output updates. -Configuration is separate from capability flags: source type, cursor preference, window area, pixel space, and regions -are requirements already expressed by the capture request. +Configuration is separate from capability flags: source type, cursor preference, window area, and regions are +requirements already expressed by the capture request. Pixel space is an observed property, not a capture request. ### System-picker creation -Application-selected sources use `create_capture()`. User-selected surfaces use an asynchronous system picker and are -bound directly to the returned capture: +Application-selected sources use `create_capture()`. User-selected surfaces use a system picker and are bound directly +to the returned capture: ```python -async def create_capture_from_picker( +def create_capture_from_picker( self, *, - allowed_to_pick: PickerTarget = PickerTarget.MONITOR | PickerTarget.WINDOW, backend: str = "auto", required_capabilities: CaptureCapability = CaptureCapability.NONE, with_cursor: bool | None = None, area: WindowArea | None = None, - pixel_space: PixelSpace | None = None, parent_window: object | None = None, ) -> Capture | None: ... ``` ```python -capture = await session.create_capture_from_picker( - allowed_to_pick=PickerTarget.WINDOW, +capture = session.create_capture_from_picker( with_cursor=True, - pixel_space=PixelSpace.PHYSICAL, ) if capture is None: return # User cancelled. ``` -`allowed_to_pick` controls the categories offered by the picker; one surface is selected. The selected item, portal -session, authorization, and stream setup are not exposed as a public `CaptureSource`. `area` applies only if the user -selects a window. `parent_window` is the platform-specific parent handle for the picker. Automatic backend fallback may -occur before UI is presented, but MSS presents at most one picker and does not reprompt after selection if capture -initialization fails. +The method blocks while the system picker is open and returns when the user selects a source, cancels, or picker +creation fails. It has no timeout. + +The operating-system picker determines which source categories are offered and one surface is selected. The selected +item and associated resources are not exposed as a public `CaptureSource`. `area` applies only +if the user selects a window. `parent_window` is the platform-specific parent handle for the picker. Automatic backend +fallback may occur before UI is presented, but MSS presents at most one picker and does not reprompt after selection if +capture initialization fails. -This path is optional on platforms with application-driven selection and required by portal-only Wayland. Although -frame delivery remains synchronous in the first version, picker creation is async because WGC, ScreenCaptureKit, and -the Wayland portal all complete selection asynchronously. +This path is required by portal-only Wayland. In the future we may also offer it for Windows WGC and MacOS +ScreenCaptureKit. Although WGC, +ScreenCaptureKit, and the Wayland portal complete selection asynchronously at the platform level, the initial public API +is synchronous. A `create_capture_from_picker_async()` variant may be added later without changing the synchronous API. ### Regions @@ -428,14 +420,15 @@ unchanged. `CaptureFailedError` chains its native cause. The deprecated `MSS.grab()`, `MSS.monitors`, `save()`, and `shot()` retain their current desktop-rectangle semantics and return types (including today's macOS nominal-resolution default and packed buffers). Their legacy backend is -initialized lazily. Constructor-level `backend=` and `with_cursor=` configure only that path; new capture selection -belongs to `create_capture()`. Compatibility helpers do not call deprecated public methods internally. +initialized lazily. Constructor-level `backend=` and `with_cursor=` configure only that path; new source-bound +capture creation belongs to `create_capture()` and `create_capture_from_picker()`. Compatibility helpers do not call +deprecated public methods internally. ### Settled for this revision - Session + source-bound `Capture`; region on `grab`, not `create_capture` -- Explicit logical/physical capture space with platform-dependent `None` default -- Separate `create_capture_from_picker`; no public intermediate picked-source object +- Pixel space is platform/process determined and inspectable, not caller-selectable +- Synchronous `create_capture_from_picker`; no target-category filter or public intermediate picked-source object - `WindowArea` instead of `client_area_only`; regions relative to the chosen area - `with_cursor: bool | None` for auto backend selection - `find_window` only (no `get_window`); window equality by identity From 3aa27220e3730c716cd35c9120d714415ed33c55 Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Fri, 31 Jul 2026 12:25:20 +0200 Subject: [PATCH 05/15] Sharpend legacy vs new path --- docs/source/capture-api-design.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index 305465cb..cd434e03 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -399,6 +399,7 @@ not silently retarget a capture. `Desktop` persists across monitor-topology chan MSSError ├── ScreenShotError legacy API ├── SessionClosedError +├── SessionModeError legacy/source-bound API paths mixed ├── SourceEnumerationUnsupportedError ├── WindowSelectionError ├── BackendUnavailableError @@ -419,10 +420,20 @@ unchanged. `CaptureFailedError` chains its native cause. ## Compatibility and deferred work The deprecated `MSS.grab()`, `MSS.monitors`, `save()`, and `shot()` retain their current desktop-rectangle semantics and -return types (including today's macOS nominal-resolution default and packed buffers). Their legacy backend is -initialized lazily. Constructor-level `backend=` and `with_cursor=` configure only that path; new source-bound -capture creation belongs to `create_capture()` and `create_capture_from_picker()`. Compatibility helpers do not call -deprecated public methods internally. +return types (including today's macOS nominal-resolution default and packed buffers). The legacy and new paths +must not be mixed on one `MSS` session. A session is initially uncommitted; its first legacy or new operation +commits it to that path for its lifetime. Using an API from the other path afterward raises `SessionModeError`. + +The legacy backend is initialized lazily by the first legacy operation. Constructor-level `backend=` and +`with_cursor=` configure only that path; they do not initialize it. New source enumeration and capture creation belong +to the source-bound (new) path, including `desktop`, `list_monitors()`, `list_windows()`, `find_window()`, `create_capture()`, +and `create_capture_from_picker()`. Compatibility helpers do not call deprecated public methods internally. + +On Windows, only initialization of the legacy GDI path attempts to establish the process DPI awareness required by its +existing physical-desktop coordinate contract. Source-bound session creation and use never change process or thread DPI +awareness implicitly. We will improve the legacy GDI initialization so it validates the resulting awareness and raises `ScreenShotError` if an +incompatible value was already established by the application manifest or by other code in the process, or if Windows +otherwise refuses the requested configuration. An already-established compatible value is accepted. ### Settled for this revision @@ -438,6 +449,7 @@ deprecated public methods internally. - No `timeout` on `grab()` - Expose `capture.backend` / `capabilities` / `backend_failures` for introspection - `MSSError` as the new-API exception root +- Legacy and source-bound operations cannot be mixed in one session; legacy initialization remains lazy ### Open From 935d94f314ca81f5382337ae35e4ee7e0bd4816e Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Fri, 31 Jul 2026 12:34:48 +0200 Subject: [PATCH 06/15] Improve picker design --- docs/source/capture-api-design.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index cd434e03..50fa2a6e 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -234,14 +234,23 @@ if capture is None: return # User cancelled. ``` -The method blocks while the system picker is open and returns when the user selects a source, cancels, or picker -creation fails. It has no timeout. +The method blocks while the system picker is open and returns when the user +selects a source or the platform reports that the user cancelled. `None` means +cancellation only. Failure to create or operate the picker, loss of the portal +or native service, permission failure, an invalid platform response, and failure +to initialize the selected capture raise an exception. It has no timeout. + +If no eligible picker backend can be initialized before UI is presented, +`BackendUnavailableError` records the attempted providers in the same way as +`create_capture()`. After a picker backend has presented UI, an abnormal picker +exit or failure to initialize the selected source raises `PickerError` and +chains the native cause where available. Automatic backend fallback does not +occur after UI has been presented. The operating-system picker determines which source categories are offered and one surface is selected. The selected item and associated resources are not exposed as a public `CaptureSource`. `area` applies only -if the user selects a window. `parent_window` is the platform-specific parent handle for the picker. Automatic backend -fallback may occur before UI is presented, but MSS presents at most one picker and does not reprompt after selection if -capture initialization fails. +if the user selects a window. `parent_window` is the platform-specific parent handle for the picker. MSS presents at +most one picker and does not reprompt after selection if capture initialization fails. This path is required by portal-only Wayland. In the future we may also offer it for Windows WGC and MacOS ScreenCaptureKit. Although WGC, @@ -403,6 +412,7 @@ MSSError ├── SourceEnumerationUnsupportedError ├── WindowSelectionError ├── BackendUnavailableError +├── PickerError abnormal picker exit or selected-source initialization failure └── CaptureError ├── SourceLostError terminal LOST ├── CaptureFailedError terminal FAILED @@ -415,7 +425,7 @@ MSSError rooted at `ScreenShotError`. Invalid argument types use `TypeError`; invalid values use `ValueError`. Window-selector callback exceptions propagate -unchanged. `CaptureFailedError` chains its native cause. +unchanged. `PickerError` and `CaptureFailedError` chain their native causes. ## Compatibility and deferred work @@ -440,6 +450,7 @@ otherwise refuses the requested configuration. An already-established compatible - Session + source-bound `Capture`; region on `grab`, not `create_capture` - Pixel space is platform/process determined and inspectable, not caller-selectable - Synchronous `create_capture_from_picker`; no target-category filter or public intermediate picked-source object +- Picker cancellation returns `None`; every abnormal picker exit raises an exception - `WindowArea` instead of `client_area_only`; regions relative to the chosen area - `with_cursor: bool | None` for auto backend selection - `find_window` only (no `get_window`); window equality by identity From b060c7caff5ed6755f78396dad026c55e5198d26 Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Fri, 31 Jul 2026 13:49:37 +0200 Subject: [PATCH 07/15] Refine language for picker --- docs/source/capture-api-design.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index 50fa2a6e..e9e11636 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -247,10 +247,13 @@ exit or failure to initialize the selected source raises `PickerError` and chains the native cause where available. Automatic backend fallback does not occur after UI has been presented. -The operating-system picker determines which source categories are offered and one surface is selected. The selected -item and associated resources are not exposed as a public `CaptureSource`. `area` applies only -if the user selects a window. `parent_window` is the platform-specific parent handle for the picker. MSS presents at -most one picker and does not reprompt after selection if capture initialization fails. +MSS requests monitor and window sources where supported. The platform picker +determines how those choices are presented and may offer a narrower set; MSS does +not request virtual-display creation. One surface is selected. The selected item +and associated resources are not exposed as a public `CaptureSource`. `area` +applies only if the user selects a window. `parent_window` is the platform-specific +parent handle for the picker. MSS presents at most one picker and does not reprompt +after selection if capture initialization fails. This path is required by portal-only Wayland. In the future we may also offer it for Windows WGC and MacOS ScreenCaptureKit. Although WGC, From 7350ec7eb4db3d97846d894b825c157a143e04c4 Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Sat, 1 Aug 2026 15:47:13 +0200 Subject: [PATCH 08/15] New version using ponytail skill for review --- .agents/skills/ponytail/SKILL.md | 120 ++++++++++++++++ docs/source/capture-api-design.md | 226 +++++++++++++++++++----------- 2 files changed, 261 insertions(+), 85 deletions(-) create mode 100644 .agents/skills/ponytail/SKILL.md diff --git a/.agents/skills/ponytail/SKILL.md b/.agents/skills/ponytail/SKILL.md new file mode 100644 index 00000000..02c0712c --- /dev/null +++ b/.agents/skills/ponytail/SKILL.md @@ -0,0 +1,120 @@ +--- +name: ponytail +description: > + Forces the laziest solution that actually works, simplest, shortest, most + minimal. Channels a senior dev who has seen everything: question whether the + task needs to exist at all (YAGNI), reach for the standard library before + custom code, native platform features before dependencies, one line before + fifty. Supports intensity levels: lite, full (default), ultra. Use on ANY + coding task: writing, adding, refactoring, fixing, reviewing, or designing + code, and choosing libraries or dependencies. Also use whenever the user + says "ponytail", "be lazy", "lazy mode", "simplest solution", "minimal + solution", "yagni", "do less", or "shortest path", or complains about + over-engineering, bloat, boilerplate, or unnecessary dependencies. Do NOT + use for non-coding requests (general knowledge, prose, translation, + summaries, recipes). +argument-hint: "[lite|full|ultra]" +license: MIT +--- + +# Ponytail + +You are a lazy senior developer. Lazy means efficient, not careless. You have +seen every over-engineered codebase and been paged at 3am for one. The best +code is the code never written. + +## Persistence + +ACTIVE EVERY RESPONSE. No drift back to over-building. Still active if +unsure. Off only: "stop ponytail" / "normal mode". Default: **full**. +Switch: `/ponytail lite|full|ultra`. + +## The ladder + +Stop at the first rung that holds: + +1. **Does this need to exist at all?** Speculative need = skip it, say so in one line. (YAGNI) +2. **Already in this codebase?** A helper, util, type, or pattern that already lives here → reuse it. Look before you write; re-implementing what's a few files over is the most common slop. +3. **Stdlib does it?** Use it. +4. **Native platform feature covers it?** `` over a picker lib, CSS over JS, DB constraint over app code. +5. **Already-installed dependency solves it?** Use it. Never add a new one for what a few lines can do. +6. **Can it be one line?** One line. +7. **Only then:** the minimum code that works. + +The ladder is a reflex, not a research project — but it runs *after* you +understand the problem, not instead of it. Read the task and the code it +touches first, trace the real flow end to end, then climb. Two rungs work → +take the higher one and move on. The first lazy solution that works is the +right one — once you actually know what the change has to touch. + +**Bug fix = root cause, not symptom.** A report names a symptom. Before you +edit, grep every caller of the function you're about to touch. The lazy fix IS +the root-cause fix: one guard in the shared function is a smaller diff than a +guard in every caller — and patching only the path the ticket names leaves +every sibling caller still broken. Fix it once, where all callers route through. + +## Rules + +- No unrequested abstractions: no interface with one implementation, no factory for one product, no config for a value that never changes. +- No boilerplate, no scaffolding "for later", later can scaffold for itself. +- Deletion over addition. Boring over clever, clever is what someone decodes at 3am. +- Fewest files possible. Shortest working diff wins — but only once you understand the problem. The smallest change in the wrong place isn't lazy, it's a second bug. +- Complex request? Ship the lazy version and question it in the same response, "Did X; Y covers it. Need full X? Say so." Never stall on an answer you can default. +- Two stdlib options, same size? Take the one that's correct on edge cases. Lazy means writing less code, not picking the flimsier algorithm. +- Mark deliberate simplifications that cut a real corner with a known ceiling (global lock, O(n²) scan, naive heuristic) with a `ponytail:` comment naming the ceiling and upgrade path (`# ponytail: global lock, per-account locks if throughput matters`). + +## Output + +Code first. Then at most three short lines: what was skipped, when to add it. +No essays, no feature tours, no design notes. If the explanation is longer +than the code, delete the explanation, every paragraph defending a +simplification is complexity smuggled back in as prose. Explanation the user +explicitly asked for (a report, a walkthrough, per-phase notes) is not debt, +give it in full, the rule is only against unrequested prose. + +Pattern: `[code] → skipped: [X], add when [Y].` + +## Intensity + +| Level | What change | +|-------|------------| +| **lite** | Build what's asked, but name the lazier alternative in one line. User picks. | +| **full** | The ladder enforced. Stdlib and native first. Shortest diff, shortest explanation. Default. | +| **ultra** | YAGNI extremist. Deletion before addition. Ship the one-liner and challenge the rest of the requirement in the same breath. | + +Example: "Add a cache for these API responses." +- lite: "Done, cache added. FYI: `functools.lru_cache` covers this in one line if you'd rather not own a cache class." +- full: "`@lru_cache(maxsize=1000)` on the fetch function. Skipped custom cache class, add when lru_cache measurably falls short." +- ultra: "No cache until a profiler says so. When it does: `@lru_cache`. A hand-rolled TTL cache class is a bug farm with a hit rate." + +## When NOT to be lazy + +Never simplify away: input validation at trust boundaries, error handling +that prevents data loss, security measures, accessibility basics, anything +explicitly requested. User insists on the full version → build it, no +re-arguing. + +Never lazy about understanding the problem. The ladder shortens the +solution, never the reading. Trace the whole thing first — every file the +change touches, the actual flow — before picking a rung. Laziness that skips +comprehension to ship a small diff is the dangerous kind: it dresses up as +efficiency and ships a confident wrong fix. Read fully, then be lazy. + +Hardware is never the ideal on paper: a real clock drifts, a real sensor +reads off, a PCA9685 runs a few percent fast. Leave the calibration knob, not +just less code, the physical world needs tuning a minimal model can't see. + +Lazy code without its check is unfinished. Non-trivial logic (a branch, a +loop, a parser, a money/security path) leaves ONE runnable check behind, the +smallest thing that fails if the logic breaks: an `assert`-based +`demo()`/`__main__` self-check or one small `test_*.py`. No frameworks, no +fixtures, no per-function suites unless asked. Trivial one-liners need no +test, YAGNI applies to tests too. + +## Boundaries + +Ponytail governs what you build, not how you talk (pair with Caveman for +terse prose). "stop ponytail" / "normal mode": revert. Level persists until +changed or session end. + +The shortest path to done is the right path. diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index e9e11636..14cd8953 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -7,8 +7,7 @@ orphan: true Status: design proposal for issues [#470](https://github.com/BoboTiG/python-mss/issues/470) and [#544](https://github.com/BoboTiG/python-mss/issues/544). -Decisions below reflect discussion on #544 through 2026-07-30. Sections marked **Open** or **Deferred** are -not settled. +Decisions below reflect discussion on #544 through 2026-08-01. Sections marked **Deferred** are not part of v1. ## Overview @@ -52,6 +51,13 @@ session.list_windows() # tuple[Window, ...] Return type is `tuple[...]` so the snapshot cannot be resized or reassigned in place. +Sources returned by a session carry private provenance. `create_capture()` +accepts only a source returned by that same session. A manually constructed +`Monitor` may still be useful as geometry, and legacy `MSS.grab()` continues to +accept monitor dictionaries, but neither is a valid source for the new API. +Passing a foreign or manually constructed source to `create_capture()` raises +`ValueError`. + Inputs are always display-oriented: a 90°-rotated 1920×1080 panel is addressed as width 1080, height 1920. Callers do not apply a rotation transform. @@ -87,7 +93,11 @@ The initial API does not provide conversion between pixel spaces and does not re ### Monitors -`Monitor` supports attributes. It may retain string-key access temporarily for migration. +`Monitor` is introduced first as a focused change for #470. It is a frozen, +slotted dataclass with required geometry and optional standard metadata such as +`is_primary`, `name`, `unique_id`, and Linux `output`. It supports attributes +and retains string-key access temporarily for migration, but does not promise +the complete `Mapping` interface. ```python monitor.width @@ -160,13 +170,6 @@ window remains lost even if another window later has the same title, PID, class, ## Capture creation ```python -class CaptureCapability(Flag): - NONE = 0 - FRAME_NOTIFICATIONS = auto() - PRESENTATION_TIMESTAMPS = auto() - SOURCE_MISSED_FRAME_COUNT = auto() - - class WindowArea(Enum): CLIENT = auto() # client area / content rect / client window FULL = auto() # entire native window, including non-client chrome @@ -178,7 +181,6 @@ def create_capture( source: CaptureSource, *, backend: str = "auto", - required_capabilities: CaptureCapability = CaptureCapability.NONE, with_cursor: bool | None = None, area: WindowArea | None = None, ) -> Capture: ... @@ -187,12 +189,13 @@ def create_capture( The caller does not choose logical or physical coordinates. The resolved space is inspectable through `session.pixel_space` and remains stable for the lifetime of the session and therefore the capture. -`area` is valid only with a `Window` source. Invalid for `Monitor` / `Desktop`. Region coordinates on later -`grab` calls are relative to the chosen extent: with `CLIENT`, `(0, 0)` is the client/content origin; with -`FULL`, `(0, 0)` is the full native window origin. +`area` is valid only with a `Window` source. It must be `None` for `Monitor` and `Desktop`. For a `Window`, `None` +resolves to `WindowArea.CLIENT`, the v1 default. Region coordinates on later `grab` calls are relative to the chosen +extent: with `CLIENT`, `(0, 0)` is the client/content origin; with `FULL`, `(0, 0)` is the full native window origin. -**Open:** default for `area` when capturing a window (`CLIENT` vs `FULL`). Lean `CLIENT` unless discussion settles -otherwise. +`FULL` means the native frame rectangle, including title bars, borders, menus, and other non-client chrome. It excludes +compositor effects outside that rectangle, such as drop shadows, glow, capture-selection borders, and other external +decoration. A backend is eligible only if it can honor the requested extent. `with_cursor` is tri-state for backend selection: @@ -204,10 +207,11 @@ None don't care (default); auto may pick the best otherwise-eligible backend; ``` When cursor inclusion is required (`True`), cursor pixels are composited only where they intersect the final clipped -output, and cursor movement, shape changes, and visibility changes count as output updates. +output. -Configuration is separate from capability flags: source type, cursor preference, window area, and regions are -requirements already expressed by the capture request. Pixel space is an observed property, not a capture request. +Source type, cursor preference, and window area are requirements expressed by the capture request. Pixel space is an +observed property, not a capture request. Streaming capabilities are deferred with `frames()` and are not part of the +v1 capture-creation API. ### System-picker creation @@ -219,9 +223,8 @@ def create_capture_from_picker( self, *, backend: str = "auto", - required_capabilities: CaptureCapability = CaptureCapability.NONE, with_cursor: bool | None = None, - area: WindowArea | None = None, + window_area: WindowArea = WindowArea.CLIENT, parent_window: object | None = None, ) -> Capture | None: ... ``` @@ -241,7 +244,7 @@ or native service, permission failure, an invalid platform response, and failure to initialize the selected capture raise an exception. It has no timeout. If no eligible picker backend can be initialized before UI is presented, -`BackendUnavailableError` records the attempted providers in the same way as +`BackendUnavailableError` reports attempted-provider context in the same way as `create_capture()`. After a picker backend has presented UI, an abnormal picker exit or failure to initialize the selected source raises `PickerError` and chains the native cause where available. Automatic backend fallback does not @@ -250,12 +253,14 @@ occur after UI has been presented. MSS requests monitor and window sources where supported. The platform picker determines how those choices are presented and may offer a narrower set; MSS does not request virtual-display creation. One surface is selected. The selected item -and associated resources are not exposed as a public `CaptureSource`. `area` -applies only if the user selects a window. `parent_window` is the platform-specific -parent handle for the picker. MSS presents at most one picker and does not reprompt -after selection if capture initialization fails. - -This path is required by portal-only Wayland. In the future we may also offer it for Windows WGC and MacOS +and associated resources are not exposed as a public `CaptureSource`. +`window_area` applies only if the user selects a window and is irrelevant when a +monitor is selected. A picker backend is eligible only if it can honor +`window_area` whenever it offers window selection. `parent_window` is the +platform-specific parent handle for the picker. MSS presents at most one picker +and does not reprompt after selection if capture initialization fails. + +This path is required by portal-only Wayland. In the future we may also offer it for Windows WGC and macOS ScreenCaptureKit. Although WGC, ScreenCaptureKit, and the Wayland portal complete selection asynchronously at the platform level, the initial public API is synchronous. A `create_capture_from_picker_async()` variant may be added later without changing the synchronous API. @@ -272,51 +277,38 @@ capture.grab(region=Region(left=10, top=20, width=640, height=480)) `None` captures the full source extent (subject to `area` for windows). Region may differ across `grab` calls. Coordinates and clipping use `capture.pixel_space`. -Region fields are integers. Negative `left` and `top` values are valid. Negative width or height raises `ValueError`; -zero is valid. The effective rectangle is recomputed for each acquisition: +Region fields are integers. Negative `left` and `top` values are valid. Width and height must be positive; zero or a +negative value raises `ValueError`. The effective rectangle is recomputed for each acquisition against the +capture-local extent: ```python +source_extent = Region(left=0, top=0, width=source_width, height=source_height) effective = requested_region.intersection(source_extent) ``` -The returned image describes the clipped rectangle. A completely clipped region returns an empty `ScreenShot` without a -native pixel copy. Continuous capture still observes source updates because a resized source may make the region -nonempty later. For an empty intersection, the position is the requested origin clamped to the nearest source boundary. +The returned image describes the clipped rectangle. A region with an empty intersection raises `ValueError`. If the +source itself currently has an empty extent, `grab()` raises `CaptureError`. V1 does not create zero-sized `ScreenShot` +objects. -Insets / negative width-height as a crop-from-edges sugar remain deferred. +Every v1 CPU backend must support a different valid region on each `grab()` call, either through native cropping or by +cropping inside MSS. Auto-selection never chooses a backend that rejects this normal `Capture` operation. -If a backend cannot re-crop without reinit, `create_capture` still succeeds; the first incompatible `grab` -request raises `UnsupportedCaptureOperationError`. Prefer backends that can re-crop. +Insets / negative width-height as a crop-from-edges sugar remain deferred. ### Backend selection ```python -@dataclass(frozen=True, slots=True) -class Backend: - name: str - capabilities: CaptureCapability - - -class BackendFailure(NamedTuple): - name: str - reason: str -``` - -```python -capture.backend.name -capture.backend.capabilities # introspection / debugging after auto-select -capture.backend_failures # tuple[BackendFailure, ...] +capture.backend # "xshmgetimage", "gdi", ... ``` -For `backend="auto"`, MSS filters a documented platform/source priority list by the capture configuration and required -capabilities, then tries eligible providers in order. `backend_failures` records each higher-priority provider considered -before the successful one and why it could not be used. If none succeeds, `BackendUnavailableError.failures` contains -the same records. +For `backend="auto"`, MSS filters the platform providers by source and capture configuration, then tries eligible +providers in implementation-defined order. If none succeeds, `BackendUnavailableError` reports useful attempted-provider +context in its message without making a structured failure history part of the public API. An explicit backend never falls back. Automatic fallback occurs only during capture creation; recovery never changes a -capture's provider. Priority does not change in patch releases. Minor releases append new providers behind established -eligible defaults; established defaults may be reordered in a major release. Providers may be disabled sooner for -correctness, security, or platform compatibility. +capture's provider. Auto-selection order is an implementation detail and may improve between releases. Users who need a +specific provider select it explicitly. Providers may be changed or disabled at any time for correctness, security, or +platform compatibility. ## Results @@ -330,6 +322,11 @@ image.pixel_space == capture.pixel_space one buffer in one space, never logical and physical copies. `capture.source_bounds` gives the captured source extent in the capture's resolved pixel space. +`source.bounds` and `capture.source_bounds` use session-global desktop coordinates. A region passed to `grab()` is +capture-local. `image.pos` remains session-global: it is the origin of `capture.source_bounds` plus the origin of the +effective clipped region. For `WindowArea.CLIENT`, `capture.source_bounds` describes the global client/content rectangle; +for `WindowArea.FULL`, it describes the global native frame rectangle. + Future direction (not v1): `ScreenShot` as a base with `ScreenShotCpu` and `ScreenShotGpu` subclasses sharing common attributes; CPU and GPU results expose different buffers. @@ -344,10 +341,10 @@ described above. Exact pixel-format negotiation (HDR, YUV, etc.) is deferred with GPU work. -### Timing and statistics — **Deferred** +### Timing, statistics, and capabilities — **Deferred** -`FrameTiming` and `CaptureStatistics` are deferred past the first cut of the source-bound API. They can be added later -without blocking capture creation, `ScreenShot`, and basic `grab` / `frames`. +`CaptureCapability`, `FrameTiming`, and `CaptureStatistics` are deferred past the first cut of the source-bound API. +They can be designed with the first operation that consumes them rather than becoming speculative v1 public surface. ### `cls_image` @@ -365,14 +362,10 @@ image = capture.grab(region=...) contain identical pixels. It does not promise a distinct source presentation, a particular rate, or no-drop delivery. **No `timeout` on `grab()`.** -### `frames()` — **Open** - -Intent: a streaming path that can deliver consecutive backend updates **without dropping frames** when the consumer -keeps up. `timeout` belongs on this path, not on `grab()`. -Would require us to add a `wait_for_next_frame` or something similar to `grab()` -function. +### `frames()` — **Deferred** -Only one streaming consumer may be active per capture at a time; details TBD with the `frames()` design. Concurrent `grab()` calls are serialized. +The streaming API, buffering, timeouts, update notifications, and concurrency rules will be designed together in a +separate change. Adding `frames()` later does not require changing source-bound capture creation or `grab()`. ## Lifetime and recovery @@ -418,24 +411,24 @@ MSSError ├── PickerError abnormal picker exit or selected-source initialization failure └── CaptureError ├── SourceLostError terminal LOST - ├── CaptureFailedError terminal FAILED - ├── CaptureClosedError - ├── CaptureBusyError - └── UnsupportedCaptureOperationError + └── CaptureClosedError ``` `MSSError` is the package top-level exception. `ScreenShotError` remains for the legacy path only; the new API is not rooted at `ScreenShotError`. Invalid argument types use `TypeError`; invalid values use `ValueError`. Window-selector callback exceptions propagate -unchanged. `PickerError` and `CaptureFailedError` chain their native causes. +unchanged. `PickerError` and `CaptureError` chain their native causes where available. ## Compatibility and deferred work -The deprecated `MSS.grab()`, `MSS.monitors`, `save()`, and `shot()` retain their current desktop-rectangle semantics and -return types (including today's macOS nominal-resolution default and packed buffers). The legacy and new paths -must not be mixed on one `MSS` session. A session is initially uncommitted; its first legacy or new operation -commits it to that path for its lifetime. Using an API from the other path afterward raises `SessionModeError`. +The deprecated `MSS.grab()`, `MSS.monitors`, `save()`, and `shot()` retain their current desktop-rectangle semantics +(including today's macOS nominal-resolution default and packed `ScreenShot` buffers). `MSS.monitors` returns the new +immutable `Monitor` objects, including the virtual-desktop entry at index zero; temporary string-key access provides the +migration bridge. Legacy `MSS.grab()` accepts those objects as well as current user-created dictionaries and PIL-style +tuples. The legacy and new paths must not be mixed on one `MSS` session. A session is initially uncommitted; its first +legacy or new operation commits it to that path for its lifetime. Using an API from the other path afterward raises +`SessionModeError`. The legacy backend is initialized lazily by the first legacy operation. Constructor-level `backend=` and `with_cursor=` configure only that path; they do not initialize it. New source enumeration and capture creation belong @@ -451,28 +444,91 @@ otherwise refuses the requested configuration. An already-established compatible ### Settled for this revision - Session + source-bound `Capture`; region on `grab`, not `create_capture` +- Sources carry private session provenance; manually constructed geometry is not a capture source - Pixel space is platform/process determined and inspectable, not caller-selectable +- Source bounds and result positions are session-global; requested regions are capture-local - Synchronous `create_capture_from_picker`; no target-category filter or public intermediate picked-source object - Picker cancellation returns `None`; every abnormal picker exit raises an exception -- `WindowArea` instead of `client_area_only`; regions relative to the chosen area +- `WindowArea` instead of `client_area_only`; `CLIENT` is the v1 default +- Picker `window_area` is conditional on the user selecting a window - `with_cursor: bool | None` for auto backend selection - `find_window` only (no `get_window`); window equality by identity - Enumeration snapshots as `tuple[...]` +- Dynamic region cropping is required for every v1 CPU backend +- Non-positive and completely clipped regions do not produce empty screenshots - Keep `ScreenShot`; allow strides; lazy packed `.bgra` - No `cls_image` on the new path - No `timeout` on `grab()` -- Expose `capture.backend` / `capabilities` / `backend_failures` for introspection +- Expose only the selected `capture.backend`; auto-selection order remains an implementation detail - `MSSError` as the new-API exception root - Legacy and source-bound operations cannot be mixed in one session; legacy initialization remains lazy -### Open - -- Default `WindowArea` for window captures -- A `frames()` generator design - ### Deferred -- `FrameTiming` / `CaptureStatistics` +- `frames()` and its buffering, timeout, notification, capability, timing, statistics, and concurrency contracts +- Native-stride and other non-packed CPU result layouts - `ScreenShotCpu` / `ScreenShotGpu` split and GPU result types - Richer pixel formats / color spaces - Region insets (negative width/height syntactic sugar) +- Structured backend-attempt diagnostics after successful auto-selection +- Public transparent-recovery guarantees for device resets and provider invalidation + +## Implementation task list + +Each task is intended to be reviewable independently and should include focused tests and documentation for its public +behavior. + +### Task 1: Immutable `Monitor` model (#470) + +- Replace the public monitor dictionary returned by `MSS.monitors` with a frozen, slotted `Monitor` dataclass. +- Include required geometry and the existing optional standard metadata. +- Preserve temporary string-key access such as `monitor["width"]`; do not promise the complete `Mapping` interface. +- Keep legacy `MSS.grab()` support for user-created monitor/region dictionaries and PIL-style tuples. +- Update platform enumeration, compatibility code, typing, tests, examples, and migration documentation. + +### Task 2: Platform session and source enumeration + +- Split shared platform/session resources from legacy capture initialization. +- Add `Desktop`, session-bound `Monitor`, and `Window` source provenance. +- Add `pixel_space`, `desktop`, `list_monitors()`, and `list_windows()` with immutable snapshot semantics. +- Define platform enumeration support and `SourceEnumerationUnsupportedError` behavior. +- Validate that foreign and manually constructed sources cannot enter the new capture path. + +### Task 3: `find_window()` convenience + +- Implement `find_window()` strictly on top of `list_windows()`; it does not participate in backend selection or capture + lifetime. +- Add the built-in ID, title, and property selectors described above. +- Preserve exact, case-sensitive string matching, regular-expression `search()`, `None` for no match, and + `WindowSelectionError` for ambiguous built-in matches. +- Validate custom-selector results and propagate custom exceptions unchanged. + +### Task 4: Source-bound CPU capture core + +- Add `Region`, `WindowArea`, `Capture`, `create_capture()`, and the v1 exception hierarchy. +- Implement context-manager lifetime, idempotent close, source ownership checks, and stable result-buffer lifetime. +- Implement global/source-local coordinate rules, clipping, positive-size validation, and dynamic per-grab regions. +- Exercise the public contract against small fake implementations before adding platform providers. + +### Task 5: Platform capture providers and auto-selection + +- Implement or adapt CPU providers for the supported X11, Windows, and macOS source types. +- Require each eligible provider to honor source type, cursor preference, window area, and dynamic region cropping. +- Add `backend="auto"`, explicit backend selection, fallback during creation, and `capture.backend` introspection. +- Keep provider ordering and successful fallback details internal. +- Test source loss, resizing, provider failure, and the no-silent-retarget rule per platform. + +### Task 6: System-picker capture + +- Add the synchronous picker path and cancellation/error semantics. +- Implement portal-based Wayland capture first; add other platform pickers only when their providers are implemented. +- Enforce conditional `window_area`, cursor requirements, parent-window handling, and no fallback after UI is shown. +- Keep asynchronous picker APIs deferred. + +### Task 7: Legacy migration and release integration + +- Initialize the legacy backend lazily and enforce the one-session/one-mode rule. +- Keep legacy coordinate, macOS resolution, cursor, and packed-buffer behavior unchanged. +- Ensure `save()` and `shot()` use private compatibility helpers without emitting misleading internal deprecation warnings. +- Add deprecation notices, upgrade documentation, release notes, and the required AI-assistance disclosure in the pull + request template. From 3c8b1cde2ca39098a097c89f4701ba901d2bfe8b Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Wed, 5 Aug 2026 15:34:19 +0200 Subject: [PATCH 09/15] Add design notes from discussion Need to update the capture design with these notes. --- design-notes.txt | 61 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 design-notes.txt diff --git a/design-notes.txt b/design-notes.txt new file mode 100644 index 00000000..6faf1777 --- /dev/null +++ b/design-notes.txt @@ -0,0 +1,61 @@ +1. Change typehint to Sequence (so we have flexiblity) but the actual return +type is Tuple + +Joels' generala rule, input types restirct, return types broadly. + +2. Manually constructed monitors, we don't do this anymore. We use `Region`. +Fix design. + +3. Display-oriented, the only API that we know that returns in scan-out + orientation is desktop duplication. Joel is OK with backend reporting whether + scanout or display orientation is used. But we would offer a flag in the + future to do the rotation for the user. We only have this for the desktop + duplication. We can make the boolean flag default to display orientation. We + can cross this bridge once we have DXGI. + +4. Joel has concerns about how App choosing DPI affects DXGI path. We need to explore + using python-mss with app choosing DPI. I can play around with it on my laptop. explore + Joel's matrix. + +5. Table this for a v1.1 design - for window events and discovery of changes. + +6. "Lost" is temporary inaccessible. Joel points out the application cannot + recover unless we give it enough information to recover. I will test "lost" + window case on Windows, to see what happens with window ID etc - because we + may then want to implement an `__eq__` operator to make this easier for the + user to match. We cannot finalize this design without some experimentation. + We should define what "lost" means and what those state transitions look + like. Need to look at WGC and newer compositor in Windows. DWM. There is + also full screen exclusive mode to look at for Windows. The lost window + happens there by just switching focus, I think. Let's figure out the states + of capture source and state machine. + +7. We should probably default to the lowest common denominator, macOS probably + doesn't do Client area. Will have to check with AI. + +8. The split between `create_capture` and `create_capture_from_picker` isn't great for + apps that should work for both Wayland and X Windows. + +9. Filtering in picker - we should bring it back and make clear that not all + platforms can support it. + +10. On many X windows impl we now have support for picker, so this could be the first route + an app tries. + +11. Desktop? It's an albatros around our neck - it may choose an inefficient + backend and doesn't work on Wayland. In some cases you will get an + inefficient backend and on Wayland you will not get a backend at all. + +12. Remove that ambiguity about virtual-display creation. Fix my language. Use one term, desktop. + +13. We need to inform the user of the actual physical size they captured on + MacOS somehow - so they understand how to map from logical capture to a + physical size. Today image.size is the size of the actual pixels and that + may be different than the width you pass in for the capture region. + So image.size is reporting physical pixels on all platforms currently. We + can keep it that way. You specify the region in logical pixels but they + return physical pixels. + +14. Joel is correct about "This would mean that legacy MSS.grab() ..." is already the case today. + +15. Fix that duplicated capture.close() (expand comment that context manager has already called) From c09ea7524f365106b274221020aa86fb4b05fc75 Mon Sep 17 00:00:00 2001 From: Halldor Fannar Date: Wed, 5 Aug 2026 17:28:47 +0200 Subject: [PATCH 10/15] Update to design according to notes --- design-notes.txt | 24 ++--- docs/source/capture-api-design.md | 166 ++++++++++++++++++++---------- 2 files changed, 126 insertions(+), 64 deletions(-) diff --git a/design-notes.txt b/design-notes.txt index 6faf1777..64060c37 100644 --- a/design-notes.txt +++ b/design-notes.txt @@ -1,12 +1,12 @@ -1. Change typehint to Sequence (so we have flexiblity) but the actual return +x 1. Change typehint to Sequence (so we have flexiblity) but the actual return type is Tuple Joels' generala rule, input types restirct, return types broadly. -2. Manually constructed monitors, we don't do this anymore. We use `Region`. +x 2. Manually constructed monitors, we don't do this anymore. We use `Region`. Fix design. -3. Display-oriented, the only API that we know that returns in scan-out +x 3. Display-oriented, the only API that we know that returns in scan-out orientation is desktop duplication. Joel is OK with backend reporting whether scanout or display orientation is used. But we would offer a flag in the future to do the rotation for the user. We only have this for the desktop @@ -17,9 +17,9 @@ Fix design. using python-mss with app choosing DPI. I can play around with it on my laptop. explore Joel's matrix. -5. Table this for a v1.1 design - for window events and discovery of changes. +x 5. Table this for a v1.1 design - for window events and discovery of changes. -6. "Lost" is temporary inaccessible. Joel points out the application cannot +x 6. "Lost" is temporary inaccessible. Joel points out the application cannot recover unless we give it enough information to recover. I will test "lost" window case on Windows, to see what happens with window ID etc - because we may then want to implement an `__eq__` operator to make this easier for the @@ -30,25 +30,25 @@ Fix design. happens there by just switching focus, I think. Let's figure out the states of capture source and state machine. -7. We should probably default to the lowest common denominator, macOS probably +x 7. We should probably default to the lowest common denominator, macOS probably doesn't do Client area. Will have to check with AI. 8. The split between `create_capture` and `create_capture_from_picker` isn't great for apps that should work for both Wayland and X Windows. -9. Filtering in picker - we should bring it back and make clear that not all +x 9. Filtering in picker - we should bring it back and make clear that not all platforms can support it. 10. On many X windows impl we now have support for picker, so this could be the first route an app tries. -11. Desktop? It's an albatros around our neck - it may choose an inefficient +x 11. Desktop? It's an albatros around our neck - it may choose an inefficient backend and doesn't work on Wayland. In some cases you will get an inefficient backend and on Wayland you will not get a backend at all. -12. Remove that ambiguity about virtual-display creation. Fix my language. Use one term, desktop. +x 12. Remove that ambiguity about virtual-display creation. Fix my language. Use one term, desktop. -13. We need to inform the user of the actual physical size they captured on +x 13. We need to inform the user of the actual physical size they captured on MacOS somehow - so they understand how to map from logical capture to a physical size. Today image.size is the size of the actual pixels and that may be different than the width you pass in for the capture region. @@ -56,6 +56,6 @@ Fix design. can keep it that way. You specify the region in logical pixels but they return physical pixels. -14. Joel is correct about "This would mean that legacy MSS.grab() ..." is already the case today. +x 14. Joel is correct about "This would mean that legacy MSS.grab() ..." is already the case today. -15. Fix that duplicated capture.close() (expand comment that context manager has already called) +x 15. Fix that duplicated capture.close() (expand comment that context manager has already called) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index 14cd8953..6424a5a1 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -45,21 +45,28 @@ class Region: ```python session.desktop -session.list_monitors() # tuple[Monitor, ...]; no desktop entry -session.list_windows() # tuple[Window, ...] +session.list_monitors() # Returns tuple[Monitor, ...]; no desktop entry +session.list_windows() # Returns tuple[Window, ...] ``` -Return type is `tuple[...]` so the snapshot cannot be resized or reassigned in place. +For typing we will mark the return type as `Sequence` for design freedom but +we will return a tuple so the snapshot cannot be resized or reassigned in place. Sources returned by a session carry private provenance. `create_capture()` accepts only a source returned by that same session. A manually constructed -`Monitor` may still be useful as geometry, and legacy `MSS.grab()` continues to -accept monitor dictionaries, but neither is a valid source for the new API. -Passing a foreign or manually constructed source to `create_capture()` raises -`ValueError`. - -Inputs are always display-oriented: a 90°-rotated 1920×1080 panel is addressed as width 1080, height 1920. Callers do -not apply a rotation transform. +`Monitor` will no longer be useful as geometry, legacy `MSS.grab()` accepts +dictionaries (for backwards compatibility) and recently introduced `Region` +type. See PR #566. Passing a foreign or manually constructed source to +`create_capture()` raises `ValueError`. + +Inputs are always display-oriented: a 90°-rotated 1920×1080 panel is addressed +as width 1080, height 1920. In rare cases a backend may operate in backbuffer +orientation (scanout) that has a different rotation than the display +orientation. When we encounter such a backend we will add an attribute to it +so that users can query for this behavior and therefore interpret the captured +image correctly. We can also add convenience flags to have MSS perform +a rotation so the image is returned display-oriented. We do not need to finalize +this design now. We can cross this bridge when we get there (most likely DXGI). ```python class PixelSpace(Enum): @@ -89,7 +96,9 @@ Linux/Wayland PHYSICAL capture-buffer pixels macOS LOGICAL ``` -The initial API does not provide conversion between pixel spaces and does not resample to produce another space. +The initial API does not resample to produce another space. Capture geometry remains in `session.pixel_space`, while +returned image buffers always contain physical pixels. The result metadata described below provides the mapping between +the two. ### Monitors @@ -164,8 +173,9 @@ Built-in selectors return `None` for no match and raise `WindowSelectionError` f exact and case-sensitive; regular expressions use `search()`. A custom selector must return one of the supplied windows or `None`, and its exceptions propagate unchanged. -Window identity is native and does not silently follow an application through native window recreation. A destroyed -window remains lost even if another window later has the same title, PID, class, or native ID. +Window identity is native and does not silently follow an application through native window recreation. Destroying a +window ends that source identity; a capture does not retarget even if another window later has the same title, PID, +class, or native ID. ## Capture creation @@ -190,12 +200,14 @@ The caller does not choose logical or physical coordinates. The resolved space i `session.pixel_space` and remains stable for the lifetime of the session and therefore the capture. `area` is valid only with a `Window` source. It must be `None` for `Monitor` and `Desktop`. For a `Window`, `None` -resolves to `WindowArea.CLIENT`, the v1 default. Region coordinates on later `grab` calls are relative to the chosen +resolves to `WindowArea.FULL`, the v1 default. Region coordinates on later `grab` calls are relative to the chosen extent: with `CLIENT`, `(0, 0)` is the client/content origin; with `FULL`, `(0, 0)` is the full native window origin. `FULL` means the native frame rectangle, including title bars, borders, menus, and other non-client chrome. It excludes compositor effects outside that rectangle, such as drop shadows, glow, capture-selection borders, and other external -decoration. A backend is eligible only if it can honor the requested extent. +decoration. A backend is eligible only if it can honor the requested extent. `CLIENT` is an optional backend capability; +an explicit request raises `BackendUnavailableError` if no eligible backend can guarantee the client extent. MSS does +not approximate it from platform-specific decoration sizes. `with_cursor` is tri-state for backend selection: @@ -219,18 +231,25 @@ Application-selected sources use `create_capture()`. User-selected surfaces use to the returned capture: ```python +class PickerTarget(Flag): + WINDOW = auto() + MONITOR = auto() + + def create_capture_from_picker( self, *, + target_hint: PickerTarget = PickerTarget.WINDOW | PickerTarget.MONITOR, backend: str = "auto", with_cursor: bool | None = None, - window_area: WindowArea = WindowArea.CLIENT, + window_area: WindowArea = WindowArea.FULL, parent_window: object | None = None, ) -> Capture | None: ... ``` ```python capture = session.create_capture_from_picker( + target_hint=PickerTarget.WINDOW, with_cursor=True, ) if capture is None: @@ -250,10 +269,13 @@ exit or failure to initialize the selected source raises `PickerError` and chains the native cause where available. Automatic backend fallback does not occur after UI has been presented. -MSS requests monitor and window sources where supported. The platform picker -determines how those choices are presented and may offer a narrower set; MSS does -not request virtual-display creation. One surface is selected. The selected item -and associated resources are not exposed as a public `CaptureSource`. +`target_hint` is a best-effort hint about which source categories to present. Its default requests no narrowing. A +backend narrows the picker when its platform API supports doing so, but the hint does not participate in backend +eligibility or fallback. A backend may ignore it and offer a broader set of categories; a selection outside +the hint is accepted normally. + +The platform picker determines how choices are presented. One surface is selected. The selected item and associated +resources are not exposed as a public `CaptureSource`. `window_area` applies only if the user selects a window and is irrelevant when a monitor is selected. A picker backend is eligible only if it can honor `window_area` whenever it offers window selection. `parent_window` is the @@ -286,9 +308,9 @@ source_extent = Region(left=0, top=0, width=source_width, height=source_height) effective = requested_region.intersection(source_extent) ``` -The returned image describes the clipped rectangle. A region with an empty intersection raises `ValueError`. If the -source itself currently has an empty extent, `grab()` raises `CaptureError`. V1 does not create zero-sized `ScreenShot` -objects. +`image.bounds` describes the effective clipped rectangle in `capture.pixel_space`. A region with an empty intersection +raises `ValueError`. If the source itself currently has an empty extent, frame acquisition is temporarily unavailable +and follows the `timeout` behavior described below. V1 does not create zero-sized `ScreenShot` objects. Every v1 CPU backend must support a different valid region on each `grab()` call, either through native cropping or by cropping inside MSS. Auto-selection never chooses a backend that rejects this normal `Capture` operation. @@ -315,17 +337,33 @@ platform compatibility. The result type remains **`ScreenShot`**. It is not replaced by a separate `Frame` type. ```python -image.pixel_space == capture.pixel_space +image.bounds # Effective captured Region in capture.pixel_space +image.pos # Origin of image.bounds in session-global coordinates +image.size # Width and height of the returned buffer in physical pixels ``` -`image.pos`, `image.size`, buffer dimensions, row stride, and array/tensor shapes use that space. Each result contains -one buffer in one space, never logical and physical copies. `capture.source_bounds` gives the captured source extent in -the capture's resolved pixel space. +`image.bounds` uses session-global coordinates and records the exact source rectangle represented by the result. +`image.pos` is its top-left origin and therefore also uses `capture.pixel_space`. `image.size`, `image.width`, +`image.height`, buffer dimensions, row stride, and array/tensor shapes always describe physical pixels. Each result +contains one physical buffer, never logical and physical copies. + +`source.bounds` and `capture.source_bounds` use session-global desktop coordinates in `capture.pixel_space`. A region +passed to `grab()` is capture-local. `image.bounds` is the effective clipped region translated by the origin of +`capture.source_bounds`; `image.pos` is the origin of that translated rectangle. For `WindowArea.CLIENT`, +`capture.source_bounds` describes the global client/content rectangle; for `WindowArea.FULL`, it describes the global +native frame rectangle. + +When `capture.pixel_space` is `PHYSICAL`, the width and height of `image.bounds` equal `image.size`. On macOS, capture +geometry is `LOGICAL` while `image.size` remains physical and may therefore differ. The exact per-result mapping is +available without a separate scale-factor API: + +```python +scale_x = image.size.width / image.bounds.width +scale_y = image.size.height / image.bounds.height +``` -`source.bounds` and `capture.source_bounds` use session-global desktop coordinates. A region passed to `grab()` is -capture-local. `image.pos` remains session-global: it is the origin of `capture.source_bounds` plus the origin of the -effective clipped region. For `WindowArea.CLIENT`, `capture.source_bounds` describes the global client/content rectangle; -for `WindowArea.FULL`, it describes the global native frame rectangle. +Callers must not combine `image.pos` and `image.size` as though they form a rectangle in one coordinate space; use +`image.bounds` for source geometry and `image.size` for indexing the pixel buffer. Future direction (not v1): `ScreenShot` as a base with `ScreenShotCpu` and `ScreenShotGpu` subclasses sharing common attributes; CPU and GPU results expose different buffers. @@ -336,9 +374,6 @@ attributes; CPU and GPU results expose different buffers. BGRA is obtained lazily via `.bgra` (may copy). NumPy/PIL/PyTorch and similar consumers can use the native layout directly when they support strides. -This would mean that legacy `MSS.grab()` stops returning packed buffers as today, but they can be obtained via attributes as -described above. - Exact pixel-format negotiation (HDR, YUV, etc.) is deferred with GPU work. ### Timing, statistics, and capabilities — **Deferred** @@ -354,13 +389,25 @@ an equivalent. ## Pull and continuous capture ```python -image = capture.grab() -image = capture.grab(region=...) +def grab( + self, + region: Region | None = None, + *, + timeout: float | None = None, +) -> ScreenShot: ... ``` `grab()` returns a newly constructed current image whenever the backend can complete the request. Repeated calls may contain identical pixels. It does not promise a distinct source presentation, a particular rate, or no-drop delivery. -**No `timeout` on `grab()`.** + +`timeout` is a non-negative duration in seconds. `None` (the default) waits indefinitely, and zero performs one +immediate acquisition attempt without waiting. If the backend does not provide usable pixels before the deadline, +`grab()` raises `CaptureTimeoutError`; the capture remains `OPEN` and may be used again. This deadline includes time +spent recovering from temporary provider failures. A negative timeout raises `ValueError`. + +MSS does not return a cached prior image merely to satisfy a timed acquisition. The caller can retain the last +successful image and decide whether to reuse it after `CaptureTimeoutError`. Pixel contents are not an availability +signal: an all-black or unchanged image may be a valid current result and is returned normally. ### `frames()` — **Deferred** @@ -375,8 +422,7 @@ separate change. Adding `frames()` later does not require changing source-bound with session.create_capture(source) as capture: image = capture.grab() -capture.close() -capture.close() # No effect. +capture.close() # No effect, was called by context close above and is idempotent ``` The caller owns captures and should close them promptly. The session weakly tracks live captures and closes them before @@ -384,7 +430,7 @@ closing shared platform resources. Returned image storage remains valid after bo ```text OPEN -├── source disappears ──────────> LOST +├── source removed ─────────────> REMOVED ├── unrecoverable provider error ─> FAILED └── close() ────────────────────> CLOSED ``` @@ -394,9 +440,19 @@ include device reset, DXGI duplication invalidation, frame-pool recreation, wind orientation changes for the same monitor. Recovery stays within the selected provider and preserves required capabilities. -A minimized, hidden, or temporarily unavailable window is not lost. Window destruction and monitor unplug are terminal -source loss. Recreated windows, reconnected monitors, matching titles, matching geometry, and reused list indices do -not silently retarget a capture. `Desktop` persists across monitor-topology changes. +A minimized, hidden, or temporarily unavailable window remains `OPEN`. Window destruction and monitor unplug are +terminal source removal. Recreated windows, reconnected monitors, matching titles, matching geometry, and reused list +indices do not silently retarget a capture. `Desktop` persists across monitor-topology changes. + +On Windows, minimizing an exclusive-fullscreen application or locking the user session may stop usable frames or make +GDI, D3D, DXGI, or frame-pool resources temporarily unusable without destroying the captured window. These are +temporary provider conditions: `grab()` follows its timeout behavior while MSS attempts recovery within the selected +provider. Capture can resume after restore or unlock if MSS can prove that the same source identity remains. The secure +lock desktop is not a replacement capture source. If the application destroys and recreates its native window during +that transition, the original source instead becomes `REMOVED`. + +`REMOVED` deliberately avoids the Direct3D "lost device" terminology. Device loss and desktop-duplication invalidation +are recoverable provider conditions when the source still exists; they are not terminal source removal. ## Exceptions @@ -410,7 +466,8 @@ MSSError ├── BackendUnavailableError ├── PickerError abnormal picker exit or selected-source initialization failure └── CaptureError - ├── SourceLostError terminal LOST + ├── CaptureTimeoutError retryable; capture remains OPEN + ├── CaptureSourceRemovedError terminal REMOVED └── CaptureClosedError ``` @@ -446,10 +503,12 @@ otherwise refuses the requested configuration. An already-established compatible - Session + source-bound `Capture`; region on `grab`, not `create_capture` - Sources carry private session provenance; manually constructed geometry is not a capture source - Pixel space is platform/process determined and inspectable, not caller-selectable -- Source bounds and result positions are session-global; requested regions are capture-local -- Synchronous `create_capture_from_picker`; no target-category filter or public intermediate picked-source object +- Source and image bounds are session-global; requested regions are capture-local +- `image.size` and buffer shapes are always physical; `image.bounds` maps them to the capture's coordinate space +- Synchronous `create_capture_from_picker`; best-effort `target_hint`, no hard target-category filter or public + intermediate picked-source object - Picker cancellation returns `None`; every abnormal picker exit raises an exception -- `WindowArea` instead of `client_area_only`; `CLIENT` is the v1 default +- `WindowArea` instead of `client_area_only`; `FULL` is the portable v1 default - Picker `window_area` is conditional on the user selecting a window - `with_cursor: bool | None` for auto backend selection - `find_window` only (no `get_window`); window equality by identity @@ -458,7 +517,8 @@ otherwise refuses the requested configuration. An already-established compatible - Non-positive and completely clipped regions do not produce empty screenshots - Keep `ScreenShot`; allow strides; lazy packed `.bgra` - No `cls_image` on the new path -- No `timeout` on `grab()` +- Per-call `grab(timeout=...)`; timeout is retryable and never returns an MSS-cached prior image +- Terminal `REMOVED` is distinct from recoverable provider or device unavailability - Expose only the selected `capture.backend`; auto-selection order remains an implementation detail - `MSSError` as the new-API exception root - Legacy and source-bound operations cannot be mixed in one session; legacy initialization remains lazy @@ -471,7 +531,6 @@ otherwise refuses the requested configuration. An already-established compatible - Richer pixel formats / color spaces - Region insets (negative width/height syntactic sugar) - Structured backend-attempt diagnostics after successful auto-selection -- Public transparent-recovery guarantees for device resets and provider invalidation ## Implementation task list @@ -507,7 +566,8 @@ behavior. - Add `Region`, `WindowArea`, `Capture`, `create_capture()`, and the v1 exception hierarchy. - Implement context-manager lifetime, idempotent close, source ownership checks, and stable result-buffer lifetime. -- Implement global/source-local coordinate rules, clipping, positive-size validation, and dynamic per-grab regions. +- Implement global/source-local coordinate rules, physical result sizes with `image.bounds` mapping, clipping, + positive-size validation, dynamic per-grab regions, and per-call acquisition timeouts. - Exercise the public contract against small fake implementations before adding platform providers. ### Task 5: Platform capture providers and auto-selection @@ -516,13 +576,15 @@ behavior. - Require each eligible provider to honor source type, cursor preference, window area, and dynamic region cropping. - Add `backend="auto"`, explicit backend selection, fallback during creation, and `capture.backend` introspection. - Keep provider ordering and successful fallback details internal. -- Test source loss, resizing, provider failure, and the no-silent-retarget rule per platform. +- Test source removal, retryable frame unavailability, resizing, provider failure, logical-to-physical result mapping, + and the no-silent-retarget rule per platform. ### Task 6: System-picker capture - Add the synchronous picker path and cancellation/error semantics. - Implement portal-based Wayland capture first; add other platform pickers only when their providers are implemented. -- Enforce conditional `window_area`, cursor requirements, parent-window handling, and no fallback after UI is shown. +- Apply `target_hint` where supported; enforce conditional `window_area`, cursor requirements, parent-window handling, + and no fallback after UI is shown. - Keep asynchronous picker APIs deferred. ### Task 7: Legacy migration and release integration From 7bf137c3d03dfb1cafb1718d1324049b999443b0 Mon Sep 17 00:00:00 2001 From: Halldor Fannar <6302686+halldorfannar@users.noreply.github.com> Date: Fri, 7 Aug 2026 14:29:59 +0200 Subject: [PATCH 11/15] Updating design based on feedback from Joel exe and pid attributes needed a note. --- docs/source/capture-api-design.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md index 6424a5a1..63fe1e1f 100644 --- a/docs/source/capture-api-design.md +++ b/docs/source/capture-api-design.md @@ -129,7 +129,9 @@ class Window: ``` Core properties describe the enumeration snapshot. Expensive properties such as `exe` may be loaded lazily and return -`None` if the process disappears or access is denied. A `Window` belongs to the session that enumerated it. +`None` if the process disappears or access is denied. It shoudl also be noted that propreties like `pid` andc `exe` +are not gauranteed to be accurate on X11, since the client application can modify them. +A `Window` belongs to the session that enumerated it. `Window` equality is **object identity**. Native IDs can be recycled after destroy, so `__eq__` is not based on `id`. Within the same session, callers who need “same OS window” compare `window.id` while that window still exists. From b083d5c8abca43a75f36436fedd2ce1a9774d617 Mon Sep 17 00:00:00 2001 From: Halldor Fannar <6302686+halldorfannar@users.noreply.github.com> Date: Fri, 28 Aug 2026 15:42:59 +0200 Subject: [PATCH 12/15] Improve the PR template We want users to check all boxes, so the display in GitHub makes sense. This is possible by just tweaking the words. We also have a better approach for the AI disclosure now. --- .github/PULL_REQUEST_TEMPLATE.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 2649d082..88023ec1 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -3,11 +3,11 @@ Fixes # (...) -- [ ] Tests added/updated -- [ ] Documentation updated -- [ ] Changelog entry added +- [ ] Tests added/updated (or your review concluded: not needed) +- [ ] Documentation updated (or your review concluded: not needed) +- [ ] Release notes added (or your review concluded: not needed) - [ ] `./check.sh` passed ### AI assistance disclosure -- [ ] No AI assistance was used to generate this contribution. +- AI was not / partially / entirely (choose one) used to create this change. From 846ba6334957f3f8ba05deec8e00fe93832f7f92 Mon Sep 17 00:00:00 2001 From: Halldor Fannar <6302686+halldorfannar@users.noreply.github.com> Date: Fri, 28 Aug 2026 15:46:23 +0200 Subject: [PATCH 13/15] Optimize the template further --- .github/PULL_REQUEST_TEMPLATE.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 88023ec1..f8bc8473 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,7 +1,6 @@ -### Changes proposed in this PR - Fixes # -(...) + + - [ ] Tests added/updated (or your review concluded: not needed) - [ ] Documentation updated (or your review concluded: not needed) From 7c33eef1f11d4fea46e06e0dfdaa00988adbf621 Mon Sep 17 00:00:00 2001 From: Halldor Fannar <6302686+halldorfannar@users.noreply.github.com> Date: Fri, 28 Aug 2026 15:58:55 +0200 Subject: [PATCH 14/15] Remove files from another task branch --- .agents/skills/ponytail/SKILL.md | 120 ------ design-notes.txt | 61 --- docs/source/capture-api-design.md | 598 ------------------------------ 3 files changed, 779 deletions(-) delete mode 100644 .agents/skills/ponytail/SKILL.md delete mode 100644 design-notes.txt delete mode 100644 docs/source/capture-api-design.md diff --git a/.agents/skills/ponytail/SKILL.md b/.agents/skills/ponytail/SKILL.md deleted file mode 100644 index 02c0712c..00000000 --- a/.agents/skills/ponytail/SKILL.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -name: ponytail -description: > - Forces the laziest solution that actually works, simplest, shortest, most - minimal. Channels a senior dev who has seen everything: question whether the - task needs to exist at all (YAGNI), reach for the standard library before - custom code, native platform features before dependencies, one line before - fifty. Supports intensity levels: lite, full (default), ultra. Use on ANY - coding task: writing, adding, refactoring, fixing, reviewing, or designing - code, and choosing libraries or dependencies. Also use whenever the user - says "ponytail", "be lazy", "lazy mode", "simplest solution", "minimal - solution", "yagni", "do less", or "shortest path", or complains about - over-engineering, bloat, boilerplate, or unnecessary dependencies. Do NOT - use for non-coding requests (general knowledge, prose, translation, - summaries, recipes). -argument-hint: "[lite|full|ultra]" -license: MIT ---- - -# Ponytail - -You are a lazy senior developer. Lazy means efficient, not careless. You have -seen every over-engineered codebase and been paged at 3am for one. The best -code is the code never written. - -## Persistence - -ACTIVE EVERY RESPONSE. No drift back to over-building. Still active if -unsure. Off only: "stop ponytail" / "normal mode". Default: **full**. -Switch: `/ponytail lite|full|ultra`. - -## The ladder - -Stop at the first rung that holds: - -1. **Does this need to exist at all?** Speculative need = skip it, say so in one line. (YAGNI) -2. **Already in this codebase?** A helper, util, type, or pattern that already lives here → reuse it. Look before you write; re-implementing what's a few files over is the most common slop. -3. **Stdlib does it?** Use it. -4. **Native platform feature covers it?** `` over a picker lib, CSS over JS, DB constraint over app code. -5. **Already-installed dependency solves it?** Use it. Never add a new one for what a few lines can do. -6. **Can it be one line?** One line. -7. **Only then:** the minimum code that works. - -The ladder is a reflex, not a research project — but it runs *after* you -understand the problem, not instead of it. Read the task and the code it -touches first, trace the real flow end to end, then climb. Two rungs work → -take the higher one and move on. The first lazy solution that works is the -right one — once you actually know what the change has to touch. - -**Bug fix = root cause, not symptom.** A report names a symptom. Before you -edit, grep every caller of the function you're about to touch. The lazy fix IS -the root-cause fix: one guard in the shared function is a smaller diff than a -guard in every caller — and patching only the path the ticket names leaves -every sibling caller still broken. Fix it once, where all callers route through. - -## Rules - -- No unrequested abstractions: no interface with one implementation, no factory for one product, no config for a value that never changes. -- No boilerplate, no scaffolding "for later", later can scaffold for itself. -- Deletion over addition. Boring over clever, clever is what someone decodes at 3am. -- Fewest files possible. Shortest working diff wins — but only once you understand the problem. The smallest change in the wrong place isn't lazy, it's a second bug. -- Complex request? Ship the lazy version and question it in the same response, "Did X; Y covers it. Need full X? Say so." Never stall on an answer you can default. -- Two stdlib options, same size? Take the one that's correct on edge cases. Lazy means writing less code, not picking the flimsier algorithm. -- Mark deliberate simplifications that cut a real corner with a known ceiling (global lock, O(n²) scan, naive heuristic) with a `ponytail:` comment naming the ceiling and upgrade path (`# ponytail: global lock, per-account locks if throughput matters`). - -## Output - -Code first. Then at most three short lines: what was skipped, when to add it. -No essays, no feature tours, no design notes. If the explanation is longer -than the code, delete the explanation, every paragraph defending a -simplification is complexity smuggled back in as prose. Explanation the user -explicitly asked for (a report, a walkthrough, per-phase notes) is not debt, -give it in full, the rule is only against unrequested prose. - -Pattern: `[code] → skipped: [X], add when [Y].` - -## Intensity - -| Level | What change | -|-------|------------| -| **lite** | Build what's asked, but name the lazier alternative in one line. User picks. | -| **full** | The ladder enforced. Stdlib and native first. Shortest diff, shortest explanation. Default. | -| **ultra** | YAGNI extremist. Deletion before addition. Ship the one-liner and challenge the rest of the requirement in the same breath. | - -Example: "Add a cache for these API responses." -- lite: "Done, cache added. FYI: `functools.lru_cache` covers this in one line if you'd rather not own a cache class." -- full: "`@lru_cache(maxsize=1000)` on the fetch function. Skipped custom cache class, add when lru_cache measurably falls short." -- ultra: "No cache until a profiler says so. When it does: `@lru_cache`. A hand-rolled TTL cache class is a bug farm with a hit rate." - -## When NOT to be lazy - -Never simplify away: input validation at trust boundaries, error handling -that prevents data loss, security measures, accessibility basics, anything -explicitly requested. User insists on the full version → build it, no -re-arguing. - -Never lazy about understanding the problem. The ladder shortens the -solution, never the reading. Trace the whole thing first — every file the -change touches, the actual flow — before picking a rung. Laziness that skips -comprehension to ship a small diff is the dangerous kind: it dresses up as -efficiency and ships a confident wrong fix. Read fully, then be lazy. - -Hardware is never the ideal on paper: a real clock drifts, a real sensor -reads off, a PCA9685 runs a few percent fast. Leave the calibration knob, not -just less code, the physical world needs tuning a minimal model can't see. - -Lazy code without its check is unfinished. Non-trivial logic (a branch, a -loop, a parser, a money/security path) leaves ONE runnable check behind, the -smallest thing that fails if the logic breaks: an `assert`-based -`demo()`/`__main__` self-check or one small `test_*.py`. No frameworks, no -fixtures, no per-function suites unless asked. Trivial one-liners need no -test, YAGNI applies to tests too. - -## Boundaries - -Ponytail governs what you build, not how you talk (pair with Caveman for -terse prose). "stop ponytail" / "normal mode": revert. Level persists until -changed or session end. - -The shortest path to done is the right path. diff --git a/design-notes.txt b/design-notes.txt deleted file mode 100644 index 64060c37..00000000 --- a/design-notes.txt +++ /dev/null @@ -1,61 +0,0 @@ -x 1. Change typehint to Sequence (so we have flexiblity) but the actual return -type is Tuple - -Joels' generala rule, input types restirct, return types broadly. - -x 2. Manually constructed monitors, we don't do this anymore. We use `Region`. -Fix design. - -x 3. Display-oriented, the only API that we know that returns in scan-out - orientation is desktop duplication. Joel is OK with backend reporting whether - scanout or display orientation is used. But we would offer a flag in the - future to do the rotation for the user. We only have this for the desktop - duplication. We can make the boolean flag default to display orientation. We - can cross this bridge once we have DXGI. - -4. Joel has concerns about how App choosing DPI affects DXGI path. We need to explore - using python-mss with app choosing DPI. I can play around with it on my laptop. explore - Joel's matrix. - -x 5. Table this for a v1.1 design - for window events and discovery of changes. - -x 6. "Lost" is temporary inaccessible. Joel points out the application cannot - recover unless we give it enough information to recover. I will test "lost" - window case on Windows, to see what happens with window ID etc - because we - may then want to implement an `__eq__` operator to make this easier for the - user to match. We cannot finalize this design without some experimentation. - We should define what "lost" means and what those state transitions look - like. Need to look at WGC and newer compositor in Windows. DWM. There is - also full screen exclusive mode to look at for Windows. The lost window - happens there by just switching focus, I think. Let's figure out the states - of capture source and state machine. - -x 7. We should probably default to the lowest common denominator, macOS probably - doesn't do Client area. Will have to check with AI. - -8. The split between `create_capture` and `create_capture_from_picker` isn't great for - apps that should work for both Wayland and X Windows. - -x 9. Filtering in picker - we should bring it back and make clear that not all - platforms can support it. - -10. On many X windows impl we now have support for picker, so this could be the first route - an app tries. - -x 11. Desktop? It's an albatros around our neck - it may choose an inefficient - backend and doesn't work on Wayland. In some cases you will get an - inefficient backend and on Wayland you will not get a backend at all. - -x 12. Remove that ambiguity about virtual-display creation. Fix my language. Use one term, desktop. - -x 13. We need to inform the user of the actual physical size they captured on - MacOS somehow - so they understand how to map from logical capture to a - physical size. Today image.size is the size of the actual pixels and that - may be different than the width you pass in for the capture region. - So image.size is reporting physical pixels on all platforms currently. We - can keep it that way. You specify the region in logical pixels but they - return physical pixels. - -x 14. Joel is correct about "This would mean that legacy MSS.grab() ..." is already the case today. - -x 15. Fix that duplicated capture.close() (expand comment that context manager has already called) diff --git a/docs/source/capture-api-design.md b/docs/source/capture-api-design.md deleted file mode 100644 index 63fe1e1f..00000000 --- a/docs/source/capture-api-design.md +++ /dev/null @@ -1,598 +0,0 @@ ---- -orphan: true ---- - -# Source-bound capture API - -Status: design proposal for issues [#470](https://github.com/BoboTiG/python-mss/issues/470) and -[#544](https://github.com/BoboTiG/python-mss/issues/544). - -Decisions below reflect discussion on #544 through 2026-08-01. Sections marked **Deferred** are not part of v1. - -## Overview - -`MSS` is a platform session. It enumerates sources and creates source-bound `Capture` objects. Captures return -`ScreenShot` objects (see the Results section below). - -```python -with MSS() as session: - monitor = session.list_monitors()[0] - - with session.create_capture(monitor) as capture: - image = capture.grab() - image = capture.grab(region=Region(left=10, top=20, width=640, height=480)) -``` - -The source types are explicit. A region restricts a source at acquisition time; it is not itself a source. - -```python -CaptureSource = Desktop | Monitor | Window - - -@dataclass(frozen=True, slots=True) -class Region: - left: int - top: int - width: int - height: int -``` - -## Public types - -### Sources - -`Desktop`, `Monitor`, and `Window` are read-only. Enumeration returns a new immutable snapshot on each call. - -```python -session.desktop -session.list_monitors() # Returns tuple[Monitor, ...]; no desktop entry -session.list_windows() # Returns tuple[Window, ...] -``` - -For typing we will mark the return type as `Sequence` for design freedom but -we will return a tuple so the snapshot cannot be resized or reassigned in place. - -Sources returned by a session carry private provenance. `create_capture()` -accepts only a source returned by that same session. A manually constructed -`Monitor` will no longer be useful as geometry, legacy `MSS.grab()` accepts -dictionaries (for backwards compatibility) and recently introduced `Region` -type. See PR #566. Passing a foreign or manually constructed source to -`create_capture()` raises `ValueError`. - -Inputs are always display-oriented: a 90°-rotated 1920×1080 panel is addressed -as width 1080, height 1920. In rare cases a backend may operate in backbuffer -orientation (scanout) that has a different rotation than the display -orientation. When we encounter such a backend we will add an attribute to it -so that users can query for this behavior and therefore interpret the captured -image correctly. We can also add convenience flags to have MSS perform -a rotation so the image is returned display-oriented. We do not need to finalize -this design now. We can cross this bridge when we get there (most likely DXGI). - -```python -class PixelSpace(Enum): - LOGICAL = auto() # points / DIPs / nominal display units - PHYSICAL = auto() # backing-store / framebuffer pixels -``` - -Pixel space is not selected by the caller. It is determined by the platform and any process or -thread DPI configuration established by the application. MSS does not change process DPI awareness implicitly, -but it does offer a utility function on Windows, for the user's convenience. - -Enumerated source geometry uses the session's effective pixel space: - -```python -session.pixel_space # PixelSpace -source.bounds # Region in session.pixel_space -``` -The session therefore informs the user of the pixel_space being used. This should be fixed for the -lifetime of an MSS session. Applications must not change their pixel space while the session or its captures are active. MSS may detect such a change and raise an exception. - -Typical behavior is: - -```text -Windows Determined by the application's DPI-awareness context -Linux/X11 PHYSICAL -Linux/Wayland PHYSICAL capture-buffer pixels -macOS LOGICAL -``` - -The initial API does not resample to produce another space. Capture geometry remains in `session.pixel_space`, while -returned image buffers always contain physical pixels. The result metadata described below provides the mapping between -the two. - -### Monitors - -`Monitor` is introduced first as a focused change for #470. It is a frozen, -slotted dataclass with required geometry and optional standard metadata such as -`is_primary`, `name`, `unique_id`, and Linux `output`. It supports attributes -and retains string-key access temporarily for migration, but does not promise -the complete `Mapping` interface. - -```python -monitor.width -monitor["width"] # Compatibility access. -``` - -### Windows - -```python -class Window: - id: int - title: str - pid: int | None - exe: str | None - class_name: str | None - bounds: Region - visible: bool - minimized: bool - attributes: Mapping[str, object] -``` - -Core properties describe the enumeration snapshot. Expensive properties such as `exe` may be loaded lazily and return -`None` if the process disappears or access is denied. It shoudl also be noted that propreties like `pid` andc `exe` -are not gauranteed to be accurate on X11, since the client application can modify them. -A `Window` belongs to the session that enumerated it. - -`Window` equality is **object identity**. Native IDs can be recycled after destroy, so `__eq__` is not based on `id`. -Within the same session, callers who need “same OS window” compare `window.id` while that window still exists. - -`list_windows()` includes top-level application windows and minimized windows. Hidden windows require -`include_hidden=True`. Child controls, shell surfaces, menus, tooltips, and similar transient windows are excluded where -the platform can identify them reliably. - -On platforms that prohibit application-driven enumeration, these APIs raise rather than returning an empty snapshot: - -```python -session.list_windows() # SourceEnumerationUnsupportedError -session.list_monitors() # SourceEnumerationUnsupportedError -session.desktop # SourceEnumerationUnsupportedError -``` - -An empty tuple means enumeration succeeded and found no sources. Portal-only Wayland uses the system-picker path below. - -```python -WindowSelector = Callable[[tuple[Window, ...]], Window | None] - - -def find_window( - self, - selector: WindowSelector, - include_hidden: bool = False, -) -> Window | None: ... -``` - -MSS supplies selectors for common cases: - -```python -session.find_window(window_by_id(hwnd)) -session.find_window(window_by_title("Game")) -session.find_window(window_by_title(re.compile(r"(^| - )Firefox$"))) -session.find_window(window_by_properties(pid=12345, exe="game.exe")) -session.find_window(lambda windows: choose_window(windows)) -``` - -Built-in selectors return `None` for no match and raise `WindowSelectionError` for multiple matches. String matching is -exact and case-sensitive; regular expressions use `search()`. A custom selector must return one of the supplied windows -or `None`, and its exceptions propagate unchanged. - -Window identity is native and does not silently follow an application through native window recreation. Destroying a -window ends that source identity; a capture does not retarget even if another window later has the same title, PID, -class, or native ID. - -## Capture creation - -```python -class WindowArea(Enum): - CLIENT = auto() # client area / content rect / client window - FULL = auto() # entire native window, including non-client chrome -``` - -```python -def create_capture( - self, - source: CaptureSource, - *, - backend: str = "auto", - with_cursor: bool | None = None, - area: WindowArea | None = None, -) -> Capture: ... -``` - -The caller does not choose logical or physical coordinates. The resolved space is inspectable through -`session.pixel_space` and remains stable for the lifetime of the session and therefore the capture. - -`area` is valid only with a `Window` source. It must be `None` for `Monitor` and `Desktop`. For a `Window`, `None` -resolves to `WindowArea.FULL`, the v1 default. Region coordinates on later `grab` calls are relative to the chosen -extent: with `CLIENT`, `(0, 0)` is the client/content origin; with `FULL`, `(0, 0)` is the full native window origin. - -`FULL` means the native frame rectangle, including title bars, borders, menus, and other non-client chrome. It excludes -compositor effects outside that rectangle, such as drop shadows, glow, capture-selection borders, and other external -decoration. A backend is eligible only if it can honor the requested extent. `CLIENT` is an optional backend capability; -an explicit request raises `BackendUnavailableError` if no eligible backend can guarantee the client extent. MSS does -not approximate it from platform-specific decoration sizes. - -`with_cursor` is tri-state for backend selection: - -```text -True cursor must be included; only backends that can guarantee that are eligible -False cursor must be excluded; same filter the other way -None don't care (default); auto may pick the best otherwise-eligible backend; - cursor presence is unspecified -``` - -When cursor inclusion is required (`True`), cursor pixels are composited only where they intersect the final clipped -output. - -Source type, cursor preference, and window area are requirements expressed by the capture request. Pixel space is an -observed property, not a capture request. Streaming capabilities are deferred with `frames()` and are not part of the -v1 capture-creation API. - -### System-picker creation - -Application-selected sources use `create_capture()`. User-selected surfaces use a system picker and are bound directly -to the returned capture: - -```python -class PickerTarget(Flag): - WINDOW = auto() - MONITOR = auto() - - -def create_capture_from_picker( - self, - *, - target_hint: PickerTarget = PickerTarget.WINDOW | PickerTarget.MONITOR, - backend: str = "auto", - with_cursor: bool | None = None, - window_area: WindowArea = WindowArea.FULL, - parent_window: object | None = None, -) -> Capture | None: ... -``` - -```python -capture = session.create_capture_from_picker( - target_hint=PickerTarget.WINDOW, - with_cursor=True, -) -if capture is None: - return # User cancelled. -``` - -The method blocks while the system picker is open and returns when the user -selects a source or the platform reports that the user cancelled. `None` means -cancellation only. Failure to create or operate the picker, loss of the portal -or native service, permission failure, an invalid platform response, and failure -to initialize the selected capture raise an exception. It has no timeout. - -If no eligible picker backend can be initialized before UI is presented, -`BackendUnavailableError` reports attempted-provider context in the same way as -`create_capture()`. After a picker backend has presented UI, an abnormal picker -exit or failure to initialize the selected source raises `PickerError` and -chains the native cause where available. Automatic backend fallback does not -occur after UI has been presented. - -`target_hint` is a best-effort hint about which source categories to present. Its default requests no narrowing. A -backend narrows the picker when its platform API supports doing so, but the hint does not participate in backend -eligibility or fallback. A backend may ignore it and offer a broader set of categories; a selection outside -the hint is accepted normally. - -The platform picker determines how choices are presented. One surface is selected. The selected item and associated -resources are not exposed as a public `CaptureSource`. -`window_area` applies only if the user selects a window and is irrelevant when a -monitor is selected. A picker backend is eligible only if it can honor -`window_area` whenever it offers window selection. `parent_window` is the -platform-specific parent handle for the picker. MSS presents at most one picker -and does not reprompt after selection if capture initialization fails. - -This path is required by portal-only Wayland. In the future we may also offer it for Windows WGC and macOS -ScreenCaptureKit. Although WGC, -ScreenCaptureKit, and the Wayland portal complete selection asynchronously at the platform level, the initial public API -is synchronous. A `create_capture_from_picker_async()` variant may be added later without changing the synchronous API. - -### Regions - -`region` is **not** a `create_capture` argument. It is optional on `grab()`: - -```python -capture.grab() -capture.grab(region=Region(left=10, top=20, width=640, height=480)) -``` - -`None` captures the full source extent (subject to `area` for windows). Region may differ across `grab` calls. -Coordinates and clipping use `capture.pixel_space`. - -Region fields are integers. Negative `left` and `top` values are valid. Width and height must be positive; zero or a -negative value raises `ValueError`. The effective rectangle is recomputed for each acquisition against the -capture-local extent: - -```python -source_extent = Region(left=0, top=0, width=source_width, height=source_height) -effective = requested_region.intersection(source_extent) -``` - -`image.bounds` describes the effective clipped rectangle in `capture.pixel_space`. A region with an empty intersection -raises `ValueError`. If the source itself currently has an empty extent, frame acquisition is temporarily unavailable -and follows the `timeout` behavior described below. V1 does not create zero-sized `ScreenShot` objects. - -Every v1 CPU backend must support a different valid region on each `grab()` call, either through native cropping or by -cropping inside MSS. Auto-selection never chooses a backend that rejects this normal `Capture` operation. - -Insets / negative width-height as a crop-from-edges sugar remain deferred. - -### Backend selection - -```python -capture.backend # "xshmgetimage", "gdi", ... -``` - -For `backend="auto"`, MSS filters the platform providers by source and capture configuration, then tries eligible -providers in implementation-defined order. If none succeeds, `BackendUnavailableError` reports useful attempted-provider -context in its message without making a structured failure history part of the public API. - -An explicit backend never falls back. Automatic fallback occurs only during capture creation; recovery never changes a -capture's provider. Auto-selection order is an implementation detail and may improve between releases. Users who need a -specific provider select it explicitly. Providers may be changed or disabled at any time for correctness, security, or -platform compatibility. - -## Results - -The result type remains **`ScreenShot`**. It is not replaced by a separate `Frame` type. - -```python -image.bounds # Effective captured Region in capture.pixel_space -image.pos # Origin of image.bounds in session-global coordinates -image.size # Width and height of the returned buffer in physical pixels -``` - -`image.bounds` uses session-global coordinates and records the exact source rectangle represented by the result. -`image.pos` is its top-left origin and therefore also uses `capture.pixel_space`. `image.size`, `image.width`, -`image.height`, buffer dimensions, row stride, and array/tensor shapes always describe physical pixels. Each result -contains one physical buffer, never logical and physical copies. - -`source.bounds` and `capture.source_bounds` use session-global desktop coordinates in `capture.pixel_space`. A region -passed to `grab()` is capture-local. `image.bounds` is the effective clipped region translated by the origin of -`capture.source_bounds`; `image.pos` is the origin of that translated rectangle. For `WindowArea.CLIENT`, -`capture.source_bounds` describes the global client/content rectangle; for `WindowArea.FULL`, it describes the global -native frame rectangle. - -When `capture.pixel_space` is `PHYSICAL`, the width and height of `image.bounds` equal `image.size`. On macOS, capture -geometry is `LOGICAL` while `image.size` remains physical and may therefore differ. The exact per-result mapping is -available without a separate scale-factor API: - -```python -scale_x = image.size.width / image.bounds.width -scale_y = image.size.height / image.bounds.height -``` - -Callers must not combine `image.pos` and `image.size` as though they form a rectangle in one coordinate space; use -`image.bounds` for source geometry and `image.size` for indexing the pixel buffer. - -Future direction (not v1): `ScreenShot` as a base with `ScreenShotCpu` and `ScreenShotGpu` subclasses sharing common -attributes; CPU and GPU results expose different buffers. - -### Buffer layout - -`ScreenShot` does **not** require tightly packed rows. Backends may return a native stride/pitch. Contiguous packed -BGRA is obtained lazily via `.bgra` (may copy). NumPy/PIL/PyTorch and similar consumers can use the native layout -directly when they support strides. - -Exact pixel-format negotiation (HDR, YUV, etc.) is deferred with GPU work. - -### Timing, statistics, and capabilities — **Deferred** - -`CaptureCapability`, `FrameTiming`, and `CaptureStatistics` are deferred past the first cut of the source-bound API. -They can be designed with the first operation that consumes them rather than becoming speculative v1 public surface. - -### `cls_image` - -Dropped from the new API. Legacy `MSS.cls_image` may remain on the deprecated path; source-bound capture does not grow -an equivalent. - -## Pull and continuous capture - -```python -def grab( - self, - region: Region | None = None, - *, - timeout: float | None = None, -) -> ScreenShot: ... -``` - -`grab()` returns a newly constructed current image whenever the backend can complete the request. Repeated calls may -contain identical pixels. It does not promise a distinct source presentation, a particular rate, or no-drop delivery. - -`timeout` is a non-negative duration in seconds. `None` (the default) waits indefinitely, and zero performs one -immediate acquisition attempt without waiting. If the backend does not provide usable pixels before the deadline, -`grab()` raises `CaptureTimeoutError`; the capture remains `OPEN` and may be used again. This deadline includes time -spent recovering from temporary provider failures. A negative timeout raises `ValueError`. - -MSS does not return a cached prior image merely to satisfy a timed acquisition. The caller can retain the last -successful image and decide whether to reuse it after `CaptureTimeoutError`. Pixel contents are not an availability -signal: an all-black or unchanged image may be a valid current result and is returned normally. - -### `frames()` — **Deferred** - -The streaming API, buffering, timeouts, update notifications, and concurrency rules will be designed together in a -separate change. Adding `frames()` later does not require changing source-bound capture creation or `grab()`. - -## Lifetime and recovery - -`Capture` is an idempotent context manager: `__exit__` closes; `close()` may be called again with no effect. - -```python -with session.create_capture(source) as capture: - image = capture.grab() - -capture.close() # No effect, was called by context close above and is idempotent -``` - -The caller owns captures and should close them promptly. The session weakly tracks live captures and closes them before -closing shared platform resources. Returned image storage remains valid after both capture and session closure. - -```text -OPEN -├── source removed ─────────────> REMOVED -├── unrecoverable provider error ─> FAILED -└── close() ────────────────────> CLOSED -``` - -MSS recovers transparently while it can prove the same source identity and capture contract remain available. Examples -include device reset, DXGI duplication invalidation, frame-pool recreation, window resizing, and resolution or -orientation changes for the same monitor. Recovery stays within the selected provider and preserves required -capabilities. - -A minimized, hidden, or temporarily unavailable window remains `OPEN`. Window destruction and monitor unplug are -terminal source removal. Recreated windows, reconnected monitors, matching titles, matching geometry, and reused list -indices do not silently retarget a capture. `Desktop` persists across monitor-topology changes. - -On Windows, minimizing an exclusive-fullscreen application or locking the user session may stop usable frames or make -GDI, D3D, DXGI, or frame-pool resources temporarily unusable without destroying the captured window. These are -temporary provider conditions: `grab()` follows its timeout behavior while MSS attempts recovery within the selected -provider. Capture can resume after restore or unlock if MSS can prove that the same source identity remains. The secure -lock desktop is not a replacement capture source. If the application destroys and recreates its native window during -that transition, the original source instead becomes `REMOVED`. - -`REMOVED` deliberately avoids the Direct3D "lost device" terminology. Device loss and desktop-duplication invalidation -are recoverable provider conditions when the source still exists; they are not terminal source removal. - -## Exceptions - -```text -MSSError -├── ScreenShotError legacy API -├── SessionClosedError -├── SessionModeError legacy/source-bound API paths mixed -├── SourceEnumerationUnsupportedError -├── WindowSelectionError -├── BackendUnavailableError -├── PickerError abnormal picker exit or selected-source initialization failure -└── CaptureError - ├── CaptureTimeoutError retryable; capture remains OPEN - ├── CaptureSourceRemovedError terminal REMOVED - └── CaptureClosedError -``` - -`MSSError` is the package top-level exception. `ScreenShotError` remains for the legacy path only; the new API is not -rooted at `ScreenShotError`. - -Invalid argument types use `TypeError`; invalid values use `ValueError`. Window-selector callback exceptions propagate -unchanged. `PickerError` and `CaptureError` chain their native causes where available. - -## Compatibility and deferred work - -The deprecated `MSS.grab()`, `MSS.monitors`, `save()`, and `shot()` retain their current desktop-rectangle semantics -(including today's macOS nominal-resolution default and packed `ScreenShot` buffers). `MSS.monitors` returns the new -immutable `Monitor` objects, including the virtual-desktop entry at index zero; temporary string-key access provides the -migration bridge. Legacy `MSS.grab()` accepts those objects as well as current user-created dictionaries and PIL-style -tuples. The legacy and new paths must not be mixed on one `MSS` session. A session is initially uncommitted; its first -legacy or new operation commits it to that path for its lifetime. Using an API from the other path afterward raises -`SessionModeError`. - -The legacy backend is initialized lazily by the first legacy operation. Constructor-level `backend=` and -`with_cursor=` configure only that path; they do not initialize it. New source enumeration and capture creation belong -to the source-bound (new) path, including `desktop`, `list_monitors()`, `list_windows()`, `find_window()`, `create_capture()`, -and `create_capture_from_picker()`. Compatibility helpers do not call deprecated public methods internally. - -On Windows, only initialization of the legacy GDI path attempts to establish the process DPI awareness required by its -existing physical-desktop coordinate contract. Source-bound session creation and use never change process or thread DPI -awareness implicitly. We will improve the legacy GDI initialization so it validates the resulting awareness and raises `ScreenShotError` if an -incompatible value was already established by the application manifest or by other code in the process, or if Windows -otherwise refuses the requested configuration. An already-established compatible value is accepted. - -### Settled for this revision - -- Session + source-bound `Capture`; region on `grab`, not `create_capture` -- Sources carry private session provenance; manually constructed geometry is not a capture source -- Pixel space is platform/process determined and inspectable, not caller-selectable -- Source and image bounds are session-global; requested regions are capture-local -- `image.size` and buffer shapes are always physical; `image.bounds` maps them to the capture's coordinate space -- Synchronous `create_capture_from_picker`; best-effort `target_hint`, no hard target-category filter or public - intermediate picked-source object -- Picker cancellation returns `None`; every abnormal picker exit raises an exception -- `WindowArea` instead of `client_area_only`; `FULL` is the portable v1 default -- Picker `window_area` is conditional on the user selecting a window -- `with_cursor: bool | None` for auto backend selection -- `find_window` only (no `get_window`); window equality by identity -- Enumeration snapshots as `tuple[...]` -- Dynamic region cropping is required for every v1 CPU backend -- Non-positive and completely clipped regions do not produce empty screenshots -- Keep `ScreenShot`; allow strides; lazy packed `.bgra` -- No `cls_image` on the new path -- Per-call `grab(timeout=...)`; timeout is retryable and never returns an MSS-cached prior image -- Terminal `REMOVED` is distinct from recoverable provider or device unavailability -- Expose only the selected `capture.backend`; auto-selection order remains an implementation detail -- `MSSError` as the new-API exception root -- Legacy and source-bound operations cannot be mixed in one session; legacy initialization remains lazy - -### Deferred - -- `frames()` and its buffering, timeout, notification, capability, timing, statistics, and concurrency contracts -- Native-stride and other non-packed CPU result layouts -- `ScreenShotCpu` / `ScreenShotGpu` split and GPU result types -- Richer pixel formats / color spaces -- Region insets (negative width/height syntactic sugar) -- Structured backend-attempt diagnostics after successful auto-selection - -## Implementation task list - -Each task is intended to be reviewable independently and should include focused tests and documentation for its public -behavior. - -### Task 1: Immutable `Monitor` model (#470) - -- Replace the public monitor dictionary returned by `MSS.monitors` with a frozen, slotted `Monitor` dataclass. -- Include required geometry and the existing optional standard metadata. -- Preserve temporary string-key access such as `monitor["width"]`; do not promise the complete `Mapping` interface. -- Keep legacy `MSS.grab()` support for user-created monitor/region dictionaries and PIL-style tuples. -- Update platform enumeration, compatibility code, typing, tests, examples, and migration documentation. - -### Task 2: Platform session and source enumeration - -- Split shared platform/session resources from legacy capture initialization. -- Add `Desktop`, session-bound `Monitor`, and `Window` source provenance. -- Add `pixel_space`, `desktop`, `list_monitors()`, and `list_windows()` with immutable snapshot semantics. -- Define platform enumeration support and `SourceEnumerationUnsupportedError` behavior. -- Validate that foreign and manually constructed sources cannot enter the new capture path. - -### Task 3: `find_window()` convenience - -- Implement `find_window()` strictly on top of `list_windows()`; it does not participate in backend selection or capture - lifetime. -- Add the built-in ID, title, and property selectors described above. -- Preserve exact, case-sensitive string matching, regular-expression `search()`, `None` for no match, and - `WindowSelectionError` for ambiguous built-in matches. -- Validate custom-selector results and propagate custom exceptions unchanged. - -### Task 4: Source-bound CPU capture core - -- Add `Region`, `WindowArea`, `Capture`, `create_capture()`, and the v1 exception hierarchy. -- Implement context-manager lifetime, idempotent close, source ownership checks, and stable result-buffer lifetime. -- Implement global/source-local coordinate rules, physical result sizes with `image.bounds` mapping, clipping, - positive-size validation, dynamic per-grab regions, and per-call acquisition timeouts. -- Exercise the public contract against small fake implementations before adding platform providers. - -### Task 5: Platform capture providers and auto-selection - -- Implement or adapt CPU providers for the supported X11, Windows, and macOS source types. -- Require each eligible provider to honor source type, cursor preference, window area, and dynamic region cropping. -- Add `backend="auto"`, explicit backend selection, fallback during creation, and `capture.backend` introspection. -- Keep provider ordering and successful fallback details internal. -- Test source removal, retryable frame unavailability, resizing, provider failure, logical-to-physical result mapping, - and the no-silent-retarget rule per platform. - -### Task 6: System-picker capture - -- Add the synchronous picker path and cancellation/error semantics. -- Implement portal-based Wayland capture first; add other platform pickers only when their providers are implemented. -- Apply `target_hint` where supported; enforce conditional `window_area`, cursor requirements, parent-window handling, - and no fallback after UI is shown. -- Keep asynchronous picker APIs deferred. - -### Task 7: Legacy migration and release integration - -- Initialize the legacy backend lazily and enforce the one-session/one-mode rule. -- Keep legacy coordinate, macOS resolution, cursor, and packed-buffer behavior unchanged. -- Ensure `save()` and `shot()` use private compatibility helpers without emitting misleading internal deprecation warnings. -- Add deprecation notices, upgrade documentation, release notes, and the required AI-assistance disclosure in the pull - request template. From e42983bab557f1d1dbdc77d6cd015a82db2faaa4 Mon Sep 17 00:00:00 2001 From: Halldor Fannar <6302686+halldorfannar@users.noreply.github.com> Date: Fri, 28 Aug 2026 16:00:30 +0200 Subject: [PATCH 15/15] Add Windows check script flavor --- .github/PULL_REQUEST_TEMPLATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index f8bc8473..3c31c095 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -5,7 +5,7 @@ Fixes # - [ ] Tests added/updated (or your review concluded: not needed) - [ ] Documentation updated (or your review concluded: not needed) - [ ] Release notes added (or your review concluded: not needed) -- [ ] `./check.sh` passed +- [ ] `./check.sh` (or `./check.ps1`) passed ### AI assistance disclosure