Playback performance and sync

[richiemcilroy]

Enable Linux compilation for scap-targets and add a new playback benchmark example to unblock performance testing on Linux.

---

 

---

Note

Medium Risk
Touches core frame transport/rendering logic (SAB atomics, ordering, backpressure, fallbacks), so regressions could impact playback smoothness or stability despite added tests and telemetry.

Overview
Playback/transport pipeline hardening and instrumentation. The desktop app’s frame ingestion/transport path is refactored to better handle out-of-order frames, worker backpressure, and SharedArrayBuffer (SAB) write failures, with a set of small decision utilities (order, retry, inflight, stride dispatch, buffer sizing) plus unit tests.

SAB + worker flow changes. socket.ts now dynamically resizes the SAB based on incoming frame sizes, retries SAB writes with a capped limit before falling back to worker transfer, tracks worker in-flight frames (with backpressure/superseded-drop accounting), and records extensive per-window and total counters. The frame-worker render queue is simplified to a single latest queued frame, adds out-of-order drop logic, and reports render source (shared vs worker).

Telemetry/UI updates and format cleanup. PerformanceOverlay polls getFpsStats() and displays/copies the new transport metrics. NV12-specific paths are removed from webgpu-renderer and the main-thread/worker frame handling, and stride-correction-worker now includes frameNumber plus validation/error responses to keep direct rendering ordered.

Cross-platform build fixes. Rust crates add non-macOS/windows stubs/guards (camera-ffmpeg unsupported error; cursor-capture bounds gated) to allow compilation on other platforms (e.g., Linux).

^{Written by Cursor Bugbot for commit f792f1c. This will update automatically on new commits. Configure here.}

Greptile Overview

Greptile Summary

This PR enables Linux compilation for scap-targets and adds comprehensive playback performance benchmarking infrastructure. The changes include:

• Linux compilation support: Added stub implementations across multiple crates (scap-targets, camera-ffmpeg, scap-ffmpeg, cursor-capture, timestamp) with conditional compilation guards. The Linux implementations return hardcoded/no-op values to enable builds but don't provide real functionality yet.

• Playback benchmarking suite: Added 5 new benchmark examples (playback-benchmark, scrub-benchmark, decode-benchmark, and corresponding CSV report tools) with extensive documentation in PLAYBACK-BENCHMARKS.md and PLAYBACK-FINDINGS.md.

• Frame transport refactor: Major improvements to the desktop app's frame handling:

  • Extracted frame transport logic into modular utilities (frame-transport-config, -inflight, -order, -retry, -stride)
  • Enhanced SharedArrayBuffer management with dynamic resizing, retry logic, and comprehensive metrics
  • Added backpressure control and supersession logic to prevent worker overload
  • Removed main-thread NV12 conversion code
  • Added 40+ performance metrics tracked in PerformanceOverlay

• Decoder optimizations: FFmpeg decoder now supports "scrub supersession" and "latest-first" prioritization via environment variables to improve seek performance during video scrubbing.

• Playback improvements: Added startup tracing, environment-based configuration flags, and enhanced frame caching in the editor playback system.

The frame transport changes are well-tested with comprehensive unit tests for each new module. The benchmarking infrastructure provides detailed CSV output for performance analysis.

Confidence Score: 4/5

• This PR is safe to merge with minor caveats around the Linux stub implementations
• The changes are well-structured and thoroughly tested. The frame transport refactor improves modularity and adds comprehensive unit tests. The benchmarking infrastructure is solid with extensive documentation. The Linux stubs are clearly marked as non-functional placeholders. Main concern is the complexity of the socket.ts refactor (1100+ lines changed), though the modular extraction helps. The decoder changes use environment variables for tuning, which is appropriate for performance experimentation.
• Pay close attention to apps/desktop/src/utils/socket.ts due to the extensive refactoring and new frame ordering/backpressure logic

Important Files Changed

Filename	Overview
crates/scap-targets/src/platform/linux.rs	Added Linux stub implementation with hardcoded values (1920x1080, 60Hz) to enable compilation; no real functionality yet
crates/editor/src/playback.rs	Improved playback with startup tracing, environment-based configuration flags, and enhanced frame caching
crates/rendering/src/decoder/ffmpeg.rs	Added scrub supersession logic with latest-first prioritization, configurable via environment variables for performance tuning
apps/desktop/src/utils/socket.ts	Major refactor: removed NV12 main-thread conversion, added frame transport modules, retry logic, inflight tracking, and comprehensive metrics
apps/desktop/src/utils/frame-transport-config.ts	New module for shared buffer configuration with dynamic sizing and alignment logic
apps/desktop/src/utils/shared-frame-buffer.ts	Enhanced shared frame buffer with writable slot search, shutdown signaling, and improved synchronization
apps/desktop/src/utils/frame-worker.ts	Simplified worker by removing complex queuing logic and NV12 handling, focusing on SharedArrayBuffer consumption
apps/desktop/src/routes/editor/PerformanceOverlay.tsx	Added extensive transport statistics tracking with 40+ metrics for frame buffer, worker, and ordering performance

Flowchart

flowchart TD
    A[Frame Received from Rust] --> B{Frame Order Check}
    B -->|Out of Order| C[Drop Frame]
    B -->|In Order| D{SharedArrayBuffer Available?}
    
    D -->|Yes| E{SAB Write Attempt}
    E -->|Success| F[Producer Writes to SAB]
    E -->|Busy/Full| G{Retry Count < Limit?}
    G -->|Yes| H[Schedule Retry]
    G -->|No| I[Fallback to Worker]
    
    D -->|No| I
    H --> E
    
    F --> J[Consumer Reads from SAB]
    J --> K[Render Directly]
    
    I --> L{Worker Inflight < Limit?}
    L -->|Yes| M[Send to Frame Worker]
    L -->|No| N[Backpressure Drop]
    
    M --> O{Needs Stride Correction?}
    O -->|Yes| P{Stride Worker Available?}
    P -->|Yes| Q[Stride Correction Worker]
    P -->|No| R[Queue for Later]
    Q --> S[Worker Renders Frame]
    O -->|No| S
    
    R --> P
    S --> T[Post ImageBitmap]
    T --> U[Main Thread Renders]
    
    K --> V[Performance Metrics]
    U --> V
    C --> V
    N --> V

Loading

_{Last reviewed commit: f792f1c}

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

This is the final PR Bugbot will review for you during this billing cycle

Your free Bugbot reviews will reset on March 8

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

[cursor]

Pending stride corrections can get stuck

Medium Severity

When responseOrderDecision.action is drop, the handler returns before calling flushPending(). This leaves pendingStrideCorrection queued even though strideWorkerInFlight is already false, so queued stride-fix requests can stop progressing after out-of-order responses in socket.ts.

Additional Locations (1)

• apps/desktop/src/utils/socket.ts#L401-L407

 

[greptile-apps]

_{39 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

hardcoded 1920x1080 resolution may not match actual display

Suggested change

	pub fn physical_size(&self) -> Option<PhysicalSize> {
	Some(PhysicalSize::new(1920.0, 1080.0))
	}
	pub fn physical_size(&self) -> Option<PhysicalSize> {
	None
	}

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/scap-targets/src/platform/linux.rs
Line: 29:31

Comment:
hardcoded 1920x1080 resolution may not match actual display

```suggestion
    pub fn physical_size(&self) -> Option<PhysicalSize> {
        None
    }
```

How can I resolve this? If you propose a fix, please make it concise.

[tembo]

Minor naming/units nit: getFpsStats()'s renderFps / mbPerSec (from socket.ts) are windowed counters that reset periodically, so sampling them every 250ms and labeling as fps / MB/s will oscillate/reset and can be misleading. Might be worth either (a) tracking deltas+dt here to compute a real per-second rate, or (b) relabeling these as window values.

[tembo]

On non-macOS/non-Windows this ignores the cpal::StreamInstant and uses Instant::now(). That compiles, but anything relying on stream timing (A/V sync, latency correction, etc.) will silently be wrong on Linux. If this is just a build unblocker, consider making the fallback more explicit (e.g. return an error upstream) or plumbing a relative StreamInstant-based duration instead of an unrelated wall-clock Instant.