Repository navigation
feat(effects): footage() reads the recording at its own resolution - #1396
Conversation
frame() holds the composed picture at the output's resolution, so an effect that magnifies (a loupe, a lens) only magnified blur. footage(p) reads the recording itself at point p of its surface, the space of fx_to_footage, transparent outside 0..1. Each backend paints the content cut into an RGBA copy at the decoder texture's resolution (at most 4096 a side) with one mode-0 draw, the same YUV conversion as the drawn screen, and binds it at t4 / texture(1) / binding 4. analyze() tells from naga's global usage whether fx_fs reads fx_footage; reads_footage() memoises it per source, so an effect that does not read it pays nothing. footage() reads count in the cost tier like frame() reads.
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Shader contract and usage analysis crates/compositor/src/shader_fx.rs, docs/shader-effects.md, electron/mcp/effects-contract.ts, electron/mcp/effects-contract.test.ts, electron/ai-edition/deep-agent/service.ts |
The shader contract adds footage(p), and analysis detects whether an effect reads it. Platform bindings, cost guidance, and effect descriptions include footage reads. |
Backend footage-copy targets crates/compositor/src/compositor_linux.rs, crates/compositor/src/compositor_macos.rs, crates/compositor/src/compositor_windows.rs |
Each compositor caches an RGBA target for the cropped source recording and recreates it when its dimensions change. |
Effect binding and composition crates/compositor/src/compositor_linux.rs, crates/compositor/src/compositor_macos.rs, crates/compositor/src/compositor_windows.rs |
Compositors render a footage copy only when an active effect reads it. They pass the optional texture to image, text, and both-target effects. Tests cover source-resolution sampling, transparent out-of-bounds samples, and skipped copies. |
Priority:
Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Feature
Sequence Diagram(s)
sequenceDiagram
participant Compose as compose_frame
participant Analysis as reads_footage
participant Copy as encode_footage_copy or fill_footage_copy
participant Effect as encode_shader_effect or apply_shader_effect
Compose->>Analysis: Check active effects for footage reads
alt An active effect reads footage
Compose->>Copy: Render or reuse the source-resolution copy
Copy-->>Compose: Return the footage texture
else No active effect reads footage
Compose->>Compose: Keep footage texture absent
end
Compose->>Effect: Pass the optional footage texture
Possibly related issues
- getopenscreen/openscreen issue 1178 requests precise cropping and mentions magnification as a possible solution. This change enables source-resolution sampling for magnification effects but does not add crop controls or a cropping UI.
Possibly related PRs
- getopenscreen/openscreen#1364: Adds an MCP effects-contract recipe that maps UI block coordinates through footage coordinates; this change defines the
footage(p)sampling space used by that recipe.
Merge Risk: 
177f5
-
The change is mergeable after normal checks; no concrete issue requiring a pre-merge fix was established.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 74.36% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 39 functions across 7 files. (1 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Title check The title clearly and concisely describes the main change: adding footage() support for reading the recording at its own resolution. Description check The description includes the required summary, issue context, change type, release impact, platform impact, visual-change note, and detailed testing results. It is complete enough for review, although… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Full details: Description check
-
Explanation
The description includes the required summary, issue context, change type, release impact, platform impact, visual-change note, and detailed testing results. It is complete enough for review, although the referenced screenshots and demo are not included in the pull request.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 74.36% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 39 functions across 7 files. (1 skipped: 1 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Warning
Some tools did not complete. Review the errors below.
🔧 Clippy (1.99.0)
-
Clippy execution failed
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
Summary
Shader effects (Off-Menu) could only read
frame(), the composed picture at the output's resolution. A recording shown smaller than its pixels has lost its detail there, so an effect that magnifies (a loupe, a lens, a close-up) only magnified blur.This adds
footage(p)to the effect contract: the recording itself at pointpof its surface (the space offx_to_footage), at the recording's own resolution, transparent outside 0..1. It holds the recording only: the cursor, webcam and annotations stay inframe().content_cut) into an RGBA copy at the decoder texture's resolution, at most 4096 a side, with one mode-0 draw: the same YUV conversion as the drawn screen, on the model of the depth-of-field pyramid fill. Bound att4(D3D11),texture(1)(Metal), binding 4 (wgpu, an empty texture otherwise).analyze()tells from naga's global usage whetherfx_fsreadsfx_footage;reads_footage()memoises it per source, and the copy is painted only on frames where such an effect is active. An effect that does not read it pays nothing.footage()reads count in the cost tier likeframe()reads.addEffectdescription anddocs/shader-effects.mddescribe it.Related issue
None. Asked for by the maintainer to reproduce Screen Studio's Glass Loupe as an Off-Menu effect.
Type of change
Release impact
Desktop impact
Screenshots / video
A glass loupe written as an effect, exported at 1080p from a 3840×2160 take: read through
frame()the magnified lines are thick and soft; read throughfootage()they are as thin and sharp as in the recording. Frames and the demo are in the maintainer's demo folder (not committed).Testing
cargo test -p openscreen-compositor --lib: 468 passed, 1 failed. The failure,pipeline_windows::tests::playback_off_the_end_of_the_programme_stays_on_the_last_clip, fails the same way withmain's compositor files on this machine; it involves no shader effect.footage()reads each column white or black, nothing outside 0..1, while the composed picture no longer separates them; and an effect that does not readfootage()never paints the copy.footagetests pass (15/15).Rust test (macOS compositor)job; not run locally (no ffmpeg on the test Mac).shader_fxunit tests (cost counting, copy size, FXC compile witht4, MSLtexture(1)),npx vitest --run electron/mcp/effects-contract.test.ts electron/ai-edition/deep-agent(116 passed), bothtscconfigs.addEffectthenexportVideo): it compiled at 26 to 30 reads (medium), and the 4K export of a 14 s project took 8.8 s against 9.2 s for the same loupe onframe().Only the targeted test runs above, not the full
npm run test: CI runs it.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
footage(p), independently of the composed frame.Documentation
footage(p)differs fromframe(uv)and how footage reads affect performance.