Skip to content

fix(export): WebCodecs video decode and full audio mixdown#122

Merged
dazzatronus merged 1 commit into
mainfrom
derk/eng-668-rebuild-studio-sdk-client-side-video-export-audio-video
Jun 5, 2026
Merged

fix(export): WebCodecs video decode and full audio mixdown#122
dazzatronus merged 1 commit into
mainfrom
derk/eng-668-rebuild-studio-sdk-client-side-video-export-audio-video

Conversation

@dazzatronus

@dazzatronus dazzatronus commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

What

Rebuilds the SDK's client-side video export, which produced broken output: video froze on its first frame and stuttered, audio was truncated to ~2% of its length and mis-placed, and audio embedded in video clips was dropped (only standalone audio clips were encoded).

Changes

  • Audio — replace the per-clip AudioSampleSource (which can't mix) with an OfflineAudioContext mixdown. Every audio-bearing clip (audio + video + text-to-speech) is scheduled with per-clip gain automation (volume + fadeIn/fadeOut) and rendered to one AudioBuffer, encoded via AudioBufferSource. Fixes the seconds/ms timing bugs (length, placement, trim) and an AudioSample leak; overlap is summed, gaps are silence, differing sample rates resample natively.
  • Video — decode source frames with mediabunny Input + CanvasSink (WebCodecs) instead of seeking an HTMLVideoElement. Frame-accurate, no stutter, and decodes in browsers that can encode but can't <video>-play the source codec. Adds a canDecode() capability gate; removes the per-frame texture churn and a GPU texture leak.
  • Tests — pure-function unit tests for the timing and volume-automation math.

Testing

  • tsc, eslint, and jest (1801) pass.
  • e2e: exported an edit covering 15 asset types (validated against the Edit API with shotstack validate); all types export, audio mixes correctly (video soundtrack + music), and source frames decode 125/125.

@dazzatronus dazzatronus force-pushed the derk/eng-668-rebuild-studio-sdk-client-side-video-export-audio-video branch from 0eeb0e2 to ad43ed2 Compare June 5, 2026 11:11
@dazzatronus dazzatronus merged commit 1ac250a into main Jun 5, 2026
1 check passed
@dazzatronus dazzatronus deleted the derk/eng-668-rebuild-studio-sdk-client-side-video-export-audio-video branch June 5, 2026 11:15
@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 2.11.4 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant