Repository navigation
fix(player): stop the preview freezing when many clips share one long recording - #4
Draft
robinebers wants to merge 20 commits into
Draft
robinebers wants to merge 20 commits into
robinebers wants to merge 20 commits into
Conversation
…view shows it (heygen-com#7723) * fix(core): at full quality a 4K clip plays a copy at the size the preview shows it * fix(core): names carry the copy rule, not comments * fix(core): a clip whose copy fails is left held as it was, so its decoder is still freed when idle * fix(core): the copy hold lives with the copy code, keeping core inside its size budget
…or's fault (heygen-com#7730) * fix(engine): an unrecognised capture failure is unknown, not the author's fault * fix(engine): carry the extractor's disk-full words in a name, not a comment
…d leaves it unchanged (heygen-com#7731) * fix(studio-server): an edit to a read-only project file is refused and leaves it unchanged * test(core): read the refused file without a stat check first
…ygen-com#7733) * fix(cli): check times a line held across a split clip as one hold A clip split in two hands its line to a second element at the same place. line_unread timed each element alone, so the first half warned while the same text stayed on screen. * refactor(cli): name the split-clip line hand-off instead of commenting it
… from finding projects (heygen-com#7734) * feat(core): a folder with index.html is one a step can work on, apart from finding projects isHyperframesProject needs index.html and a marker file, which is right for finding projects on a computer (a website's folder must not be listed) but not for what a step works on. isCompositionFolder says that: any folder whose index.html is the composition. * build(core): raise core's unpacked size budget for the step folder rule
… as the film's warnings (heygen-com#7736) check copies each canvas into a 2D context to see whether it moved. On WebGL that readback makes Chrome log a GL driver Performance note, which check then listed as a console warning from the composition.
… the word ends (heygen-com#7735) * fix(registry): caption-highlight clears a short word's highlight when the word ends On a word shorter than 0.05 s the 0.15 s fade-in outlived the fade-out, so a frame inside that tail lit the highlight again until the phrase ended. * fix(registry): leave the generated caption-highlight catalog files to the generator
…7740) The error only said to run --help, so an agent guessed another flag. It now names every long flag the command declares, and --search points to --query.
…com#7745) The filter for check's own readback stalls matched every GL driver Performance note. It now matches only the ReadPixels stall.
…spect ratio (heygen-com#7739) * fix(studio): HEVC with long-term reference pictures keeps its pixel aspect ratio * style(studio): format the HEVC SPS test
…eygen-com#7737) * fix(studio-server): check and render keep their history on a drive without hard links The history lock had its own list of the errors that mean a drive cannot hard-link, and it lacked EISDIR, which Windows returns on a Google Drive folder. Every check report and render record there failed. The lock now uses core's list, the one owner, which also gains ENOSYS from the lock. * fix(engine): downloads and plan blobs publish on a drive without hard links The producer's plan publisher kept a third list of no-hard-link errors without EISDIR, and the downloader had no fallback, so both failed on a Google Drive folder. Both now use core's cannotHardLink; the downloader copies into an exclusively created file, so it still never overwrites. * fix(engine): a download on a drive without hard links publishes atomically Instead of copying into an exclusively created file, the fallback renames the staged download into place under the cache path lock when nothing is there yet, so a killed run never leaves a truncated file to reuse.
…gen-com#7746) * fix(cli): an unknown flag's list leaves out hidden rename shims The list pointed agents at init's --template, --video-legacy and --agent, which only print an error. It now shows the flags --help shows. * fix(cli): an unknown flag never suggests a hidden rename shim The did-you-mean search still used every spelling, so '--templat' on init was pointed at the hidden --template. It now uses the same visible flags the list shows.
…ng the app (heygen-com#7744) * fix(studio-server): save an edit Windows briefly holds without freezing the app * fix(studio-server): keep later edits behind a held one, one backup per replay * fix(studio-server): send a held cut again, and keep the sync retry for rollbacks
…into the export once (heygen-com#7747) * fix(producer): a clip whose file name has a space or accent is mixed into the export once * fix(producer): compare a clip's src decoded, keeping the authored spelling either way
Releasing a clip dropped its src, called load() and set the src again.
Firefox treats preload="none" as "metadata" while an explicit load() is
pending, so every release fetched the clip's metadata and first frame. The
runtime then saw a far clip loading and released it again, in a loop: about
20,700 media requests over one 2:16 film, with 2-4 s stalls at cuts.
Setting src already runs the media load algorithm, which ends the fetch, so
the release now sets it again without load(). Chrome and Safari behave as
before: Blink clears its ignore-preload-none flag when src is set, and
WebKit's src set cancels the prepareToPlay that load() queued.
To reproduce: in Firefox, give a <video> with a src preload="none", then call
removeAttribute("src"), load() and setAttribute("src", src) in one task. It
loads to readyState 4; with the src set in a later task it stays at 0.
…me and Firefox A film whose clips are cut from one or two long recordings froze a few seconds in, in Chrome and Firefox. Both keep a paused clip's request open with its unread bytes, and every request to one host shares one HTTP/2 or HTTP/3 connection window. With every clip of the file loading at parse, those idle streams fill the window, and the request for the clip coming up is never answered. The player's frame now holds clips back on every engine, as it already did on WebKit: each file keeps the clip on screen and the next one loading, and frees the rest. The player also names its frame, because a composition in an opaque sandbox cannot read frameElement to find the player's mark. To reproduce: play a film of 37 clips cut from a 363 MB and a 180 MB .mov served over HTTP/3 in <hyperframes-player>. On main it stops at 0:17 in Chrome and at 0:09 in Firefox and never recovers.
Naming the player's frame adds 44 bytes to the packed player (158,706 to 158,750 bytes with pnpm 10.17.1). main was already 10 bytes under the 1% tolerance of its 157,145-byte budget, so any growth fails the check. The budget moves to 5% above the new measured size, as the check suggests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A film whose clips are cut from one or two long recordings froze a few seconds into playback in the preview player in Chrome and Firefox. A typical case is 37 short clips taken from a camera file and a screen recording. The picture and the time stopped, and only a reload brought them back.
What
The player now asks each recording only for what it needs: the clip on screen and the next one. It lets go of the rest until they come up. Safari has worked this way since heygen-com#7012, and this does the same in Chrome and Firefox. Films whose clips come from separate files play as before, and rendering and export are unchanged.
Before
Chrome,
main. The film stops at 0:17.4 while the wall clock keeps counting. The picture is pixelated on purpose, and the film and wall clocks are in the top bar. Each frame is one second of real time, played at 4× speed.Watch the Before recording (MP4, 15 s)
After
Chrome, this branch. The same film plays to the end at 2:16, with the film clock keeping pace with the wall clock.
Watch the After recording (MP4, 35 s)
Test plan
The test film is 2:16 long, with 37 clips cut from two .mov files (363 MB and 180 MB) served from a CDN over HTTP/3. Each run plays it headless in
<hyperframes-player>and records where the film time stops moving.main, opaque sandboxSeeking in Chrome. A paused seek to 0:20, 1:00 and 1:40 shows the right frame each time. Playing after a seek to 0:10, 0:17, 0:33, 1:00, 1:30 and 2:05 keeps moving. On
main, the paused seek to 1:00 never showed its frame, and 4 of the 6 plays after a seek stayed stuck.Requests in Firefox. Firefox made about 100 media requests over the film. With clips held back but without the release fix described below, it made 20,687 requests and stalled for 2–4 s at cuts.
Pages outside the player. A composition page opened on its own still loads every clip at once and still stops on this film, at 0:15 in Chrome and 0:20 in Firefox. This PR changes only pages inside the player and Studio's preview.
Upstream's playbench. This ran in Chrome, 5 runs per arm with
--audio sink, againstmain. Its server is local HTTP/1.1, so it measures timing at cuts, not the connection window. fix(core): iPhone Safari loads the first take before the clips behind it heygen-com/hyperframes#7001 saw a cost to holding clips back; with this rule, playbench shows none:main→ this branchmain→ this branchplay()(median)Audio fade landings stay within
main's spread on both edits.Unit tests.
entry.test.tsnow runs the per-file rule in Chrome as well as Safari, and covers a frame the composition cannot read.preloadMedia.test.tschecks that a release never callsload(), andhyperframes-player.test.tschecks the frame's name. The core, runtime CI and player suites pass locally.Unit tests added/updated
Manual testing performed
Documentation updated (if applicable): no docs describe this behaviour
Comments follow CONTRIBUTING.md "Comments": they say why, not what, and a bug fix says what the code must do and how to reproduce the bug
Related work
Refs heygen-com#7012, which added the per-file rule for WebKit only. Refs heygen-com#7001, closed, where holding clips back in Chromium measured worse with Studio's look-ahead rule; this PR uses the rule from heygen-com#7012 instead. heygen-com#6029 and heygen-com#6033 are about a different limit, HTTP/1.1 servers running out of connections.
How
inPlayerFrameno longer requires WebKit. Player frames now get that per-file rule on every engine: each file keeps the clip on screen and the next take loading, and frees the rest. Studio's look-ahead rule was not enough. In Chrome it still froze 1 of 2 runs, because the clips it kept for scrubbing back held the window.hyperframes-player. A composition in an opaque sandbox cannot readframeElement, soinPlayerFramealso acceptswindow.name.releaseMediasets thesrcagain without callingload(). While an explicitload()is pending, Firefox treatspreload="none"as metadata (mIsDoingExplicitLoad, cleared when resource selection runs). So every release fetched metadata and a first frame, and the runtime then released the far clip again. Settingsrcruns the load algorithm and ends the fetch on every engine. Chrome and Safari behave as before: Blink resetsignore_preload_none_whensrcis set, and WebKit'ssrcset cancels theprepareToPlaythatload()queued.mainwas already 10 bytes under the 1% tolerance of the player's budget, so this raises the packed budget to 5% above the new size, as the check suggests. feat(player): add customizable branding to the preview player heygen-com/hyperframes#7718, fix(player): keep the video playing when you jump to another moment heygen-com/hyperframes#7719 and feat(player): add a fullscreen button to the preview player heygen-com/hyperframes#7726 raise the same line, so whichever merges second needs a fresh measurement. The runtime grows by 80 bytes.