Skip to content

Infra: send video.ready once HLS is ready - #4200

Merged
richiemcilroy merged 7 commits into
mainfrom
fix/infra-ready-after-hls
Nov 13, 2026
Merged

richiemcilroy merged 7 commits into
mainfrom
fix/infra-ready-after-hls

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

What

video.ready now fires once the video is ready and every output has settled (complete, failed, or unavailable on the media server), so its body carries every playback URL that exists. hlsUrl is set unless HLS failed. Before this, the event fired on the status change to ready, which happens when the first output (usually the MP4) finishes, so every video.ready carried hlsUrl: null, and retries re-sent that stale body.

Also in this PR (docs and error wording, no behaviour change beyond the messages):

  • Webhooks guide: new ready timing, a complete node:http receiver (run by a test), the Node example in 2-space indent, and a note that a second DELETE of an endpoint returns 200.
  • Get Started: step 4 links Webhooks; the hosted-player paragraph points look and behaviour at Embed Options, Embed Parameters and Player Events, and says to build your own player only for full control of the UI.
  • Playlist order: Get Started and the REST reference said "lists a rendition that starts quickly first". Since perf(web): start Cap Infra playback on a mid rendition and compress playlists #4097, the master playlist leads with the largest rendition whose peak bitrate is at most 2.6 Mbps, at 1080p or smaller (720p on most videos). The docs now say that.
  • Webhook errors: docs points at /docs/api/webhooks for webhook error codes; unknown and missing events get hand-written messages.

How

The decision is made in mutateDeveloperVideo, inside the transaction that holds the video's row lock, by the pure developerVideoEventStep(previous, next, now):

Write Result
Status becomes ready, all outputs settled video.ready now
Status becomes ready, an output still encoding No event. processing.readyEventDueAt = now is written with the change
Video ready, marked, the write settles the last output video.ready, and the mark is removed in the same write
Video ready, marked, 30 minutes since the mark video.ready with the URLs that exist, mark removed
Video ready, no mark Nothing, ever
Status becomes errored video.errored, as before

Because every writer takes the row lock, exactly one write can remove the mark. enqueueDeveloperVideoEvent builds the body from the committed row after that transaction ends, and also refuses video.ready while the committed row still carries the mark, so a rolled-back write sends nothing. The body is built when the event fires (now after HLS), and the delivery workflow re-sends that same body and event id on every retry.

Stuck HLS backstop. The processing workflow keeps polling a ready video while any output is pending or running (every 20 s at most). readDeveloperVideoParts, which each poll calls, now sends a due video.ready. The bound: video.ready arrives at most 30 minutes plus one poll (20 s) after the video became ready, with hlsUrl: null if HLS is still encoding. Independently, the workflow's existing 3 h deadline and its crash handler both mark unfinished outputs failed, which settles them and fires the event. The final poll also sends a marked video whose last output was settled by a write that did not send (for example by the previous deployment during rollout). A failure there is logged and retried on the next poll, never thrown, so it cannot fail the video.

Videos ready before deploy. Old code fired at the status change and never wrote the mark. A ready video without the mark never fires again, whatever update arrives later. That covers videos announced by old code and videos that became ready before webhooks existed.

The mark lives in the processing JSON column. No migration. REST status and Get Video are unchanged: a polling client can still read ready with hlsUrl: null. Regular Cap videos are not touched: this is only the developer video path.

Proof

Each condition the launch coordinator set, with its test in developer-webhook-dispatch.test.ts (real processing code over the fake row-locked database):

Condition Test
HLS never settles: fires once, hlsUrl null, within the bound is sent once, with hlsUrl null, when HLS never settles: no event at 29:59 after ready, one at 30:00 with hlsUrl: null, none when HLS completes later
Processing deadline on a stuck job is sent with hlsUrl null when processing times out on a stuck HLS job
MP4 or status first waits for HLS when the MP4 makes the video ready first
HLS first waits for the MP4 when HLS makes the video ready first
Both in the same tick queues one event carrying both URLs when both outputs finish at once (concurrent) and queues one event when both outputs settle in one update
Repeat updates queues nothing more when the encoder's callbacks arrive again
Retried delivery, same id and body re-sends the same event id and body on every retry: runs the real delivery workflow through retry, retry, delivered; all three attempts get the identical job with hlsUrl set
HLS failed is sent with hlsUrl null when HLS fails
Ready on old code, event already sent, then an HLS update sends no second event for a video that was ready before this change: HLS completes, then a poll 60 minutes later; no endpoint lookup, no event
Ready before webhooks existed sends no event for a video that was ready before webhooks existed: unavailable HLS is retried and completes; no event
Mark settled by a write that did not send is sent on the next poll when the last output was settled without sending
Rolled-back announcement is not sent while the committed row still carries the mark
Errored unchanged existing video.errored tests, plus the pure-step tests

Mutation check: with the poll backstop removed, the never-settles test fails; with the mark check removed, both pre-deploy tests fail; with the old fire-at-flip rule restored, 8 tests fail; with the committed-mark guard removed, the rollback test fails.

Docs: developer-webhook-docs.test.ts now extracts the receiver from the guide, runs it with node, and checks that a forged signature gets 400, Cap's signature gets 200, and a duplicate event id is handled once. It also checks the guide and OpenAPI state the 30-minute bound from the constant, and that no code block uses tab indents.

Review follow-ups: a failed send on the workflow's final poll now reports finished parts as waiting so the workflow polls again (keeps the workflow polling when sending a due event fails); the docs receiver refuses bodies over 1 MB with 413 (tested with a 2 MB POST); a non-string event no longer reaches String() in the error message (tested with {"toString": null}, 400).

Production baseline before this change, using the proof script that will run after merge (one endpoint behind a tunnel, one 12 s 1080p upload, Get Video Status polled every second): the video read ready with hlsUrl null, video.ready arrived with hlsUrl: null, and hlsUrl appeared on Get Video Status 0.6 s after the event. After merge the same script requires one video.ready with hlsUrl set and no second event.

Tests

  • Related suites (developer-*, deliver-developer-*, process-developer-*, OpenAPI, docs, API contracts, infra-*, media-server progress callback, media processing workflows): 140 files, all pass.
  • Whole apps/web unit suite: 9,519 passed vs 9,499 on main (+20 new), 7 failed on both, all in share-stage-identity.test.ts, which this PR does not touch.
  • tsc --noEmit on apps/web: no new errors (one existing error in organization-settings-ownership.test.ts, untouched).
  • workflow build --target standalone: builds; the flow bundle contains no drizzle-orm or @cap/database runtime.

Production proof after deploy

Run on production after the merge commit deployed, with the same script as the baseline (real endpoint on a tunnel, one upload through the documented flow):

Check Result
Exactly one video.ready pass
Signature valid pass
hlsUrl set in the event pass
mp4Url set pass
Event id header matches body pass
No video.errored pass

Timings: upload complete 36.93 s, status ready with hlsUrl null 38.21 s, video.ready received with hlsUrl set 40.34 s. No second event in the following 30 s. The endpoint and video were deleted after the run.

Richie McIlroy added 3 commits November 13, 2026 06:27
video.ready fired on the status change to ready, which happens when the
first output (usually the MP4) finishes, so every event carried hlsUrl null.

The event is now decided inside the locked status write. A video that
becomes ready with an output still encoding gets a readyEventDueAt mark in
its processing state, and the write that settles the last output clears it
and sends the event. A video already announced, or ready before this change,
never carries the mark and sends nothing more. The processing workflow's poll
sends it after 30 minutes if an output never settles.
Unknown or missing events get hand-written messages instead of zod's
wording, and errors with a webhook code link to the webhooks guide.
The webhooks guide says video.ready waits for every playback URL, adds a
runnable node:http receiver (tested), uses 2 space indents, and says a
second DELETE returns 200. Get Started links webhooks from step 4, points
look and behaviour at the embed options, and the playlist start sentence
states the rule from #4097 (720p on most videos).
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai please review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

View 1 other finding in Capy.

Open in Capy

Comment thread apps/web/content/docs/api/webhooks.mdx Resolved
Richie McIlroy added 3 commits November 13, 2026 06:34
A failed send on the final poll reported both parts done, so the workflow
ended with the event unsent. Finished parts now read as waiting, and the
next poll tries again. The hold-up tests now finish HLS so they reach
dispatch.
String() on a caller's object can throw, which turned a bad event into a
500. A value that is not a string is no longer quoted.
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Ignored Ignored Preview Nov 13, 2026 12:09pm UTC

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 11:34 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai please review again

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

@sandchest-com

sandchest-com Bot commented Nov 13, 2026

Copy link
Copy Markdown

Sandchest

Sandchest reviewed 70c330a: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · Review details

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

@richiemcilroy
richiemcilroy merged commit a374cd1 into main Nov 13, 2026
40 checks passed

This branch was successfully deployed

1 active and 1 inactive (outdated) deployments
Preview – cap-web — 70c330a1 Deployed Nov 13, 2026 by vercel[bot]
Preview – cap-status — 6669a20d Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant