Skip to content

feat(infra): transcribe a stored video by id and write its captions - #4134

Merged
richiemcilroy merged 21 commits into
mainfrom
feat/infra-video-captions
Nov 13, 2026
Merged

richiemcilroy merged 21 commits into
mainfrom
feat/infra-video-captions

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

A developer can now transcribe a video their app already stores by sending its ID, and every such transcript produces a WebVTT caption file for that video. This is the transcript half of captions. The player and playback routes are not touched here.

curl -X POST https://cap.so/api/developer/v1/transcripts \
  -H "Authorization: Bearer csk_your_secret_key_here" \
  -H "Content-Type: application/json" \
  -d '{"videoId": "abc123def456"}'

Migration 0057, to ship before merge

packages/database/migrations/0057_developer_transcript_video.sql, additive only. It was generated with drizzle-kit after #4142's 0056, chains its snapshot to 0056, and its journal when is above every entry on main. It was applied only to scratch databases.

ALTER TABLE `developer_transcripts` ADD `videoId` varchar(15);
ALTER TABLE `developer_transcripts` ADD `requestedAt` timestamp(3);
CREATE INDEX `dev_transcripts_app_video_idx` ON `developer_transcripts` (`appId`,`videoId`,`requestedAt`,`id`);

The code selects videoId and requestedAt, so both columns have to exist before this deploys. Every existing row keeps NULL in both and behaves as before. requestedAt is written only for video transcripts and orders their captions to the millisecond.

What a developer gets

  • POST /api/developer/v1/transcripts accepts videoId instead of url. Exactly one of the two: both, or neither, is 400 invalid_request. The body is still strict.
  • The video must belong to the same app, not be deleted and be ready. Otherwise 404 video_not_found (unknown, deleted, or another app's) or 409 video_not_ready (still uploading or processing, or failed with no file to transcribe).
  • The source is the video's stored MP4, read from storage. Nothing is fetched from a public URL and nothing is copied, so the URL size and download limits do not apply. The 4 hour audio limit still does.
  • It costs the transcript price only. The hold, the settle, the no-speech rule and the failure rules are the existing ones. One difference in the hold: when the video's length is known it is held for that length instead of the file size, because a video file's size says little about its audio. Extraction is still capped at what the hold covers and raises the hold if the audio turns out longer.
  • name (defaults to the video's name), language, userId, metadata and Idempotency-Key work as they do for a URL. A replay with a different videoId is refused and names the field.
  • The transcript object gains videoId (null for every other transcript) and source.type is video. The agent API's transcript carries videoId too.

Captions

When a video transcript completes with speech, a WebVTT file is written at developer/<appId>/<videoId>/captions/<language>.vtt.

  • Cues are built from word timings. A cue ends at the end of a sentence or at a pause of more than a second, holds at most two lines of 42 characters and lasts at most 7 seconds. A sentence too long for one cue breaks at a comma when there is one in its second half.
  • Timestamps never overlap or go backwards, including when the word timings themselves do.
  • Fast speech stays on screen long enough to read (50 ms per character), but never past the start of the next cue.
  • &, < and > are escaped, so an arrow in the text cannot end a cue. Control characters and line breaks inside a word become spaces.
  • No file is written when no speech was found.
  • The file always holds the newest completed transcript for that video and language. Newest means requested last, to the millisecond, so two transcripts finishing in either order end the same way. Every write or removal is checked again afterwards and redone if another transcript completed or was deleted meanwhile.
  • Deleting a transcript rewrites the file from the one before it, or removes the file when it was the last.
  • Deleting the video removes the file. The key sits inside the video's folder, which the existing purge in lib/developer-video-purge.ts deletes as a whole, so that code needed no change. A test pins that the key is inside that folder and that an encode cleanup leaves it alone.

Interface for the playback side

apps/web/lib/developer-video-captions.ts

developerVideoCaptions(video: { id: string; appId: string }): Promise<
  { language: string; label: string; key: string }[]
>
  • One SELECT DISTINCT language on the new index. No storage listing.
  • Returns [] for a video with no transcripts.
  • One entry per language, ordered by label. label is the language's English name. key is the storage key of the caption file.
  • A transcript whose language was not reported is listed as und with the label Captions.

Proof

What Result
New and changed tests run against main's code 14 fail in the existing files, and the 4 new test files cannot load
The same on this branch all pass
New unit tests 67: WebVTT writer 15 (golden file from fixed word timings), store and captions 41, workflow with a video source 8, purge 3
Developer, infra and agent unit suites, after merging main 2,179 pass
Whole web suite, after merging main 8,111 pass. The only failures are two files that fail on this machine and pass in CI
Real MySQL scratch database with the migration applied, 3 runs transcripts suite 30 of 30 (7 new), holds suite 7 of 7
CLI unit tests 114 pass
Typecheck and Biome clean, apart from one known unrelated typecheck error

The real database suite includes the existing 23 cases, which never set videoId, so they run with it NULL on every row.

Query plans on the scratch database, in an app with 3,005 transcripts across 60 videos:

Query Plan
developerVideoCaptions ref on dev_transcripts_app_video_idx, 5 rows examined
Newest transcript for a video and language ref on dev_transcripts_app_video_idx, backward index scan, no filesort

Local end to end, no browser: the local web server, a scratch database, the local object store, local ffmpeg and a stub in place of the speech service. A stored video was seeded directly as a ready row with its MP4 in the object store, because the encode pipeline is not part of this change. 25 of 25 checks pass:

  • Both, neither, unknown, another app's, deleted and not-ready videos are refused with the codes above.
  • A 13.2 second video completes, costs 36 micro-credits with one ledger row, and leaves no hold.
  • Nothing is copied to the uploads folder and the video's MP4 is untouched.
  • The caption file exists at the documented key and parses as WebVTT with ordered cues.
  • A second transcript replaces the file. Deleting the newer one falls back to the older, and deleting the last removes the file.
  • A video with no speech completes, is billed and gets no file.
  • Marking the video deleted and running the existing purge removes the caption file with the rest of the folder.

Not covered

  • The speech service was a stub in the local run, so real word timings were not exercised end to end. The writer is tested on fixed timings.
  • The encode pipeline was not run locally. The stored video was seeded.
  • Languages written without spaces between words were not checked against real output. Words are joined with spaces, as transcript segments already are.
  • developerVideoCaptions answers from the database. If a caption write failed after its transcript completed, a file would be listed that does not exist. The write is a workflow step and is retried.
  • The agent API, CLI and MCP cannot create a transcript by videoId yet. They report videoId on transcripts.
  • The console has no "Transcribe" action on a video.
  • Docs cover the transcript sections only. Player behaviour and a captions URL are left to the playback change.

After review

  • A caption removal no longer has the last word unchecked. Every write or removal re-reads the newest source afterwards and goes round again if it changed, so a removal that races another transcript's write ends by writing that transcript.
  • Removing a caption file uses a delete that fails on a storage error, so a delete request fails instead of reporting success.
  • Every delete path updates captions: the REST and console delete, the agent API delete, and a repeated delete of an already deleted transcript, which finishes the update before answering 404. The decision re-reads the row after deletedAt is written, so a transcript that completed during the delete is handled.
  • The hourly transcript cleanup re-syncs captions for video transcripts deleted in the last 3 hours. It pages through all of them with a 2 minute budget, so a failed cleanup is repaired even if the delete is never sent again. Its query uses the (status, updatedAt) index.
  • The migration now also adds requestedAt and indexes it in place of createdAt (SQL above). It has only run on dropped scratch databases.
  • After the merge: developer, infra and agent unit suites 2,198 pass. Real MySQL scratch database with the new migration, 3 runs: transcripts suite 32 of 32, holds suite 7 of 7.
  • The captions sentence in the docs and OpenAPI no longer says the hosted player shows them, since it does not yet. It now says what ships: when a transcript of a video completes with speech, Cap writes a WebVTT captions file for the video in that language, from the newest completed transcript. A docs test fails if the transcript docs mention a player or a captions path.
  • Merged main after fix(infra): charge transcripts deleted during speech and cap running transcripts per app #4143: creating a transcript by videoId is held to the ceiling of 20 queued or processing transcripts per app, like URL and upload transcripts. A repeat of an admitted request is still let through.
  • Merged main after perf(web): cut Cap Infra billing reads on the console, usage API, ledger and storage cron #4142, which took 0056, and renumbered this migration to 0057 (SQL above). packages/database/migrate.ts on an empty scratch database applies all 58 migrations, and a second run applies nothing.
  • After these merges: developer, infra and agent unit suites 2,503 pass. Real MySQL, 2 runs: transcripts and holds suites 46 of 46.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge once migration `0057` has been applied.

Summary

Adds transcription by stored videoId and writes WebVTT captions from completed transcripts.

  • Developers can start a transcript from a ready video stored in their app.
  • Completed video transcripts keep a WebVTT caption file for each language.

richiemcilroy acknowledges that a failed caption write can leave a listed file missing and states that the workflow step retries the write.

Reviews (7) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 3 potential issues (3 medium).

View 1 other finding in Capy.

Open in Capy

@sandchest-com

sandchest-com Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest reviewed 044f519: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 7 credits · Review details

@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Ignored Ignored Preview Nov 13, 2026 5:22am UTC

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 2 other findings in Capy.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 02:06 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

View 2 other findings in Capy.

Open in Capy

…ions

# Conflicts:
#	apps/web/workflows/developer-transcribe.ts
#	packages/database/migrations/meta/0056_snapshot.json
#	packages/database/migrations/meta/_journal.json
#	packages/database/schema.ts
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

This branch was successfully deployed

1 active and 1 inactive (outdated) deployments
Preview – cap-web — 044f519b Deployed Nov 13, 2026 by vercel[bot]
Preview – cap-status — c821549d Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants