Skip to content

fix(web): Infra request limits per account and address, retire empty uploads, wait out a busy farm - #4160

Merged
richiemcilroy merged 14 commits into
mainfrom
fix/infra-encode-fairness
Nov 13, 2026
Merged

richiemcilroy merged 14 commits into
mainfrom
fix/infra-encode-fairness

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Three Cap Infra changes for launch: API request limits per account and per address, retirement of uploads that never got a file, and encodes that wait out a busy farm instead of timing out. All of it runs only on the Infra API routes, developer_videos rows and the Infra encode workflow.

Request limits

  • Per key. A secret key gets 300 requests a minute, up from 60.
  • Per account. All the secret keys of one account together get 1,200 requests a minute (new firewall rule rl_developer_api_owner, keyed by owner id). This only bites an account spreading load over more than four keys.
  • Per address, for credentials that were not accepted. A well formed csk_ key that matches nothing, or an upload token whose signature fails, counts against the sending address (new rule rl_developer_auth_failure, 30 a minute per IP). Past that the address gets 429 until the minute ends. A key this instance accepted in the last minute still gets through, so a customer behind a shared address keeps working.
  • Rejected credentials no longer spend a key's budget or a firewall call, and the key lookup now happens once per request (the rate limiter sets the app and owner, and withDeveloperSecretAuth reuses them).
  • Both paths that accept a secret key set the same variables, including developerKeyId from fix(infra): bind upload tokens to their key, cap part numbers, sniff past ID3 tags #4138, so upload tokens minted after the rate limiter's lookup still name their key and stop working when it is revoked. withDeveloperSecretAuth skips its own lookup only when an earlier middleware accepted a secret key (developerKeyType === "secret"), not whenever an app id is set, so a router that stacks the upload token middleware first cannot skip the secret key check.

How these relate to the firewall rules:

Rule Key Limit Change
rl_developer_api_key secret key 300 / min raised from 60; now counted only for accepted keys
rl_developer_upload_token upload token 1,200 / min unchanged; now counted only for valid tokens
rl_developer_app_create, rl_developer_key_rotate user 10 / hour unchanged
rl_developer_api_owner owner id 1,200 / min new, sits above the per-key limit
rl_developer_auth_failure IP 30 / min new, counts only rejected credentials

None of them count the same request twice against the same budget. App create and key rotation limit how fast an account can mint keys; the owner limit caps what those keys can do together. The firewall rules for rl_developer_api_key and rl_developer_api_owner move to 300 and 1,200 when this deploys, so the headers and the firewall agree. Where a rule cannot answer, each server falls back to its own window with the same numbers. The REST docs, the get-started guide, the OpenAPI description and the agent setup prompt state the new numbers, and the tests derive them from DEVELOPER_KEY_REQUESTS_PER_MINUTE and DEVELOPER_OWNER_REQUESTS_PER_MINUTE rather than repeating them.

Retiring uploads that never received a file

The daily sweep already retired day-old Infra videos whose file was never recorded, but only when the file existed and was old. A video whose upload never sent a byte stayed listed for good. Now:

  • file present and over a day old: retired and removed, as before
  • no file and no open multipart upload for its key: retired (nothing to remove)
  • an upload still open, or storage cannot say for certain: left alone

Before it retires a video with no file, the sweep locks the row, skips it if a completion claimed it in the last 15 minutes, and asks storage again. Storage is read as file, then open uploads, then file once more, so an upload assembled between the first two reads still shows. Initiate opens uploads under the same row lock, and completion can only assemble a file from an upload that is still open. A claim left behind by a completion that died before assembling keeps the video only until that claim goes stale.

Waiting out a busy farm

Since #4151 the media server keeps a slot on each worker for Cap's own recordings and runs at most one Infra 1440p or 2160p ladder per worker, so Infra jobs get busy answers more often. The workflow already retried a busy answer every 20 seconds, but that wait counted toward the three hour processing deadline. A batch of 4K uploads could fail with "Processing took too long" without ever reaching an encoder. The deadline now restarts whenever every dispatch answered busy and no output of the video is encoding. A running output still counts toward the deadline, as before.

The six-hour rule that restarts a stuck encode when its status is read now runs from the later of the encode's start and its last busy answer (busyAt in the processing JSON, no schema change). A video that waited five hours and then started encoding is not restarted while it encodes. If its workflow stops, the busy answers stop too, and the restart happens six hours later as before.

Dropped from the original branch

  • Farm-wide encode queue with fairness between owners. It held Infra to a fixed number of encodes across the farm and ordered waiters by owner. fix(media-server): keep room on each worker for Cap's own recordings #4151 now enforces Infra's share on each worker, which is the real limit, so a second fixed farm-wide cap would only leave capacity idle. Sharing slots out by owner is the kind of per-account encode treatment that was decided against. It also read every processing row on each check, which would load the database. The one gap it covered, waiting instead of failing when the farm is full, is the narrow deadline change above.
  • Faststart remux retry. Ships separately.

Regular Cap

Nothing regular Cap runs changes. The only shared file is lib/rate-limit.ts, which gains two rule ids and one fallback entry; its tests pass unchanged. The middleware in app/api/utils.ts is mounted only on /api/developer/v1 and /api/developer/sdk/v1.

Tests

  • vitest on the Infra workflow, processing, state, player, SDK upload, SDK upload billing, public key refusal and API contract suites: all pass. Each new test fails without its fix: waiting out a busy farm, the restart clock following busyAt, busyAt being recorded, and races in the sweep (a recent completion claim, an upload assembled between storage reads, and an upload that opens before the row lock). Another test shows that a video with one output running and one busy still stops at three hours.
  • rate-limit.test.ts and save-video-edits-rate-limit.test.ts: 24 passed.
  • tsc --noEmit in apps/web (after tsc -b on its references): clean apart from one existing error in organization-settings-ownership.test.ts, a file this PR does not touch.

RetriggerView in GreptileConfidence Score: 4/5

Separate connection failures from busy answers before merging so uploads retain a timeout during media-server outages.

Findings

  1. P1 Outages leave uploads waiting forever ▶
Fix with agent prompt
### Issue 1
apps/web/workflows/process-developer-video.ts:109-115
When the media server refuses connections or DNS fails, `dispatchDeveloperVideoPart` returns `"busy"` just as it does when workers are full. This new reset therefore extends the deadline on every retry during an outage. Those retries also refresh `busyAt`, so the video can stay `processing` indefinitely instead of reaching the existing timeout.

Keep connection failures separate from busy answers and preserve a timeout for them.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds account-wide and rejected-credential request limits, retires abandoned uploads with no file, and lets encodes wait for worker capacity.

  • Infra API requests share limits across an account and block repeated bad credentials by address.
  • The daily sweep retires old Infra videos that never received a file.
  • Infra encodes wait through busy periods without spending their processing deadline.

Reviews (13) · Last reviewed commit: "Merge branch 'main' into fix/infra-encod..." · Reviewed by Greptile

xxxxxxxxxxxxx and others added 3 commits November 13, 2026 00:05
…ad of timing them out

A busy answer from every worker while no output of the video is encoding no longer counts toward the three hour processing deadline. An output that is running still does.

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 2 potential issues (2 medium).

View 1 other finding in Capy.

Open in Capy

Comment thread apps/web/lib/infra-video-upload.ts Resolved
Comment thread apps/web/lib/infra-video-upload.ts Resolved
@sandchest-com

sandchest-com Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest reviewed e50fa2d: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 13 credits · Review details

@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Skipped Skipped Nov 13, 2026 8:35am UTC

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 05:23 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

View 1 other finding in Capy.

Open in Capy

Comment thread apps/web/lib/infra-video-upload.ts Resolved
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 05:28 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

Open in Capy

Comment thread apps/web/lib/infra-video-upload.ts Resolved
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 05:34 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 2 other findings in Capy.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 06:01 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

View 3 other findings in Capy.

Open in Capy

Comment thread apps/web/lib/rate-limit.ts Resolved

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

…ness

# Conflicts:
#	apps/web/app/api/utils.ts
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

Comment thread apps/web/app/api/utils.ts Resolved
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 06:37 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

…ness

# Conflicts:
#	apps/web/app/api/utils.ts
#	apps/web/data/infra-agent-prompt.ts
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 08:11 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 08:35 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

@richiemcilroy
richiemcilroy merged commit fb59fa2 into main Nov 13, 2026
40 checks passed
Comment on lines +109 to +115
if (
dispatched.length > 0 &&
dispatched.every((answer) => answer === "busy") &&
silent.length === 0 &&
!waiting
) {
deadline = Date.now() + PROCESSING_DEADLINE_MS;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Outages leave uploads waiting forever

When the media server refuses connections or DNS fails, dispatchDeveloperVideoPart returns "busy" just as it does when workers are full. This new reset therefore extends the deadline on every retry during an outage. Those retries also refresh busyAt, so the video can stay processing indefinitely instead of reaching the existing timeout.

Keep connection failures separate from busy answers and preserve a timeout for them.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/web/workflows/process-developer-video.ts
Line: 109-115

Comment:
**Outages leave uploads waiting forever**

When the media server refuses connections or DNS fails, `dispatchDeveloperVideoPart` returns `"busy"` just as it does when workers are full. This new reset therefore extends the deadline on every retry during an outage. Those retries also refresh `busyAt`, so the video can stay `processing` indefinitely instead of reaching the existing timeout.

Keep connection failures separate from busy answers and preserve a timeout for them.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

This branch was successfully deployed

1 active and 1 inactive deployments
Preview – cap-web — e50fa2dc Deployed Nov 13, 2026 by vercel[bot]
Preview – cap-status — e50fa2dc Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants