Repository navigation
fix(web): Infra request limits per account and address, retire empty uploads, wait out a busy farm - #4160
Conversation
…tials per address
…ad of timing them out A busy answer from every worker while no output of the video is encoding no longer counts toward the three hour processing deadline. An output that is running still does.
|
Sandchest reviewed Sandchest reviews every change for vulnerabilities an attacker could use · 13 credits · Review details |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
hey @greptileai, please re-review the PR |
…ion left its claim
|
hey @greptileai, please re-review the PR |
…ent claims, before retiring an Infra video
|
hey @greptileai, please re-review the PR |
…nt limit to 1,200 a minute
|
hey @greptileai, please re-review the PR |
…ness # Conflicts: # apps/web/app/api/utils.ts
|
hey @greptileai, please re-review the PR |
|
hey @greptileai, please re-review the PR |
…ness # Conflicts: # apps/web/app/api/utils.ts # apps/web/data/infra-agent-prompt.ts
|
hey @greptileai, please re-review the PR |
| if ( | ||
| dispatched.length > 0 && | ||
| dispatched.every((answer) => answer === "busy") && | ||
| silent.length === 0 && | ||
| !waiting | ||
| ) { | ||
| deadline = Date.now() + PROCESSING_DEADLINE_MS; |
There was a problem hiding this comment.
Outages leave uploads waiting forever
When the media server refuses connections or DNS fails, dispatchDeveloperVideoPart returns "busy" just as it does when workers are full. This new reset therefore extends the deadline on every retry during an outage. Those retries also refresh busyAt, so the video can stay processing indefinitely instead of reaching the existing timeout.
Keep connection failures separate from busy answers and preserve a timeout for them.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/web/workflows/process-developer-video.ts
Line: 109-115
Comment:
**Outages leave uploads waiting forever**
When the media server refuses connections or DNS fails, `dispatchDeveloperVideoPart` returns `"busy"` just as it does when workers are full. This new reset therefore extends the deadline on every retry during an outage. Those retries also refresh `busyAt`, so the video can stay `processing` indefinitely instead of reaching the existing timeout.
Keep connection failures separate from busy answers and preserve a timeout for them.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Three Cap Infra changes for launch: API request limits per account and per address, retirement of uploads that never got a file, and encodes that wait out a busy farm instead of timing out. All of it runs only on the Infra API routes,
developer_videosrows and the Infra encode workflow.Request limits
rl_developer_api_owner, keyed by owner id). This only bites an account spreading load over more than four keys.csk_key that matches nothing, or an upload token whose signature fails, counts against the sending address (new rulerl_developer_auth_failure, 30 a minute per IP). Past that the address gets 429 until the minute ends. A key this instance accepted in the last minute still gets through, so a customer behind a shared address keeps working.withDeveloperSecretAuthreuses them).developerKeyIdfrom fix(infra): bind upload tokens to their key, cap part numbers, sniff past ID3 tags #4138, so upload tokens minted after the rate limiter's lookup still name their key and stop working when it is revoked.withDeveloperSecretAuthskips its own lookup only when an earlier middleware accepted a secret key (developerKeyType === "secret"), not whenever an app id is set, so a router that stacks the upload token middleware first cannot skip the secret key check.How these relate to the firewall rules:
rl_developer_api_keyrl_developer_upload_tokenrl_developer_app_create,rl_developer_key_rotaterl_developer_api_ownerrl_developer_auth_failureNone of them count the same request twice against the same budget. App create and key rotation limit how fast an account can mint keys; the owner limit caps what those keys can do together. The firewall rules for
rl_developer_api_keyandrl_developer_api_ownermove to 300 and 1,200 when this deploys, so the headers and the firewall agree. Where a rule cannot answer, each server falls back to its own window with the same numbers. The REST docs, the get-started guide, the OpenAPI description and the agent setup prompt state the new numbers, and the tests derive them fromDEVELOPER_KEY_REQUESTS_PER_MINUTEandDEVELOPER_OWNER_REQUESTS_PER_MINUTErather than repeating them.Retiring uploads that never received a file
The daily sweep already retired day-old Infra videos whose file was never recorded, but only when the file existed and was old. A video whose upload never sent a byte stayed listed for good. Now:
Before it retires a video with no file, the sweep locks the row, skips it if a completion claimed it in the last 15 minutes, and asks storage again. Storage is read as file, then open uploads, then file once more, so an upload assembled between the first two reads still shows. Initiate opens uploads under the same row lock, and completion can only assemble a file from an upload that is still open. A claim left behind by a completion that died before assembling keeps the video only until that claim goes stale.
Waiting out a busy farm
Since #4151 the media server keeps a slot on each worker for Cap's own recordings and runs at most one Infra 1440p or 2160p ladder per worker, so Infra jobs get busy answers more often. The workflow already retried a busy answer every 20 seconds, but that wait counted toward the three hour processing deadline. A batch of 4K uploads could fail with "Processing took too long" without ever reaching an encoder. The deadline now restarts whenever every dispatch answered busy and no output of the video is encoding. A running output still counts toward the deadline, as before.
The six-hour rule that restarts a stuck encode when its status is read now runs from the later of the encode's start and its last busy answer (
busyAtin the processing JSON, no schema change). A video that waited five hours and then started encoding is not restarted while it encodes. If its workflow stops, the busy answers stop too, and the restart happens six hours later as before.Dropped from the original branch
Regular Cap
Nothing regular Cap runs changes. The only shared file is
lib/rate-limit.ts, which gains two rule ids and one fallback entry; its tests pass unchanged. The middleware inapp/api/utils.tsis mounted only on/api/developer/v1and/api/developer/sdk/v1.Tests
viteston the Infra workflow, processing, state, player, SDK upload, SDK upload billing, public key refusal and API contract suites: all pass. Each new test fails without its fix: waiting out a busy farm, the restart clock followingbusyAt,busyAtbeing recorded, and races in the sweep (a recent completion claim, an upload assembled between storage reads, and an upload that opens before the row lock). Another test shows that a video with one output running and one busy still stops at three hours.rate-limit.test.tsandsave-video-edits-rate-limit.test.ts: 24 passed.tsc --noEmitinapps/web(aftertsc -bon its references): clean apart from one existing error inorganization-settings-ownership.test.ts, a file this PR does not touch.Separate connection failures from busy answers before merging so uploads retain a timeout during media-server outages.
Findings
Fix with agent prompt
Summary
Adds account-wide and rejected-credential request limits, retires abandoned uploads with no file, and lets encodes wait for worker capacity.
Reviews (13) · Last reviewed commit: "Merge branch 'main' into fix/infra-encod..." · Reviewed by Greptile