Skip to content

fix: Cap Infra auto top-up as soon as the balance falls below the threshold - #4156

Merged
richiemcilroy merged 33 commits into
mainfrom
fix/infra-auto-top-up-on-debit
Nov 13, 2026
Merged

richiemcilroy merged 33 commits into
mainfrom
fix/infra-auto-top-up-on-debit

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Auto top-up now works the way people expect from a usage-billed API: the card is charged as soon as the available balance falls below the threshold, as often as that happens, with a safety limit.

What was wrong

  • Only the storage run triggered it. runDueAutoTopUps ran only from the storage cron (41 */6 * * *). An app that spent its balance on transcription, or was refused an upload for low balance, got 402 for up to six hours with auto top-up on and a card saved.
  • One a day at most. runAutoTopUp refused for 24 hours after any attempt (lastAttempt included successful top-ups), and the Stripe key was infra-auto-top-up:<account>:<UTC day>. A busy app that used up one top-up stayed stuck until the next day.

What changed

flowchart LR
  A[Transcript hold, reserve or settle] --> S[scheduleAutoTopUp]
  B[402 at upload create or complete] --> S
  C[402 at transcript create or start] --> S
  S -->|in a request: after the response<br/>otherwise: awaited inline| R[runAutoTopUp]
  K[Storage cron, unchanged] --> R
  R -->|snapshot under account lock| D{below threshold?<br/>not disputed, not processing,<br/>no decline in 24h, under 10 in 24h}
  D -->|yes| P[PaymentIntent<br/>key = account : top-ups : declines]
Loading
  • Trigger (lib/infra-auto-top-up.ts): scheduleAutoTopUp({ accountId } | { appId }) decides by whether there is a request.
    • Inside a request it hands the attempt to after() and returns at once, so the attempt is never on the request's critical path.
    • Outside one, after() throws, so it awaits the attempt inline. The attempt is bounded by the 15 s Stripe timeout and cannot be frozen half-way.
    • Either way it catches and logs every error and never throws into the caller.
    • It is called after:
    • transcript holds, reservations and settlements that charged or ran short (lib/developer-transcript-billing.ts);
    • 402 refusals at upload create and upload complete (lib/infra-video-upload.ts);
    • 402 refusals at transcript create and start (lib/developer-transcript-store.ts).
    • The storage day charge already ran runDueAutoTopUps straight after charging, in the same cron run, so it needs no extra call.
  • Several top-ups a day. A top-up now waits only in these cases:
    • after a decline, for 24 hours;
    • while a payment is still processing, however long that takes. As a backstop, a run that finds a payment pending for over 72 hours reads it from Stripe once and records the outcome: a success is granted, any other final state is recorded as a decline, and one still processing is left.
    • once there have been 10 automatic top-ups in a rolling 24 hours (it logs and waits);
    • while a dispute is open (unchanged).
  • One charge per attempt. The available balance, dispute state and attempt history are read in one transaction that holds the account row lock. That is the same lock every grant and debit takes. The Stripe key is infra-auto-top-up:<account>:<recorded top-ups>:<recorded declines>, with nothing from the settings in it. A recorded decline therefore gives the retry 24 hours later a new key, so Stripe tries the card again instead of replaying the stored decline.
    • A retried or racing trigger sends the same key, so Stripe returns the same PaymentIntent. The grant is deduplicated by PaymentIntent under that lock.
    • A racer that reaches Stripe while the first request is still in flight gets idempotency_key_in_use (409).
    • A trigger that reads the same attempt after the amount or card changed gets StripeIdempotencyError; this was measured against the real API. So a settings change cannot start a second payment.
    • Both errors count as an attempt in progress, never as a decline: no ledger row is written and auto top-up stays on.
  • Stripe timeout. The off-session PaymentIntent create has a 15 s request timeout and no SDK retry. A timeout raises StripeConnectionError (measured). It is logged like any other failure to reach Stripe and writes no ledger row. The next attempt sends the same idempotency key, so a retry cannot charge twice.
  • Unchanged: amounts, threshold rules, the decline email and switch-off, the dispute pause, and the cron backstop.
  • Console: after a purchase, while auto top-up is off, one line under the purchase form links to the Auto top-up block. The card notes no longer say the card is charged at most once a day.
  • Docs: billing.mdx says auto top-up runs as soon as the available balance falls below the threshold, up to 10 times in any 24 hours, and waits 24 hours after a decline. developer-billing-docs.test.ts checks both against the code.

Two checks on the transcription workflow

1. The workflow bundle is unchanged. I ran the workflow builder the way next build runs it: the withWorkflow config at phase-production-build, which writes app/.well-known/workflow/v1/flow/route.js and step/route.js. I ran it on the base commit (802e2ac) and on this branch.

  • The workflow bundle, flow/route.js (10,862,920 bytes), is byte-identical on both (cmp finds no difference).
  • It contains none of infra-auto-top-up, scheduleAutoTopUp, developer-transcript-billing, api.stripe.com, infra-billing-emails, next/server, AUTO_TOP_UP or createHash. The builder already drops the step-only billing import from the workflow bundle.
  • The step bundle is identical too. So no dynamic import() is needed.

2. after() inside a workflow step.

  • On Vercel a step runs inside the step route's request: @workflow/world-vercel's queue handler awaits the step within the route handler. So after() has a scope there and the attempt runs after the step answers.
  • Anywhere without a request scope, scheduleAutoTopUp now awaits the attempt instead of leaving a detached promise. A unit test covers each path.
  • The 826 ms figure below came from calling settleTranscript directly, outside a request, so it took the out-of-request path.
  • I also ran the proof once through developerTranscribeWorkflow itself: the real hold, reserve and settle steps on scratch MySQL with Stripe test mode, with only speech, storage and the media server faked.
    • after() was called 3 times and threw each time, so each attempt ran inline.
    • The first step's hold dropped the available balance below the $10 threshold, and a $5 PaymentIntent succeeded.
    • The ledger shows the top-up ahead of the transcription debit (topup 500000 -> 1500050, then transcription -83 -> 1499967).
    • The whole workflow, top-up included, took 1,615 ms.

Owner ceiling: $1,000 a day of automatic top-ups across all apps

Each app has its own card and its own 10-a-day limit, and an owner can have 25 apps. Without an owner-level ceiling, one stolen card saved on every app could be charged far more than any single app's limit. So:

  • The ceiling. Automatic top-ups across all of an owner's apps stop at $1,000 in any rolling 24 hours.
    • The sum covers settled automatic top-ups plus payments still processing, plus this attempt's amount.
    • Past the ceiling, the attempt waits and logs owner_daily_limit. Auto top-up stays on, no email is sent, and nothing is written to the ledger.
    • Credit bought by hand does not count, and manual checkout is unchanged.
  • Reserve under the lock, call Stripe outside it.
    • Under the owner lock (every account row of the owner, in id order) the attempt re-reads the account's switch, threshold, amount, customer and card, and runs the four waits against that amount.
    • It then inserts a reservation and commits at once. The reservation is a topup_pending ledger row with amount 0, referenced by the attempt's idempotency key. Its metadata holds the amount, customer, card, receipt email and description. On a real database the lock is held for 6 to 47 ms.
    • Stripe is called after commit, outside any transaction, with exactly the reserved values and key and a 15 s timeout. The amount the ceiling counted is the amount charged.
    • The owner ceiling counts every reservation still pending, so a lost answer keeps counting until it settles. The account's own reservation counts once, as the attempt's amount.
    • Success grants and removes the reservation in one short transaction, and a decline records itself and removes it. A payment still processing relinks the reservation to its PaymentIntent.
    • A timeout or "in progress" leaves the reservation. The account's next run replays it with the same key and the same reserved values for up to a day (Stripe keeps keys for at least that long). After that it waits, and at 72 hours the reconcile finds it through Stripe's search by attempt, settling it or voiding it if Stripe never charged.
    • An error that never reached Stripe voids the reservation. A timeout never does.
    • Recovery comes first. Payments and reservations pending for 72 hours are reconciled before the switch, the balance, the decline wait or any limit, and the sweep selects those accounts whatever else applies. A reconcile charges nothing new.
    • Lock order. Every settle path locks the account row before the ledger row, the same order the reservation step uses. An earlier ordering deadlocked under the 10-trigger race; the races now pass 10 runs out of 10.
    • I chose row locks over GET_LOCK because the codebase already relies on SELECT ... FOR UPDATE on PlanetScale, and named locks there need a reserved connection I could not verify without production.
  • The customer ledger never shows a pending row. Reservations and payments still processing are zero-amount rows that only the billing code needs. The console, the CSV export and the transactions API leave them out, and the API reference no longer lists that row type.
  • Description. Each automatic charge reads Cap Infra credits for <app name>, auto top-up. The name is trimmed with its whitespace collapsed, and the whole description is kept within Stripe's 1,000 characters.
  • Index use (EXPLAIN on a scratch database, 40 owners with 5 apps each and 5,000 ledger rows):
    • Owner to accounts uses owner_idx, and the ledger sum uses account_type_created_idx.
    • The "has this pending payment settled" check uses account_ref_dedup_idx.
    • Every other wait is ref or range on account_type_created_idx.
    • In the due query, the outer scan stays on indexes (owner_deleted_idx, then app_id_unique), and the only unindexed step is the 2-row derived held-credit table.

One waiting rule for the guard and the sweep

autoTopUpWaits(now) in lib/infra-auto-top-up.ts is one SQL fragment with four conditions, correlated to the account row:

  • an unresolved pending payment younger than 72 hours;
  • a decline in the last 24 hours with no top-up after it;
  • 10 or more top-ups in the last 24 hours;
  • the owner's $1,000-a-day ceiling.

It is used in two places:

The two therefore cannot drift apart. Details:

  • Index use. Every subquery filters on accountId, type and a createdAt range, which account_type_created_idx serves. The "is this pending payment settled" check uses account_ref_dedup_idx.
  • The 72-hour backstop is reachable. A pending payment older than 72 hours no longer blocks, so an account stuck behind one is released into the sweep even with no other trigger, and runAutoTopUp then reconciles it from Stripe.
  • Explicit references. Every column in the fragment is written as `alias`.`column`. In a single-table select, drizzle drops table names from columns, and a correlated subquery would then compare its own columns; the real-database tests caught this.

The #4143 merge keeps withTranscriptSlot as it is. The only transcript trigger is on the 402 low-balance refusal in startGate; the 429 too_many_transcripts does not ask for a top-up.

For #4145

The first-month charge can call scheduleAutoTopUp({ accountId }) after it commits.

Proof

Stripe test mode, real scratch MySQL, rerun on the final code (reservations, owner ceiling, with #4142 and #4143). All seven live checks pass: the transcription and same-day top-ups, the declined card, the race, the settings change, the decline retry, the 72-hour reconcile and the full workflow. The scratch databases were dropped afterwards. The timings below come from the first run. On the merged code the top-up runs inline outside a request, so it has finished by the time settleTranscript returns, and the full workflow including the top-up took 915 ms.

First run. Accounts were set at a $10 threshold, $5 top-ups, with a real test card. The transcription debit ran through settleTranscript, the same function the transcription workflow calls. These are reproduced timings:

settled (UTC) top-up in ledger delay
Transcription 1, balance from $10.05 to $9.95 04:27:55.309 04:27:56.135 826 ms
Transcription 2, same UTC day 04:27:56.171 04:27:57.404 1,233 ms
  • Two succeeded $5 PaymentIntents, each with its own key. The ledger reads: transcription, top-up, transcription, top-up.
  • Declined card (pm_card_chargeCustomerFail):
    • The charge is declined and auto top-up switches off.
    • Switched back on, it skips with recent_decline straight away and still at 23.9 hours, with 1 Stripe attempt.
    • At 24.1 hours it tries again, and Stripe shows 2 attempts.
  • Six triggers racing on one account: one result was succeeded and five were in_progress. Stripe holds 1 PaymentIntent and the ledger 1 top-up row.
  • Amount changed while an attempt was unresolved. The first PaymentIntent was created at Stripe and its response was dropped, so no ledger row was written. Then the amount went from $5 to $25.
    • The next trigger got StripeIdempotencyError and answered in_progress. Stripe held one PaymentIntent ($5, attempt 0:0).
    • Settling it through the webhook path lifted the balance above the threshold.
    • The next attempt charged $25 at attempt 1:0. The new amount applies from the next attempt, and there is never a second payment for one attempt.
  • Payments left pending (real PaymentIntents, pending rows backdated in the scratch database):
    • pending for 25 hours: awaiting_payment, and Stripe is not asked;
    • pending for 73 hours and succeeded at Stripe: reconciled, the $5 top-up is recorded, and it stays a single PaymentIntent;
    • pending for 73 hours and declined at Stripe: reconciled, a failure is recorded, and auto top-up switches off.
  • Decline, then the retry at 24 hours. The decline was attempt 0:0. The account waited at 23.9 hours. At 24.1 hours the retry sent attempt 0:1, a new key, and Stripe made a second, separate PaymentIntent and declined the card afresh. The stored decline was not replayed.

Committed tests

  • __tests__/integration/infra-auto-top-up-database.test.ts runs against a real MySQL with a Stripe fake that honours idempotency keys:
    • ten triggers racing produce one charge and one grant, and the balance rises once;
    • triggers racing while the amount changes produce one charge (this fails if the amount goes back into the key);
    • an amount change after a lost response makes no second payment, and the new amount applies from the next attempt;
    • a second top-up the same day uses a new key;
    • the 10-a-day limit holds.
    • Making the key random fails all three of the race and settings tests.
  • Lock hold, on a real database. One app's Stripe call was made to take 2 s. The owner lock was held 7 to 22 ms, and a FOR UPDATE on another app of the same owner (the lock that upload completion and transcript holds take) waited 1 to 2 ms during the call.
  • Crash recovery, on a real database:
    • a reservation left before Stripe was called is reused by the next attempt, with one charge and no reservation left behind;
    • a payment Stripe made before the response was lost is replayed under the same key and granted once.
  • Greptile's three P1s, on a real database:
    • with $500 already topped up on another app, $600 waits; lowered to $5, the attempt is charged $5 even though the setting goes back to $600 during the Stripe call, and the owner's total is $505;
    • app A's $600 charge goes through at Stripe but the answer is lost, and app B's $600 then waits with owner_daily_limit (if the ceiling counts only settled top-ups, this fails);
    • a $1,000 payment pending 73 hours, with another app topped up $5 today and the first app switched off and over its threshold, is still granted by the sweep.
  • Ledger, on a real database: a reservation and a processing row are absent from the ledger page and the export, which show only the real rows.
  • Owner ceiling, on a real database:
    • two apps of one owner at $600 each: the first is charged, and the second waits with owner_daily_limit and stays switched on;
    • two apps of one owner triggered at the same moment near the ceiling make exactly one charge. If the lock covers only the app's own row, this test fails with 2 charges;
    • a $900 manual purchase on one app does not hold back the other app's automatic top-up.
  • Owner ceiling, unit: the attempt waits, stays switched on, sends no email and logs owner_daily_limit; it goes through again once 24 hours have passed; and credit bought by hand is not counted.
  • Live race with the owner lock, Stripe test mode: six triggers on one account make 1 PaymentIntent and 1 ledger row, and the other five see above_threshold.
  • On a real database, the locked check and the sweep read the same waits:
    • a decline 2 hours ago makes the guard wait;
    • a payment pending 25 hours keeps the account out of the sweep;
    • one pending 73 hours is picked up by the sweep with no other trigger and reconciled from Stripe;
    • an account with nothing blocking is charged.
  • fix(infra): charge transcripts deleted during speech and cap running transcripts per app #4143's suites pass with this branch: developer-transcript-concurrency, developer-transcript-console-ceiling, agent-developer-transcript-limit and the transcripts database suite (32 tests on a local cap_speech_* database).
  • Unit tests cover:
    • the new waiting rules, the key, and "key in use" and a parameter mismatch both counting as in progress;
    • a processing payment blocking the next one at 25 and 71.5 hours, and the next attempt going ahead once it settles;
    • the 72-hour reconcile recording a success (with a late webhook for the same payment granting nothing more) and a failure, leaving a payment Stripe still reports as processing, and not asking Stripe before 72 hours;
    • scheduleAutoTopUp returning at once inside a request, finishing the attempt before returning outside one, and never throwing;
    • each trigger site, in both directions: a top-up is asked for after charges and 402s, and not when nothing changed;
    • the post-purchase line.
  • Whole web unit suite, same base (802e2ac): 8,216 tests on main and 8,229 on this branch. The same 8 tests fail on both sides (homepage-seo and share-stage-identity, failing on main too). No test changed status. The only removed tests are the two rewritten for the new rules: "keys by UTC day" and "one attempt per 24 hours".

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; the deleted-app recovery issue is fixed.

Summary

Auto top-up now responds to low balances after transcript charges and billing refusals, with repeat payments, saved attempts, and daily limits.

  • Low-balance usage events now schedule an automatic top-up right away.
  • Auto top-ups can repeat within shared daily limits while uncertain payments stay tied to one attempt.
  • Pending automatic payments no longer appear in customer transaction lists.
  • Billing screens explain the new top-up limits and link buyers to its settings.

Reviews (13) · Last reviewed commit: "test: the sweep voids a deleted app's lo..." · Reviewed by Greptile

A top-up now waits only after a decline, while a payment is still processing, or at ten automatic top-ups in any 24 hours. Each attempt is keyed by the account's count of settled top-ups and the exact request, read under the account lock, so a retried or racing trigger makes one payment. Adds scheduleAutoTopUp, which runs a top-up after the response is sent and never throws into the caller.
Transcript holds, reservations and settlements, and low-balance refusals at upload create, upload complete and transcript create or start, now ask for an auto top-up straight away instead of waiting for the next storage run.
After a purchase, while auto top-up is off, the Billing page says it can add credit as soon as the balance falls below a threshold. The card notes no longer say the card is charged at most once a day.
A hung Stripe call could use up the storage run's time. A timeout is logged like any other failure to reach Stripe, writes no ledger row, and the next attempt sends the same idempotency key.
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Skipped Skipped Nov 13, 2026 9:19am UTC

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

Open in Capy

Comment thread apps/web/lib/infra-auto-top-up.ts Resolved
Comment thread apps/web/lib/infra-auto-top-up.ts Resolved
Comment thread apps/web/lib/infra-auto-top-up.ts Resolved
@sandchest-com

sandchest-com Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest reviewed 8525be7: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 35 credits · Review details

The Stripe key is now the account and its count of settled top-ups only, so a trigger that reads the same attempt after the amount or card changed gets Stripe's idempotency error instead of a second payment, and that error counts as an attempt in progress. A payment still processing blocks the next one however long it takes. Outside a request, scheduleAutoTopUp now finishes the attempt before it returns instead of leaving a detached promise.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 04:45 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 04:49 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

A pending payment blocks the next auto top-up until it settles. So an account is never stuck behind a payment whose webhook never arrives, a run that finds one older than 72 hours reads it from Stripe once and records the outcome, or leaves it if Stripe still reports it as processing.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 04:52 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

The key was already counting declines alongside top-ups. It now names both counts, so a retry after a recorded decline visibly sends a new key while a settings change during an attempt keeps the old one.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 04:55 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

…-on-debit

# Conflicts:
#	apps/web/lib/infra-ledger-store.ts
…-on-debit

# Conflicts:
#	apps/web/lib/developer-transcript-store.ts
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

Comment thread apps/web/lib/infra-auto-top-up.ts Resolved
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 09:03 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

Settled automatic top-ups count for 24 hours. A reservation or a payment still processing counts until it settles or is reconciled, because it may already have been charged.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 09:05 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

Open in Capy

Comment thread apps/web/lib/infra-auto-top-up.ts Resolved
Comment thread apps/web/lib/infra-auto-top-up.ts Resolved
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 09:15 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 13, 2026 09:19 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

This branch was successfully deployed

1 active and 1 inactive deployments
Preview – cap-web — 8525be7b Deployed Nov 13, 2026 by vercel[bot]
Preview – cap-status — 8525be7b Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants