fix(core): re-route runs delivered to the wrong deployment - #2960
Conversation
🦋 Changeset detectedLatest commit: 6921d21 The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 133947ms → this run 152192ms (Δ +18245ms, +14%) 1020 steps (queue-hop) Cumulative STSO time: main 2956ms → this run 3886ms (Δ +930ms, +31%) 📜 Previous results (1)6921d21Wed, 05 Aug 2026 15:55:06 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
1da67fa to
3ef6bcc
Compare
a359b5e to
adba26d
Compare
TooTallNate
left a comment
TooTallNate
left a comment
There was a problem hiding this comment.
Reviewed at d3ae266 (single commit; base 23h / 12 commits behind main — merges clean, and see the merged-verification note below since #3046 heavily rewrote runtime.ts in the interim).
Verified locally:
- PR branch: full core suite green (1814 passed / 3 expected fail), errors package green, all 106 guard/classify/describe/runtime tests pass.
- Test-merged with current main (post-#3046 retained-VM rewrite): build + full core suite green, 1929 passed — both features' suites coexist. The guard sits before the main replay loop (once per delivery), so it has no interaction with per-iteration VM retention.
Adversarial checks that held up:
- The turbo self-defeat question: for a misrouted turbo initial delivery, this (wrong) deployment materializes the run itself — if
run_created/the synthesized snapshot stamped the ambient deployment id, the guard would never fire and the run would be permanently mispinned. Verified it stampsrunInput.deploymentId(the producer's pin) in both the resilient-start eventData and the synthesizedworkflowRun, so the guard trips correctly and the backend record stays pinned right regardless of who wrote it. - "Strictly better-addressed" claim: confirmed in world-vercel's queue — the original send inherits ambient
VERCEL_DEPLOYMENT_ID(opts?.deploymentId ?? process.env.VERCEL_DEPLOYMENT_ID), and the re-route passes an explicitdeploymentId, which the transport also preserves across its own redeliveries. - Payload hygiene on re-route:
runInputdeliberately dropped (re-engaging turbo would wedge the run),hookInputcarried (the lazy-hook re-ensure runs after the guard, so the re-routed message can be the only copy — and the re-ensure is idempotent perresumeId),stepId/stepNamepreserved on the step path, delivery-chain budgets (replayDivergence,serverErrorRetryCount) correctly reset. All four asserted in the runtime integration tests. - Bounded: the count rides the message (
deploymentMismatchRetryCount, old messages default 0), so ping-ponging across deployments converges at the budget. Non-continue outcomes return-and-ack, and theEntityConflictError/RunExpiredErrorswallow infailRunkeeps that true for terminal races. - Unencrypted
run_failedpayload rationale is sound (pinned deployment's key may be unreachable — the exact failure being handled),SPEC_VERSION_CURRENTon the event matches all four existingrun_failedwriters, and describe-error checksDEPLOYMENT_MISMATCHbefore the genericWorkflowRuntimeErrorbranch (the subclass ordering trap was anticipated). - Docs:
WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIESdocumented in runtime-tuning ✓; error page added for both v4 and v5 — necessary, since/err/:slugredirects into the default (v4) tree ✓.
Two non-blocking asks:
- Changeset bump types:
@workflow/errors(new exportedWorkflowDeploymentMismatchError+DEPLOYMENT_MISMATCHcode/slug) and@workflow/world(newWorldCapabilities.deploymentAffinityfield) add public API surface — those two should beminor, notpatch. On main the beta numbering doesn't care, but this is exactly the kind of stability fix that gets backported tostable, where the bump type is preserved. (@workflow/core/@workflow/world-vercelaspatchare fine.) - Stale CI, please rerun: the
astro - stable/sveltekit - stableE2E failures are tarball-staging errors from a run against an outdated merge-base — the job names even predate the current engine-matrix naming, and the equivalent lanes pass today on fresher PRs (e.g. #3329) and in my local merged-with-main run. Rebase or rerun to green the required check; nothing in this diff touches packaging.
A run is pinned to one deployment when it starts — the deployment that
called start(), or whatever start({ deploymentId }) resolved to (an
explicit id or 'latest'). When a queue callback reaches any other
deployment, the per-run encryption key is derived from the wrong master
key and the delivery throws RuntimeDecryptionError before user code runs;
the queue retry callback swallows it, so the run dies as a blank
"exceeded max retries".
A misrouted delivery is not treated as permanent. It is re-enqueued
explicitly addressed to the run's own deployment, which is strictly
better-addressed than the send that misrouted (that one inherited the
producing deployment's ambient id). The run is failed with the new
DEPLOYMENT_MISMATCH error code only once
WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES (default 3) is spent, mirroring
the bounded recovery replays a replay divergence gets before being
recorded as a corrupted event log. A deployment that cannot be reached at
all fails on the first delivery instead of burning the budget on an
unroutable target.
Both paths that execute a run are covered — queued step executions and
flow replays — and each already holds the run entity for other reasons,
so the guard adds no backend round trip. Nothing at all happens on the
receiving deployment: no workflow code, no step body, no step_started,
and no lazy-resume hook_received re-ensure. Whatever the delivery was
carrying travels with it, so a re-routed step keeps its identity and a
re-routed lazy hook resume keeps its payload — the pinned deployment
re-ensures the same single event, keyed by resumeId.
Recovery attempts create no events, so a run that recovers looks normal.
They are reported on the invocation span
(workflow.deployment.pinned_id, workflow.deployment_mismatch.retry_count,
workflow.deployment_mismatch.recovered) and as a runtime warning, and the
attempt count is carried in the terminal error message. The recovered
attribute distinguishes a delivery a re-route fixed from one that kept
misrouting, so the ratio is queryable rather than only visible once a run
dies.
Gated on the new World capability deploymentAffinity, which world-vercel
declares. Worlds whose deployment id is synthetic or version-tagged (e.g.
dpl_local@<sdk-version>) leave it unset, so a version bump cannot fail
their runs. The terminal failure is recorded without resolving the run's
encryption key, which is fetched from the pinned deployment's API and is
often gone by the time this fires.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
|
No backport to Although motivated by a real defect (a misrouted queue callback derives the wrong encryption key and the run dies as a blank "exceeded max retries"), this lands as substantial new machinery rather than a narrow fix: a new To override, re-run the Backport to stable workflow manually via |
2 participants
Summary & Motivation
A queue callback that reaches a deployment other than the one its run is pinned to derives the per-run encryption key from the wrong master key, so the delivery fails before user code runs and the run dies as a blank "exceeded max retries". The delivery is re-enqueued explicitly addressed to the run's own deployment — strictly better-targeted than the send that misrouted — and the run is failed with the new
DEPLOYMENT_MISMATCHerror code only onceWORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES(default 3) is spent. Gated on the new World capabilitydeploymentAffinity, so worlds with synthetic or version-tagged deployment ids are unaffected.Test Plan
Unit tests added for the guard and both runtime paths; local vitest and typechecks pass.