feat(core): experimental in-process VM continuation for inline replay - #2966
feat(core): experimental in-process VM continuation for inline replay#2966VaguelySerious wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 8467bdf The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
|
Deployment failed with the following error: |
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (2)ff3959bFri, 17 Jul 2026 17:06:35 GMT · run logs
ff3959bFri, 17 Jul 2026 16:25:18 GMT · run logs
Avg deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Combines the retained-session architecture from #2984 with the env kill switch and loop-level single-VM test from #2966. - executeWorkflow with discriminated request/result types and a WorkflowSession state machine (running/suspended/failed/replay/completed) - EventsConsumer.append: only newly durable events feed the live VM - WORKFLOW_RETAINED_VM=0 kill switch (default on) - retained-vm-loop.test.ts: proves one VM per run and byte-identical output vs the from-scratch replay path
Add an opt-in path (WORKFLOW_VM_CONTINUATION=1, off by default) that keeps a suspended workflow VM alive across inline-loop iterations instead of rebuilding the vm.Context and replaying the event log from scratch each pass. On a step-only suspension, runWorkflow attaches a `continuation` handle to the WorkflowSuspension. The inline loop, after appending the step's terminal events, feeds them into the SAME live VM via EventsConsumer.resume(), resolving the pending step promise so the workflow body advances to its next checkpoint. Scope and safety: - Only within a single invocation's inline loop (same process owns the writes, so the durable event log stays authoritative). - Only for step-only suspensions; hooks/waits/attribute writes fall back to a full replay (they involve out-of-band invocations and delivery-barrier ordering the replay path handles specially). - resume() prefix-checks the consumed events against the authoritative log by eventId and throws ReplayDivergenceError on any mismatch; the loop swallows it and falls back to a fresh replay. Worst case == today's behavior. Verified: full core suite green with the flag both off and on (1488 passed); a loop-level test proves the VM is constructed once (resumed per step) with the flag on vs rebuilt per replay off, producing byte-identical output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flip the flag to default-on so CI and benchmarks exercise and measure the continuation path. `WORKFLOW_VM_CONTINUATION=0` reverts to rebuilding the VM and replaying from scratch. Updates the loop test to drive the baseline via the kill switch and the ON case via the (now default) unset env. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ff3959b to
8467bdf
Compare
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Trace-grounded context (durabench prod, SDK 5.0.0-beta.34, iad1, 2026-07-20): warm STSO is ~150-270 ms/step (mean ~190) vs Temporal 45-50. Decomposing Datadog flow trace 2f55f784951f4eba33987bf54463b247 (run wrun_41KXZ0W77G0GJEPSCQEKGE823D), each warm step is ~65-92 ms of step_started POST + ~50-70 ms of step_completed POST + 28-78 ms of workflow.run full replay (O(journal) per step, so O(steps^2) per run) + 40-130 ms of CPU. Keeping the VM alive removes the 28-78 ms replay slice (~30-40% of the warm gap) and most of the per-step CPU; the two serialized POSTs (~120-160 ms) remain, and are targeted by the complementary batched-transition work in vercel/workflow-server@pgp/batch-step-transitions + vercel/workflow@pgp/batch-transition-client. So the mechanism here goes after the right slice.
The implementation is clean and the fallback is well-scoped, but the safety argument is incomplete relative to the competing #2990: it covers synchronous RNG/clock/correlation-id determinism but not host-timed async progress, which a live (non-quiesced) VM can observe across the inline step's real wall-clock gap. That is the one thing that makes this unsafe to run default-on as written -- details inline.
Convergence: #2990 implements the same idea with the sandbox quiescence this PR is missing, and should be the base. I'd close this in favor of it, or keep this branch only as a minimal default-off variant if the team wants a smaller first step. What is worth carrying from here into #2990 is exactly what it already credits absorbing: the minimal EventsConsumer.resume mechanic and the createContext-called-once A/B test.
Head-to-head durabench benchmark cells for this branch and #2990 are being added; numbers to follow as a comment.
| // delivery-barrier ordering that the from-scratch replay path handles | ||
| // specially. Anything else leaves `continuation` unset so the caller falls | ||
| // back to a fresh replay. | ||
| const isContinuationEligible = (s: WorkflowSuspension): boolean => |
There was a problem hiding this comment.
This is the core soundness gap versus #2990. Eligibility here is purely structural (no hooks/waits/attrs/disposals/aborts) but says nothing about whether the suspended VM can still make progress on host timing. The sandbox on this branch still exposes Atomics.waitAsync, async WebAssembly.compile/instantiate, WeakRef/FinalizationRegistry, and a threadpool-backed async crypto.subtle.digest. A live VM kept parked across the inline step's real wall-clock gap (the two POSTs are ~120-160 ms) can observe those settle, whereas a from-scratch replay consumes the log with ~no wall-clock elapsed. Concretely:
const ia = new Int32Array(new SharedArrayBuffer(8));
const timeout = Atomics.waitAsync(ia, 0, 0, 50).value.then(() => 'timeout');
const r = await Promise.race([someStep(), timeout]);Cold replay always resolves someStep() first (its result is already in the log) -> deterministic. Retained across a ~150 ms inline step, the 50 ms waitAsync fires during the gap and the race picks 'timeout' -- and because retained-mode writes are authoritative, the run commits a branch no replay can reproduce. The summary's "RNG/clock/correlation-id advance naturally, exactly matching a from-scratch replay" is true for those synchronous draws but does not cover this. #2990 closes it by removing exactly these APIs from the sandbox and making digest synchronous; that hardening (or an equivalent quiescence guarantee) is the prerequisite for this to be safe default-on.
| * so the caller falls back to a full replay in a fresh VM. Matching is by | ||
| * `eventId`, which is immutable per event across reads. | ||
| */ | ||
| resume(events: Event[]): void { |
There was a problem hiding this comment.
The prefix guard is necessary but not sufficient. It validates that the already-consumed prefix (0..eventIndex) is byte-stable, which catches a rewritten log -- but it cannot detect a VM that took a different future branch after the cursor because it advanced on host timing during the gap (see the isContinuationEligible comment). The consumed prefix still matches; the divergence is in the events the retained VM is about to produce. #2990 adds a boundary-stability check (isSameSuspensionBoundary, demoting to replay if the VM re-suspends on anything other than the same steps) precisely for this; this PR trusts a non-quiesced advance silently. At minimum, assert the VM re-suspends on the same step set before adopting the appended events.
Minor, same method: dropping readonly and doing this.events = events makes the consumer alias the caller's array rather than own a copy (#2990 copies in the ctor and append()s only the delta). It works today because the loop happens to pass the same array, but it couples the consumer's internal invariant to the caller's array handling; copy-and-append is less fragile.
| */ | ||
| export function isVmContinuationEnabled(): boolean { | ||
| const raw = process.env.WORKFLOW_VM_CONTINUATION; | ||
| if (raw === undefined || raw === '') return true; |
There was a problem hiding this comment.
Default-on posture: this flag is read at runtime in production too (as the PR notes), and combined with the open quiescence gap above, the draft/POC status, and no execution-mode telemetry, that is a lot to enable by default. I'd flip the default to off here until the sandbox is quiesced (or this converges onto #2990's hardened sandbox), and keep default-on only in the CI/benchmark lanes via an explicit env in those workflows. As written, the first production workflow that races a host-timed primitive against a step gets a silently corrupted run with no signal -- the worst failure mode for a durability product.
|
Head-to-head durabench benchmark (follows up the review): this branch vs #2990 vs
Takeaways: (1) −30% warm STSO and −83% per-step CPU vs baseline, with the win growing with step count (replay was O(journal)/step); (2) #2990 and #2966 are statistically indistinguishable on latency — the decision between them is purely the correctness/architecture question from the reviews; (3) the remaining ~120-135ms/step is the two serialized event POSTs, which retention can't touch — that's the batch-transition track ( 🤖 Generated with Claude Code |
…#3046) * perf(core): retain workflow VM across inline steps Combines the retained-session architecture from #2984 with the env kill switch and loop-level single-VM test from #2966. - executeWorkflow with discriminated request/result types and a WorkflowSession state machine (running/suspended/failed/replay/completed) - EventsConsumer.append: only newly durable events feed the live VM - WORKFLOW_RETAINED_VM=0 kill switch (default on) - retained-vm-loop.test.ts: proves one VM per run and byte-identical output vs the from-scratch replay path * refactor(core): simplify retained-session control flow - executeWorkflow overloads: a fresh replay request can no longer return { type: 'replay' }, deleting the runtime invariant throw and runWorkflow's dead branch - isSameSuspensionBoundary reduced to the steps-array comparison (all suspension counts are derived from steps in the constructor) - runtime loop initializes workflowResult with a ternary * fix(core): decline retention for VMs that ran host-timed async work crypto.subtle.digest is the only sandbox API whose promise resolves on host timing rather than from the event log, so a workflow racing it against a step can advance while suspended and diverge from what replay reconstructs. A sticky usedHostAsync bit on the VM context makes canRetainWorkflowSession fall back to ordinary replay for such VMs; a quiescent step-only VM remains a pure function of the consumed event prefix and stays retainable. * fix(core): track all host-timed async VM APIs for retention Atomics.waitAsync (a wall-clock timer via SharedArrayBuffer) and the async WebAssembly compilation entry points resolve on host timing just like crypto.subtle.digest. Wrap every such intrinsic in createContext so usedHostAsync covers the complete set; dynamic import() settles within a microtask and cannot advance a suspended VM. * feat(core): compute crypto.subtle.digest synchronously in the sandbox node:crypto createHash produces byte-identical values to WebCrypto and settles the digest promise on a deterministic microtask instead of host threadpool timing. A digest can therefore never advance a suspended workflow, so digest-using VMs stay retainable; only Atomics.waitAsync and async WebAssembly compilation remain host-timed. createHash is stable and undeprecated on Node 18-26 (DEP0179 only removed the direct Hash constructor). * fix(core): remove WeakRef and FinalizationRegistry from the sandbox GC observation depends on host GC timing that neither replay nor a retained VM can reconstruct from the event log. WeakMap/WeakSet stay available (they do not expose GC state). * fix(core): enforce the BufferSource contract in the sandbox digest Reject non-BufferSource digest input with TypeError like WebCrypto does, via the native ArrayBuffer.prototype.byteLength brand check (works across vm realms). Previously a plain number was treated as a Uint8Array length, turning a small input into a giant allocation. * fix(core): demote retention when suspension serialization draws randomness handleSuspension dehydrates step arguments with the live VM, and that serialization can execute user code (getters, WORKFLOW_SERIALIZE hooks). Randomness drawn there would desync the retained VM's future correlation IDs from what a fresh replay regenerates. Count every draw from the seeded stream at its single source in createContext and fall back to ordinary replay if handleSuspension consumed any. * refactor(core): make VM quiescence unconditional, cut tracking machinery Delete Atomics.waitAsync and the async WebAssembly entry points from the sandbox instead of tracking their use — with digest synchronous and GC intrinsics removed, no sandbox API settles a promise on host timing, so a suspended VM provably cannot advance. This deletes the trackHostAsync wrapper, the usedHostAsync bit and session method, the runtime gate clause, the session 'failed' state (unreachable), and the background-progress test scenarios (impossible by construction). * refactor(core): gate retention on passively cloneable step inputs Replace the RNG draw-counter demotion with prevention: when a session is a retention candidate, new step inputs take a passive descriptor walk (never invoking getters; proxies, accessors, functions, custom classes, and platform wrappers decline) and safe values are structuredClone'd into the host realm before dehydration, so serialization never executes workflow-owned code against a retained VM. Unsafe inputs serialize the old way and the session falls back to ordinary replay. * fix(core): harden the passive step-input walker - require enumerable on array index descriptors: structuredClone drops non-enumerable indices that devalue persists - read workflow globals and constructor prototypes via own-property descriptors only, so validation can never execute workflow-owned accessors on redefined globals * fix(core): guard proxied constructors in the passive-input walker constructorPrototype reads both realms' constructors via own-property descriptors only and refuses proxies before any descriptor read, so a proxied redefined global can never observe validation. * fix(core): preserve retention gate after rebase * fix(core): all-or-nothing clone batches; reject SAB views in digest - A mixed step batch (one unsafe sibling input) now serializes every input through the ordinary VM path: a clone snapshotted before an unsafe sibling's serialization runs its getters could otherwise durably capture stale sibling state. - crypto.subtle.digest rejects SharedArrayBuffer-backed views with TypeError, matching WebCrypto's BufferSource contract. * fix(core): narrow the fast path to prototype-independent types devalue serializes Map/Set through the realm's iterator protocol and Date/RegExp/typed arrays through prototype getters, all of which workflow code can mutate — so their serialization is not provably passive and their bytes could differ between retained and cold modes. The fast path now accepts only primitives, plain objects, and plain arrays, which devalue traverses exclusively via own-property reads. Slot-bearing exotics decline even with a swapped prototype. The sandbox digest now reads view metadata (buffer/byteOffset/ byteLength) through captured intrinsic getters, so own properties shadowing them cannot change which bytes are hashed or bypass the SharedArrayBuffer rejection. * fix(core): freeze serialization-consulted sandbox intrinsics instanceof dispatch (Symbol.hasInstance via the constructor, Function.prototype, and Object.prototype), the class reducer's value.constructor walk, and devalue's Object/Array traversal all consult intrinsics workflow code could redefine — legally and deterministically — which would make the durable step input depend on WORKFLOW_RETAINED_VM (spoofed values serialize as e.g. Maps on the cold path but as plain clones on the retained path). Freeze Object/Array/Function (constructors and prototypes), the VM collection constructors, and every reducer-referenced global binding (absent ones pinned to undefined) right before the workflow bundle evaluates, so the retained-input equivalence holds by construction. Host-realm constructor escapes (e.g. TextEncoder.constructor) remain out of the determinism contract: code scheduling host timers was never deterministic under ordinary replay either; documented on canRetainWorkflowSession. * fix(core): freeze every non-shared serialization constructor Typed-array constructors (and their shared %TypedArray% parent), the Date wrapper, and the session-local AbortController/AbortSignal/ Request/Response bindings were pinned but not frozen, so workflow code could still add Symbol.hasInstance statics that diverge reducer dispatch between the retained clone (host constructors) and ordinary VM serialization. Freeze every binding value that is not the shared host intrinsic; shared host objects are dispatched identically by both paths, so mutations there cannot cause mode divergence. * fix(core): build retained clones in a pristine realm Replace structuredClone with an explicit deep copy into an SDK-private realm: clones previously inherited host prototypes, which workflow code can reach (e.g. via structuredClone's return values) and vandalize with Symbol.toStringTag or constructor overrides, shifting devalue's classification of the clone relative to the ordinary VM path. The pristine realm is unreachable by any user code, and the explicit copy serializes exactly what devalue traverses (own indices, own enumerable string props). Arrays also now decline own constructor properties, which the class reducer reads even when non-enumerable. * fix(core): verify host dispatch pristineness before retained cloning Host intrinsics are shared with the whole process and cannot be frozen, but workflow code can reach them (structuredClone results, exposed host classes) and install Symbol.hasInstance predicates that distinguish the original from its clone — or WORKFLOW_SERIALIZE statics on host Object/Array that the class reducer reads for host-prototype originals (hydrated step results). prepareRetainedStepInput now verifies, via own-descriptor reads only, that every host dispatch point is pristine and declines retention before any clone exists — so a spoofed predicate can never observe or capture a pristine-realm object. * fix(core): reject symbol properties from retained step inputs Reducers dispatch on symbol tags (e.g. the workflow abort-signal markers) that are non-enumerable and dropped by the pristine-realm copy, so a tagged object would serialize as an abort descriptor on the cold path but as plain data on the retained path. * fix(core): retained inputs accept only own enumerable data properties Hidden own keys of any kind — non-enumerable properties, accessors, symbols — can be observed by serialization dispatch (reducer probes like .signal, thenable checks, the class reducer) while the pristine clone drops them. With no hidden own keys, every probe on an accepted object resolves deterministically through validated data or pristine prototypes. * fix(core): freeze binding prototype chains for hasInstance lookup Symbol.hasInstance dispatch walks the constructor's prototype chain, so the frozen Date wrapper still exposed the unfrozen original VM Date it delegates statics to. Freeze each non-shared binding's full chain (stopping at host Function/Object prototypes) and verify host Object.prototype carries no added hasInstance on the detection side. * refactor(core): single-path retained serialization via pinned members (v2) Serialize step inputs for retained boundaries through the one ordinary pipeline (original value, workflow global) instead of cloning into a pristine realm and serializing under the host global. With a single serialization event shared by every mode, durable bytes structurally cannot depend on WORKFLOW_RETAINED_VM; the only property retention needs is that serialization executes no workflow code, established by: - the passive walker (descriptor-only, unchanged in spirit), now also accepting Map/Set/Date/typed arrays/ArrayBuffer — the common built-in step arguments — via prototype-identity checks - vm/serialization-pins.ts: the 10 prototype members serialization executes for those built-ins (measured empirically), captured at context creation and identity-verified at each retained boundary; the 'touches only pinned members' test instruments every member and locks the list against serde drift - host-realm instances (hydrated step results) accepted without member verification: host members run host code, which cannot touch retained VM state Deletes the pristine clone realm, the host-dispatch pristineness checks, and the batch clone bookkeeping. * refactor(core): freeze built-in prototypes instead of pinning members (v3) Review found the pin approach's structural hole: the class reducer READS value.constructor through Map.prototype — a data property when pristine (so member instrumentation never listed it), but executable the moment workflow code redefines it as a getter. Pinning what serialization executes misses what it reads. Freeze the accepted built-ins' prototypes wholesale (Map/Set/Date + iterator prototypes, %TypedArray% + subclass prototypes, ArrayBuffer): reads and executes are both immutable, and a patch attempt now throws loudly at the patch site instead of silently degrading. Deletes vm/serialization-pins.ts; the walker requires Object.isFrozen on the realm prototype (also covering realms where the freeze never ran). Also restores the host-dispatch pristineness check the v2 cut lost: workflow code can reach shared host constructors (exposed classes, structuredClone results) and plant workflow-realm Symbol.hasInstance hooks or WORKFLOW_SERIALIZE statics that reducers would execute during retained serialization. Host-realm built-in instances decline for the same reason; host-realm plain data (hydrated results) stays retainable. * fix(core): harden the passivity checker's own execution surface - Capture Map/Set forEach and the %TypedArray% buffer getter as module- load primordials: the checker previously invoked live host methods that workflow code can reach (structuredClone(new Map()).constructor) and replace with delegating workflow-realm closures. - Typed arrays must have one of the realm's real frozen subclass prototypes by identity — 'frozen and chains to %TypedArray%' admitted manufactured frozen hostile prototypes with delegating buffer getters. * fix(core): checker uses module-load primordials; verify inherited serializer statics - The walker resolved Object.getOwnPropertyDescriptor, Reflect.ownKeys, Array.isArray, Number/String helpers, and Object.getPrototypeOf/isFrozen from live host globals workflow code can reach and replace; all are now module-load captures, so the checker can never execute a planted delegate. - The class reducer reads cls[WORKFLOW_SERIALIZE]/cls.classId as inherited Gets, so isHostDispatchPristine now also verifies host Function.prototype and Object.prototype carry no serializer statics. Generic replacement of shared host statics (Object.keys, Array.from, …) via realm escape remains the documented host-reachability boundary, tracked by the realm-local intrinsics follow-up. * fix(core): stale-suspension generation token; cover BigInt toString - Suspension signals capture ctx.suspensionGeneration when scheduled and no-op if the session resumed past that boundary. The harmful interleaving was already unreachable (queue items are deleted on consume, completion writes state synchronously, nextTick precedes timers) — the token turns those ordering facts into an explicit invariant. - The BigInt reducer calls .toString() on primitives from host code, which resolves on host BigInt.prototype: its identity joins the host dispatch check, and the VM BigInt.prototype is frozen besides. * feat(core): deterministic sandbox hardening - crypto.subtle.digest computes synchronously via node:crypto: byte-identical values, promise settles on a deterministic microtask, full BufferSource validation (internal-slot view reads, SAB rejection) - Atomics.waitAsync (a wall-clock timer), async WebAssembly compilation, WeakRef, and FinalizationRegistry are removed from the sandbox — wall clock and GC observation are unreplayable; sync WebAssembly constructors remain - freezeSerializationIntrinsics pins the universal dispatch surfaces: Object.prototype/Array.prototype/Function.prototype are frozen (every missed property read and hasInstance lookup terminates there) and serialization-referenced global bindings are non-writable. Value-type prototypes and constructor statics stay patchable so polyfills (Temporal's Date.prototype.toTemporalInstant, core-js Set.prototype .union / Object.groupBy) keep working — the retained-input gate verifies the members serialization executes per boundary instead. Groundwork for retained-VM replay (#2990). * feat(core): retain the workflow VM across inline steps (primitive args) Keeps the suspended workflow VM, its events consumer, and the paused async stack alive across inline step executions within one invocation. Each loop iteration appends only the newly written events instead of replaying the entire event log in a fresh VM, so step-to-step overhead stays flat as runs grow. - WorkflowSession wraps executeWorkflow: suspended sessions expose resume(events) which appends to the retained EventsConsumer and lets the parked run() continuation settle; any divergence (unexpected suspension shape, consumer error) demotes to full replay permanently - Retention is gated per boundary: only suspensions whose queued step inputs are all primitives (null/undefined/boolean/number/string) are retainable, because serializing primitives executes no workflow code; a follow-up widens this to plain data and standard built-ins - Suspensions with hooks, waits, or attributes always fall back - A suspension generation token invalidates stale timer callbacks from an abandoned suspension so they cannot advance a resumed VM - WORKFLOW_RETAINED_VM=0 kill switch; telemetry records workflow.execution.mode = replay | retained Part 2 of the retained-VM stack (#2990); requires the determinism hardening in part 1. * chore: retrigger vercel deployments * Drop serialization intrinsic freezing from the sandbox The retained-VM passivity design moved from pinning/verifying the sandbox surfaces serialization dispatches on to injecting hardened operations into devalue itself (with taint-based de-opt), so freezing Object/Array/Function prototypes and pinning global bindings is no longer needed. Keep only the determinism hardening (sync digest, removal of wall-clock/GC-observing APIs). * Document and lock in why async crypto.subtle methods cannot break quiescence The remaining async subtle methods reject immediately through the crypto proxy (brand check — the receiver is not a real SubtleCrypto), so they can never mint a host-timing promise. Narrow the quiescence comment to what the code actually enforces and add a test so the unreachability is not silently "fixed" later. * simplify sandbox hardening: lean digest input conversion, async digest, explicit subtle throwers * simplify retention: single decision site in suspension catch, steps-only allow-list gate, drop prepareForRetention param * mark sandbox API removals as a major change * simplify retention further: one staleness mechanism (generation bump on suspend), whole predicate in canRetainWorkflowSession, lazy hook/wait scan, prewarm on resume path * simplify session API and tests: replace executeWorkflow overloads with replayWorkflow/resumeWorkflow, drop low-value events-consumer tests, compact session and retained-loop tests * add parallel-batch retention test (sibling signal absorption) and document the unguarded-signaler invariant * simplify workflow.ts types: 5 named types (WorkflowResult/WorkflowResumeResult), async resume(), rename runtime local to retainedSession * add retention-interleaving e2e (retained/demoted/wait/hook boundaries), drop session telemetry test * discard the retained session on every in-process 412 restart Review finding (both panel reviewers): restartReplayInProcess — added on main by #3145 while this branch was in flight — reset the cached log but not the parked VM session. Any stale-snapshot continue then resumed a session belonging to the discarded log: after a run_completed 412 the completed session's resume() throws and the run is durably failed despite having completed; after a suspension-create 412 the session is resumed without ever passing the retention decision, bypassing both the WORKFLOW_RETAINED_VM kill switch and the step-input gate. A restart now always falls back to a fresh replay. Regression test injects a 412 on run_completed and proves fresh-replay completion (red without the fix). * review round 2: set suspensionGeneration in typed test harness contexts; correct the open-hook/wait scan comment (this suspension's writes are not merged into the cached log — non-step suspensions never reach the scan) * simplify pass: reuse once() from @workflow/utils for the open-hook/wait memo; drop optional-chaining that contradicted the surrounding guards --------- Co-authored-by: Peter Wielander <mittgfu@gmail.com>
2 participants
Summary
Adds in-process VM continuation — on by default, kill switch
WORKFLOW_VM_CONTINUATION=0. When active, the inline replay loop keeps a suspended workflow VM alive across iterations and feeds newly-appended events into it, instead of rebuilding thevm.Contextand replaying the event log from event 0 on every pass.Default-on so CI lanes and benchmarks actually exercise and measure the new path. The kill switch restores the previous rebuild-and-replay behavior exactly.
This is the "cache the VM" idea, scoped to the only place it's sound: within a single invocation's inline loop, where the same process owns the appended writes so the durable event log stays authoritative. It does not cache a VM across process/invocation boundaries (sleeps, hooks, redelivery) — there is no live VM to resume there and a snapshot would drift from the log.
How it works
runWorkflowattaches acontinuationhandle to the thrownWorkflowSuspension.runWorkflow, it callscontinuation.resume(events):EventsConsumer.resume()adopts the authoritative event array and re-drives the still-registered step consumers, resolving the pending step promise so the same live VM advances to its next checkpoint.The step consumer already stays registered after hitting end-of-events (it only removes itself on terminal events), so resuming is just feeding it the events a fresh replay would have consumed next — the same deterministic call sequence, not restarted. RNG/clock/correlation-id state advance naturally in the live VM, exactly matching what a from-scratch replay reconstructs.
Safety (why this can't corrupt a run)
continuationunset → the loop falls back to a full replay. Those paths involve out-of-band resume/parallel invocations and delivery-barrier ordering that the replay path handles specially and that aren't validated for continuation here.resume()checks that the events already consumed by the live VM are a byte-stable prefix (byeventId) of the authoritative log. Any mismatch throwsReplayDivergenceError; the loop swallows it and falls back to a fresh replay.WORKFLOW_VM_CONTINUATION=0disables the continuation attach entirely; the suspend/complete/error control flow is byte-identical to before (verified by the full suite passing under the kill switch).Scope / limitations (intentionally not in this PR)
Verification
packages/corefull suite green with the default (on) and under the kill switch (=0): 1488 passed, 3 pre-existing expected-fails.vm-continuation.test.ts: provesresume()advances a live two-step workflow across appended events in one VM, plus the divergence guard.vm-continuation-loop.test.ts: drives a 2-step sequential workflow through the realworkflowEntrypointloop and assertscreateContext(VM construction) is called once by default (resumed per step) vs >1 under the kill switch (rebuilt per replay), and that the dehydratedrun_completedoutput is byte-identical between the two modes.pnpm --filter @workflow/core build+typecheckclean.Notes
WORKFLOW_VM_CONTINUATION=0is the per-deployment kill switch.world-localpaths.🤖 Generated with Claude Code