QuickJS engine: inline step execution + WASM module caching - #3049
QuickJS engine: inline step execution + WASM module caching#3049TooTallNate wants to merge 19 commits into
Conversation
|
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (1 failed)vite-quickjs (1 failed):
💻 Local Development (2 failed)astro-stable-node (1 failed):
express-stable-node (1 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
❌ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
af8a0d6 to
732019f
Compare
…-safe requeue, stable PRNG seed
… getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard
…les; harden step-listing e2e assertions against eventually-consistent reads
…l column can lag terminal status)
… assertions (analytics listing can omit attempt entirely)
…ed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM.
…import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap.
…tion + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green.
… WASM module caching
…loop event ceiling - Inline steps now claim via a lazy step_started carrying the input (step_created deferred, atomic create-claim in the world), with ownerMessageId stamped and authoritativeAttempt=1 — a concurrent invocation racing on the same fresh step loses with EntityConflictError and skips instead of both bare-starting the step and double-running the body. This also removes the stepsCreatedByUs set, whose 'created by us' invariant didn't survive the swallowed create-race conflict; redelivery backstops now key on hasCreatedEvent. - dispatchPendingOps' createdAttributeEvent/createdGetConflictHook signals are consumed again: when the loop exits suspended without ever reading back a self-written attr_set / getConflict hook_created (eventually-consistent listing lag), the entrypoint requeues immediately instead of parking the run awaiting_external with its unblocking event already written. - The server-supplied event ceiling is re-checked at the top of every continuation-loop turn (seenEventIds.size), so a single invocation fanning out inline can no longer grow the log arbitrarily past the operator's limit. The quickjs dispatch in runtime.ts converts MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the guard's throw previously nacked forever, parking runaway runs in 'running'. - Documented the deliberate decision that the platform function timeout is the only bound on inline chaining (budget parked per batch), matching the node engine.
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 152131ms → this run 126888ms (Δ -25243ms, -17%) 1020 steps (queue-hop) Cumulative STSO time: main 2084ms → this run 3091ms (Δ +1007ms, +48%) 📜 Previous results (1)b232778Fri, 31 Jul 2026 23:32:26 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Root cause of every red quickjs CI leg on this stack is in this PR, and it's deterministic — full mechanism in the inline comment at the overflow handoff. The A/B is airtight: #3048's latest run passes 42/42 quickjs local-e2e legs; this PR and everything stacked on it pass 0/42, all on promiseRaceStressTestWorkflow with the same event-log shape (5× step_created, 3× step_started, wedged in running). Independently reproduced locally three ways, including the WORKFLOW_MAX_INLINE_STEPS=5 control passing in 22 s.
I have a validated fix on local branch pgp/quickjs-vm-perf-fix (can push): overflow enqueue moved before the cheap-progress feed, plus namespace threading and run-origin trace-carrier propagation for all entrypoint publishes. With it: 136/136 e2e under quickjs, core suite 1793 passed/3 xfail, and the WORKFLOW_MAX_INLINE_STEPS=0 kill-switch path works (it was also broken — every multi-step run wedged under it).
The rest of the red CI is environmental, not yours: every E2E Vercel Prod/Multi-Region/Benchmark (vercel, *) failure on this stack is the same HTTP 429 (api-workflow-deployment-key) from four PR heads pushed within one second (~112 concurrent Vercel legs). The benchmark "target looks systematically broken" lines are the harness's own 429 guard, not a perf signal. Stagger the reruns after the fix lands.
Remaining findings below (inline) are correctness issues in the new machinery worth addressing before merge or as immediate follow-ups; also note there is zero unit coverage for continueWithEvents/the inline loop — a 6-step fan-out test would have caught the wedge deterministically.
Verified clean, for the record: the exclusive inline claim genuinely prevents double-runs through the storage gate; the WASM module cache has no rejection-memoization bug; MAX_EVENTS_EXCEEDED converts durably; session disposal covers all exits; wait-continuation dispatch matches node.
| // Steps beyond the inline cap: their step_created was written by | ||
| // dispatch above (they are not in the lazy-claim set), so hand them | ||
| // to the queue. | ||
| const overflowSteps = freshSteps.slice(inlineCandidates.length); |
There was a problem hiding this comment.
CRITICAL — this line is unreachable on the iteration that computes the overflow set, so overflow steps are never enqueued and the run wedges forever. This is the cause of the 0/42 quickjs CI legs (promiseRaceStressTestWorkflow, 60 s timeout).
Mechanism: dispatchPendingOps above writes step_created for exactly the non-claimed overflow steps; the "cheap progress" feed then always sees those very writes as unseen events and continues the loop before reaching this handoff. Next iteration, those steps have hasCreatedEvent === true, so the !op.hasCreatedEvent predicate in freshSteps excludes them permanently. The deliveryAttempt > 1 backstop doesn't fire on first deliveries, the loop breaks on inlineCandidates.length === 0, and the exit path acks with nothing scheduled. Net: step_created with no step_started, forever. Only fan-out > MAX_INLINE_STEPS (3) breaks, which is why small tests stay green. The b232778 refactor from the sticky stepsCreatedByUs set to the live-VM predicate introduced it (the sticky set used to survive the continue). With WORKFLOW_MAX_INLINE_STEPS=0, every fresh step is overflow, so the kill-switch wedges every multi-step run too.
Fix (validated, 136/136 e2e): move this handoff to immediately after dispatchPendingOps, before the feed can restart the loop; queuedStepIds keeps it idempotent. On branch pgp/quickjs-vm-perf-fix.
There was a problem hiding this comment.
Fixed in 8f88b5e — the overflow handoff now runs in the same turn the dispatch writes those steps' step_created, BEFORE the event feed, so the feed's continue can no longer preempt it (and the steps are queued exactly when they're still classified as fresh). Verified: promiseRaceStressTestWorkflow passes under WORKFLOW_VM=quickjs, and the full e2e suite is 136/136 on the merged branch.
|
|
||
| // Feed the inline batch's terminal events into the live VM. | ||
| const newEvents = await fetchUnseenEvents(); | ||
| if (newEvents.length === 0) break; |
There was a problem hiding this comment.
Ack-without-requeue: when step terminals land late in the eventually-consistent listing, this break walks the exit path to awaiting_external, which returns undefined — an ack with nothing scheduled. pendingRequeueSignal only covers attr_set/getConflict, not step terminals this invocation just caused. This is also the amplifier that turns the overflow bug from "one late step" into "run never finishes", and fixing the overflow alone doesn't close it. Suggest folding "this invocation caused a terminal it hasn't fed back" into the requeue signal, or returning { timeoutSeconds: 0 } from this break.
There was a problem hiding this comment.
Fixed in 8f88b5e — two changes: (1) when the post-batch feed returns 0 events, the loop raises pendingRequeueSignal before breaking, so terminals this invocation caused but hasn't fed back always requeue instead of acking into awaiting_external; (2) all exit requeues (budget, elapsed wait, unread self-write) are now FRESH message enqueues rather than { timeoutSeconds } visibility-redelivery, carrying only runId so their delivery always reaches replay.
| // already-completed steps as 'skipped'. First deliveries skip this: | ||
| // the step is most likely executing in a live invocation, and a | ||
| // backstop would routinely double-run bodies. | ||
| if ((deliveryAttempt ?? 1) > 1) { |
There was a problem hiding this comment.
The backstop condition is wrong: deliveryAttempt > 1 is the common case, not a crash signal. world-local advances the attempt counter on every handled response, including the { timeoutSeconds } redeliveries this engine uses as its normal requeue idiom — so this fires backstop messages for steps actively executing inline in a live invocation. ownerMessageId is threaded in and stamped on claims, but nothing here reads it; the node engine gates on the lease (isStepOwnershipActive/stepLeaseRemainingSeconds) before requeueing. The double-run window is strictly wider than node's. Gate on lease expiry, not the attempt counter.
There was a problem hiding this comment.
Fixed in 8f88b5e — the deliveryAttempt > 1 gate is gone. The pass now mirrors the node engine's decision table (step-ownership.ts): ownership is derived host-side from every observed step_started/step_retrying (initial log + every feed, latest-wins, retrying lapses permanently); lease-active steps owned by ANOTHER message arm a DELAYED backstop for the clamped lease remainder under an epoch-scoped key (cid:backstop:<lastStartedAt>); owner redeliveries and expired/unstamped steps dispatch immediately under the bare cid (dedup handles repeats). deliveryAttempt remains only as an enter diagnostic.
| requestedAt: new Date(), | ||
| }, | ||
| { | ||
| idempotencyKey: step.correlationId, |
There was a problem hiding this comment.
Idempotency-key reuse across purposes: the same step.correlationId key covers the overflow handoff, the crash backstop, and delayed retry/throttle re-enqueues. Once a world retires the key (VQS retention TTL, world-postgres completed-keys cache), a later publish for the same step is silently dropped — wait-continuation.ts documents this exact hazard and the node engine buckets its keys. Give each purpose its own suffix. (Also: getWorkflowQueueName here at L118 drops the delivery's namespace — same class as the earlier ff400af fix; the env-var fallback doesn't save generated routes, which bake the namespace at build time.)
There was a problem hiding this comment.
Fixed in 8f88b5e — queueStepMessage now takes a required purpose (dispatch | backstop:<epoch> | retry:<n>) that buckets the idempotency key; dispatch keeps the bare correlationId so it stays mutually exclusive with the node engine's handoff of the same step. It also now takes namespace + the run-origin nextTraceCarrier (threaded from runtime.ts through runWorkflowWithQuickJS, per the same fix on #3048) — as do the hook_conflict requeue, wait continuations, and exit requeues.
| vm.evalCode(`!!globalThis.__resolvers[${cidJs}]`) | ||
| ); | ||
| const rawOutput = eventData?.result ?? eventData?.output; | ||
| if (hasResolver) { |
There was a problem hiding this comment.
The live-VM delta feed silently drops resolver-less terminal events. continueWithEvents re-scans only the delta; the step_completed/step_failed/wait_completed cases do nothing beyond markCreated when no resolver exists yet. Harmless in fresh-VM replay (replay reconstructs awaits before events arrive) — but in the live-continuation path, a concurrent invocation's terminal arriving before this VM constructs the corresponding resolver loses the value, and the await never settles. Note the asymmetry: hook_received is buffered (__hookPayloadBuffer) precisely so a payload can't outrun its resolver; steps and waits have no equivalent buffer.
There was a problem hiding this comment.
Fixed in 8f88b5e — added __terminalBuffer (mirroring __hookPayloadBuffer): when a step/wait/attr terminal is scanned with no resolver present, the host prepares the outcome (decrypts bytes) and buffers it in the VM heap; __registerResolver drains the buffer at promise construction so the await settles immediately. Covers step_completed, step_failed (both byte-pipeline and legacy shapes), wait_completed, and workflow-writer attr_set. Fresh-VM replay behavior is unchanged in ordering (settle-at-construction follows creation order along each dependency chain); determinism suites and the full e2e (136/136 under quickjs) pass.
|
Fix branch pushed: |
…hreads Merge resolution — main's #3048 finals carried into the inline-loop architecture: - namespace + run-origin nextTraceCarrier threaded through runWorkflowWithQuickJS into every publish (step handoffs, hook_conflict requeue, wait continuations, immediate requeues) - suspended-exit requeues converted to FRESH messages (never { timeoutSeconds } visibility-redelivery of the current message — the hookInput redelivery trap fixed on #3048); exit wait sweep enqueues the continuation for the soonest unscheduled wait directly - entrypoint-side hookInput materialization dropped in favor of main's engine-agnostic prologue re-ensure in runtime.ts (with #3230's (runId, resumeId) claim protocol); dispatch stays inside the replay loop's try so engine failures classify into run_failed - interrupt handler keeps the perf branch's per-burst mutable budget, with main's configurable getReplayTimeoutMs() as the ceiling Review fixes (PR #3049 threads): - CRITICAL overflow wedge: overflow steps are handed to the queue in the same turn their step_created is written, BEFORE the event feed — the feed always observes those writes and continued the loop, so the old handoff was unreachable on the only turn that classified the steps as fresh (the cause of promiseRaceStressTestWorkflow hanging in the quickjs CI legs) - backstop gating: the deliveryAttempt > 1 gate (common case on worlds that advance attempts on routine redeliveries) is replaced with the node engine's ownership decision table — lease-active steps owned by another message arm a DELAYED backstop for the lease remainder under an epoch-scoped key; owner redeliveries and expired/unstamped steps dispatch immediately under the bare-correlationId key. Ownership is derived host-side from observed step_started/step_retrying events - ack-without-requeue: inline step terminals the feed has not surfaced raise the requeue signal, so the loop never acks with durably written terminals and nothing scheduled to consume them - idempotency keys bucketed by purpose (dispatch / backstop:<epoch> / retry:<n>) so worlds that retire used keys cannot swallow a later publish for the same step - live-feed terminal buffering: step/wait/attr terminals arriving before this VM constructs the corresponding resolver are buffered (__terminalBuffer, mirroring __hookPayloadBuffer) and settle the promise at construction — the single-scan continuation path previously dropped them and the await never settled Validated: core 1888 passed, full e2e 136/136 under WORKFLOW_VM=quickjs (nextjs-turbopack dev, world-local).
Summary
PR 2 of the QuickJS VM roadmap: performance work for the
WORKFLOW_VM=quickjsengine.Live-VM inline step execution. The QuickJS entrypoint now keeps the suspended VM alive (
startQuickJSWorkflowreturns a session withcontinueWithEvents) and drives an inline continuation loop per invocation:hook.getConflict(),setAttributes(), racing sleeps — advance before any step body blocks the invocation.WORKFLOW_MAX_INLINE_STEPSsteps created by this invocation are executed inline, in parallel (executeStep+ in-process single-flight), with theReplayBudgetpaused during step bodies. Terminal events are fed back into the same live VM — no fresh-VM re-replay and no queue round-trip per step.wait_completedat the right log position while this one is busy. This is what keepsPromise.race([step, sleep])semantics.deliveryAttempt > 1triggers backstop step messages. (Ownership-lease stamping like the node engine'sownerMessageIdmodel is a noted follow-up; the interim model can rarely double-run a step body cross-instance, within the documented at-least-once step contract.)attr_set/getConflictimmediate-requeue round-trips — those now resolve in-process.Process-wide WASM module caching.
WebAssembly.compileof the QuickJS runtime (~600 KB) and its native extensions now happens once per process (shared promise), instead of on every invocation.The VM interrupt budget is now per-execution-burst (reset on each
continueWithEvents) instead of per-VM-lifetime, since sessions legitimately live for minutes across inline step bodies.Results (local dev, world-local, nextjs-turbopack)
hookWorkflowacross ~25 runs — under investigation, also being watched on the base branch)fibonacciWorkflow−10%,writableForwardedFromStepWorkflow−36%); the intended payoff is cloud worlds where each queue hop is a network round-trip. Suite wall-clock is dominated by fixed sleeps and unchanged (±2%).Notes
dispatchPendingOps) no longer queues steps at all — queueing is a caller decision (queueStepMessage), keeping drain semantics identical to the node engine'sdrainPendingQueueItems.