[benchmarks] Add local inline-step STSO benchmark by shalabhc · Pull Request #3232 · vercel/workflow

shalabhc · 2026-07-30T21:52:43Z

Description

Adds workbench/inline-step-bench, a local benchmark for step-to-step overhead (STSO) when steps run eagerly inline — many sequential steps inside a single flow-handler invocation, with no queue hop between them. It runs against @workflow/world-local, so unlike packages/core/e2e/benchmark.test.ts it needs no deployment.

Nothing in packages/* is touched; this is workbench + docs only.

What it measures

A workflow of N sequential no-op steps. Each body stamps performance.now() on entry/exit, so STSO[i] = t0[i] − t1[i−1] is pure runtime overhead. The driver also parses the runtime's own DEBUG=workflow:runtime:debug output (Starting workflow replay / Workflow suspended) to split each gap into replay vs world I/O.

Findings at 1000 null steps (`RESULTS-1000.txt`)

The in-process loop re-executes the workflow function once per step: 1000 steps → 1 flow invocation, 1001 replays, 3 events/step in the log.
STSO(i) ≈ 7.3 ms + 0.074 ms × i. First few steps ~8 ms, step 1000 ~80 ms (~10×). Total is quadratic: 44 s, of which 35 s is replay and ~1 ms is step-body work.
Replay overtakes world I/O around step 15–20 and is ~80% of the gap past step 300.
WORKFLOW_TURBO=0 does not change the slope — turbo removes start-up round-trips, not replay.

The return-value cliff (`RESULTS-payload-600.txt`)

A step's return value doubles the slope, but not because of its size. ReplayPayloadCache memoizes the hydrated value only for primitives ≤ MAX_MEMOIZED_PRIMITIVE_LENGTH (4096); everything else re-hydrates against each fresh VM realm.

returns	payload	slope ms/step
`undefined`	0 B	0.028
a number	~1 B	0.028
a 4000-char string	4 KB	0.028
a 5000-char string	5 KB	0.074
a 4-field object	~40 B	0.072
a 40-field object	~600 B	0.077

A 4 KB string is as cheap as returning nothing; 25% more string is 2.6× the slope; a 40-byte object costs the same as a 5 KB string; 15× the object payload moves it 7%. The penalty is mostly the per-hydration reviver-table construction in deserializePreparedReplayPayload (getWorkflowRevivers(global) allocates a fresh closure table per step per replay), not decoding.

How did you test your changes?

This is a benchmark, so the artifact is the test run. Committed outputs are full runs on the machine described at the top of each file:

RESULTS-1000.txt — 5 scenarios × 1000 steps
RESULTS-payload-600.txt — 6 scenarios × 600 steps

Correctness self-checks the harness asserts inline on every run: exactly one flow-handler invocation, exactly N step bodies each executed once, 3 events/step, and loopIteration reaching N+1. npx biome check workbench/inline-step-bench/ exits 0.

Warning

Reviewer caveats, please push back on these:

The absolute ms are machine- and world-specific. world-local is a filesystem world; its fixed per-step term is not a Vercel latency prediction, and its other term drifts up because the inline-delta cursor query readdirs a directory of 3000 event files. Only the slopes and ratios are portable. Committed benchmark output tends to get quoted without its caveats — happy to strip the absolute numbers down to slopes if preferred.
runScenario in bench.mjs trips noExcessiveCognitiveComplexity (34 vs 15). It's warn-level so lint passes, but I can split the reporting out.
Should this live in workbench/ at all, or be folded into packages/core/e2e/benchmark.test.ts as a local-world mode?

PR Checklist - Required to merge

📦 pnpm changeset was run to create a changelog for this PR — empty changeset (workbench app only)
🔒 DCO sign-off passes
📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

🤖 Generated with Claude Code

Measures step-to-step overhead for steps that run eagerly inline — many sequential steps inside a single flow-handler invocation, no queue hop. Runs against @workflow/world-local, so it needs no deployment (unlike packages/core/e2e/benchmark.test.ts). Findings at 1000 null steps (see RESULTS-1000.txt): - The in-process loop re-executes the workflow function once per step: 1000 steps => 1 flow invocation, 1001 replays, 3 events/step in the log. - STSO(i) ~= 7.3 ms + 0.074 ms * i. First few steps ~8 ms, step 1000 ~80 ms (~10x). Total is quadratic: 44 s, of which 35 s is replay and ~1 ms is step-body work. - Replay overtakes world I/O around step 15-20 and is ~80% of the gap past step 300. A step's return value doubles the slope, but not because of its size (RESULTS-payload-600.txt). ReplayPayloadCache memoizes the hydrated value only for primitives <= MAX_MEMOIZED_PRIMITIVE_LENGTH; everything else re-hydrates against each fresh VM realm. So a 4000-char string costs the same as returning nothing (0.028 ms/step) while a 5000-char string costs 0.074, and a 40-byte object costs the same as a 5 KB string. 15x the object payload moves the slope 7% — the penalty is mostly the per-hydration reviver-table construction, not decoding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

changeset-bot · 2026-07-30T21:52:49Z

🦋 Changeset detected

Latest commit: 448ee39

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2026-07-30T21:52:50Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview	Jul 30, 2026 9:56pm
example-nextjs-workflow-webpack	Ready	Preview	Jul 30, 2026 9:56pm
example-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-astro-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-express-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-fastify-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-hono-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-nestjs-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-nitro-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-nuxt-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-sveltekit-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-tanstack-start-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workbench-vite-workflow	Ready	Preview	Jul 30, 2026 9:56pm
workflow-docs	Ready	Preview, v0	Jul 30, 2026 9:56pm
workflow-swc-playground	Ready	Preview	Jul 30, 2026 9:56pm
workflow-tarballs	Ready	Preview	Jul 30, 2026 9:56pm
workflow-web	Ready	Preview	Jul 30, 2026 9:56pm

github-actions · 2026-07-30T21:53:05Z

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (2 failed)

nextjs-webpack (1 failed):

outputStreamWorkflow no startIndex (reads all chunks)

nitro (1 failed):

outputStreamWorkflow - getTailIndex and getChunks getTailIndex returns correct index after stream completes

📦 Local Production (1 failed)

nextjs-webpack-stable (1 failed):

webhookWorkflow | wrun_41KYTGBRYR0GJPK0JW25C7QF23

📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

outputStreamWorkflow positive startIndex (skips first chunk)

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	1453	2	239	1694
✅ 💻 Local Development	1486	0	208	1694
❌ 📦 Local Production	1620	1	227	1848
✅ 🐘 Local Postgres	1621	0	227	1848
✅ 🪟 Windows	154	0	0	154
❌ 📋 Other	1019	1	212	1232
✅ vercel-multi-region	27	0	0	27
Total	7380	4	1113	8497

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro	126	0	28
✅ example	126	0	28
✅ express	126	0	28
✅ fastify	126	0	28
✅ hono	126	0	28
✅ nextjs-turbopack	151	0	3
❌ nextjs-webpack	150	1	3
❌ nitro	125	1	28
✅ nuxt	126	0	28
✅ sveltekit	145	0	9
✅ vite	126	0	28

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable	128	26
✅ express-stable	128	26
✅ fastify-stable	128	26
✅ hono-stable	128	26
✅ nextjs-turbopack-canary	135	19
✅ nextjs-turbopack-stable	154	0
✅ nextjs-webpack-stable	154	0
✅ nitro-stable	128	26
✅ nuxt-stable	128	26
✅ sveltekit-stable	147	7
✅ vite-stable	128	26

❌ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable	128	0	26
✅ express-stable	128	0	26
✅ fastify-stable	128	0	26
✅ hono-stable	128	0	26
✅ nextjs-turbopack-canary	135	0	19
✅ nextjs-turbopack-stable	154	0	0
✅ nextjs-webpack-canary	135	0	19
❌ nextjs-webpack-stable	153	1	0
✅ nitro-stable	128	0	26
✅ nuxt-stable	128	0	26
✅ sveltekit-stable	147	0	7
✅ vite-stable	128	0	26

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable	128	26
✅ express-stable	128	26
✅ fastify-stable	128	26
✅ hono-stable	128	26
✅ nextjs-turbopack-canary	135	19
✅ nextjs-turbopack-stable	154	0
✅ nextjs-webpack-canary	135	19
✅ nextjs-webpack-stable	154	0
✅ nitro-stable	128	26
✅ nuxt-stable	128	26
✅ sveltekit-stable	147	7
✅ vite-stable	128	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack	154	0	0

❌ 📋 Other

App	Passed	Failed	Skipped
✅ e2e-local-dev-nest-stable	128	0	26
✅ e2e-local-dev-tanstack-start-	128	0	26
✅ e2e-local-postgres-nest-stable	128	0	26
✅ e2e-local-postgres-tanstack-start-	128	0	26
✅ e2e-local-prod-nest-stable	128	0	26
✅ e2e-local-prod-tanstack-start-	128	0	26
✅ e2e-vercel-prod-nest	126	0	28
❌ e2e-vercel-prod-tanstack-start	125	1	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

📋 View full workflow run

github-actions · 2026-07-30T21:53:10Z

📊 Workflow Benchmarks

commit 448ee39 · Thu, 30 Jul 2026 22:16:00 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	273 (+30%) 🔻	1379 🔴 (+30%) 🔻	1467 🔴 (+35%) 🔻	1879 🔴 (+54%) 🔻	30
TTFS	stream	247 (-8.2%)	1400 🔴 (+30%) 🔻	1421 🔴 (+30%) 🔻	1523 🔴 (+29%) 🔻	30
TTFS	hook + stream	406 (+7.4%)	1577 🔴 (+26%) 🔻	1614 🔴 (+25%) 🔻	1668 🔴 (+5.9%)	30
STSO	1020 steps (inline)	144 (-19%) 💚	486 (+1.0%)	540 (+0.9%)	683 (±0%)	1016
STSO	1020 steps (queue-hop)	1552 (-4.7%)	3279 (+9.5%)	3279 (+9.5%)	3279 (+9.5%)	3
WO	1020 steps	420090 (+0.9%)	420090 (+0.9%)	420090 (+0.9%)	420090 (+0.9%)	1
SL	stream latency	95 (+17%) 🔻	154 🔴 (+4.8%)	177 🔴 (-3.3%)	225 🔴 (-37%) 💚	30
SO	stream overhead (text)	133 (+6.4%)	287 🔴 (+31%) 🔻	415 (+65%) 🔻	520 (+21%) 🔻	30
SO	stream overhead (structured)	123 (+6.0%)	273 🔴 (+18%) 🔻	300 (-0.7%)	1063 🔴 (+186%) 🔻	30

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 408200ms → this run 410851ms (Δ +2651ms, +1%)

  100-150 ms  ┃                         main   0  this   1    +1
  150-200 ms  █┃                        main  11  this  18    +7
  200-250 ms  ████████┃██               main  81  this  69   -12
  250-300 ms  ██████████████████┃       main 142  this 143    +1
  300-350 ms  ██████████████┃           main 112  this 116    +4
  350-400 ms  ██████████████████┃       main 136  this 140    +4
  400-450 ms  ██████████████████┃█████  main 181  this 146   -35
  450-500 ms  ████████████████████░░┃   main 153  this 171   +18
  500-550 ms  ███████████████┃          main 119  this 123    +4
  550-600 ms  █████┃                    main  44  this  49    +5
  600-650 ms  ██┃                       main  19  this  20    +1
  650-700 ms  █┃                        main   9  this  13    +4
  700-750 ms  ┃                         main   3  this   3    +0
  750-800 ms  ┃                         main   2  this   2    +0
  800-850 ms  ┃                         main   0  this   1    +1
  850-900 ms  ┃                         main   3  this   0    -3
1000-1050 ms  ┃                         main   1  this   0    -1
1200-1250 ms  ┃                         main   0  this   1    +1

1020 steps (queue-hop)

Cumulative STSO time: main 7007ms → this run 7918ms (Δ +911ms, +13%)

1500-2000 ms  ███████████┃              main 1  this 1  +0
2000-2500 ms  ┃███████████              main 1  this 0  -1
2500-3000 ms  ┃███████████              main 1  this 0  -1
3000-3500 ms  ░░░░░░░░░░░░░░░░░░░░░░░┃  main 0  this 2  +2

ℹ️ Metric definitions & methodology

_{The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000}

_{All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.}

_{Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.}

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workflow-web July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-hono-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workflow-tarballs July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-astro-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-express-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – example-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack July 30, 2026 21:53 View deployment

vercel Bot deployed to Preview – workflow-docs July 30, 2026 21:54 View deployment

vercel Bot deployed to Preview – workflow-swc-playground July 30, 2026 21:56 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[benchmarks] Add local inline-step STSO benchmark - #3232

[benchmarks] Add local inline-step STSO benchmark#3232
shalabhc wants to merge 1 commit into
mainfrom
benchmark-eager-step-overhead

shalabhc commented Jul 30, 2026

Uh oh!

changeset-bot Bot commented Jul 30, 2026

Uh oh!

vercel Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

shalabhc commented Jul 30, 2026

Description

What it measures

Findings at 1000 null steps (RESULTS-1000.txt)

The return-value cliff (RESULTS-payload-600.txt)

How did you test your changes?

PR Checklist - Required to merge

Uh oh!

changeset-bot Bot commented Jul 30, 2026

🦋 Changeset detected

Uh oh!

vercel Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

❌ Failed E2E Tests

▲ Vercel Production (2 failed)

📦 Local Production (1 failed)

📋 Other (1 failed)

E2E Test Summary

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Findings at 1000 null steps (`RESULTS-1000.txt`)

The return-value cliff (`RESULTS-payload-600.txt`)

vercel Bot commented Jul 30, 2026 •

edited

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading