Skip to content

Test-gate timeouts are still library defaults — the unit, tui and launcher projects and all 788 Testing Library waits fail correct tests under concurrent-worktree load #2323

Description

@cliffhall

Requirement

Every wall-clock budget in this repo's test gates must be a value someone chose, sized for the machine the team actually runs on: three to four concurrent agent sessions in separate worktrees, each free to run the full npm run local:gate. A budget still sitting on a Vitest or Testing Library default was sized for an idle machine, and it fails tests that are not wrong.

This is not a request to relax the bar. #1596 settled the right stance — "Do not merely raise a global timeout to hide the races" — and every fix it produced (fake timers, awaited conditions, { delay: null }) stands. That issue was about races: tests that were wrong and passed by luck. This one is about the complement: tests that are correct and deterministic but are handed a budget nobody picked. Raising a budget nobody chose does not hide a race; it stops a correct test from being cut off mid-flight.

Evidence that the two are separable is already in the tree. clients/web/src/App.test.tsx:4298-4300 carries a per-test raise whose own comment states the cause:

// Three full modal interaction sequences against the whole App tree, so this
// one runs long enough to trip the 5s default when the suite is under load.
it(
  "suppresses a second clear for another entry with the same URL, and allows one after it settles",
  { timeout: 20000 },

That is one site patched by hand. It leaves the other 341 unit test files on the same unchosen 5000ms.

Evidence

The machine

Apple M3, 4P + 4E cores, 8 logical CPUs, 24 GB. Two samples taken minutes apart during ordinary parallel work:

load averages: 60.41 68.40 56.68
load averages: 49.70 61.86 55.40

6x to 8.5x oversubscription, sustained — not a spike. At the time: 4 worktrees (v2/main, 2264, 2305, 2319), 27 node processes, 13 processes above 10% CPU, 9 matching vitest|eslint. npm run local:gate runs lint (which fans out ESLint workers per client), then four test:coverage runs under v8 instrumentation, then the smokes, then Storybook in a real Chromium. Three sessions doing that concurrently is the intended working mode, and no in-repo config can see across worktrees to throttle it.

The failures this has already produced

Every one of these was filed, diagnosed, and fixed one site at a time:

Issue Site Shape
#2292 JsonObjectInput.stories.tsx "Annotates The Offending Line" waitFor({ timeout: 5000 }) inside a 5000ms per-test budget — the wait could never win. File took 6288ms on the failing run; 3/3 green in isolation. Fixed by raising the storybook project testTimeout to 15000
#2278 SkillsScreen.stories.tsx geometry assertion Post-click geometry race; the panel has transitionDuration={0}, so what it was losing to was a React commit + layout pass under load
#2250 ServerImportJsonModal debounce guard Wall-clock race in the test's own timing
#1942 TUI App.test.tsx step-up frames Poll budget exceeded "under CI load with v8 coverage" — fixed by raising POLL_TRIES
#1742 TUI OAuth step-up Same helper, earlier round

#2292 is the clearest statement of the class: the fix was to raise a project-level budget, and it worked. The same fix has not been applied to the other four projects.

The three properties that make this worth doing once, properly

  1. A budget below its enclosing budget can never be observed. waitFor({ timeout: 5000 }) inside a 5000ms test reports as a timeout with a message naming neither the wait nor its subject. Flaky Storybook test: JsonObjectInput "Annotates The Offending Line" exceeds its 5s wait for the Ace worker under load #2292 was exactly this; AppRenderer.test.tsx:572 still is.
  2. local:gate is mandatory and a strict superset of CI (AGENTS.md). A gate that goes red on a PR whose diff cannot have caused it trains people to re-run rather than read — the same argument AGENTS.md's "Build output is never a gate target" and "Lint has no warning tier" make from other directions.
  3. Testing Library's failure message is assertion-shaped, not timeout-shaped. An asyncUtilTimeout expiry reads Unable to find an element with… — indistinguishable from a genuine product defect. That governs 868 call sites here (below) and is the single largest unchosen budget in the repo.

Enumeration

default = inherited from Vitest 4 (testTimeout: 5000, hookTimeout: 10000, teardownTimeout: 10000 — confirmed against the installed CLI's help text) or Testing Library (asyncUtilTimeout: 1000@testing-library/dom/dist/config.js:15). Chosen by nobody in this repo.

Group A — Vitest project budgets

There is no shared defaults object. vitest.shared.mts exports resolve aliases and dedupe pins only, and clients/launcher/vitest.config.ts does not import it at all. Each of the six project configs answers the timeout question independently, which is why five of them never answered it.

# Project / site testTimeout hookTimeout teardownTimeout Files Recommend
A1 web unit (clients/web/vite.config.ts:280-333) 5000 default 10000 default 10000 default 342 15000 / 30000 / 30000. The largest surface in the repo and entirely unpinned. 15000 matches the value cli and storybook independently arrived at
A2 web integration (:337-365) 30000 explicit 30000 explicit 10000 default 77 keep both; add teardownTimeout: 30000 — these suites spawn real HTTP/stdio servers and unlink filesystem-backed storage in teardown
A3 web storybook (:369-400) 15000 explicit (#2292) 10000 default 10000 default 123 keep testTimeout; add the other two. This is the precedent the rest should follow
A4 clients/cli/vitest.config.ts:16 15000 explicit 10000 default 10000 default 29 keep; add the other two. Note pool: "forks" is pinned here, so each file pays process startup
A5 clients/tui/vitest.config.ts 5000 default 10000 default 10000 default 30 15000 / 30000 / 30000. Ink + React commits under v8 coverage is the #1942 / #1742 shape
A6 clients/launcher/vitest.config.ts 5000 default 10000 default 10000 default 1 same, for consistency — a one-file project is cheap to pin and expensive to remember
A7 vitest.shared.mts add an exported frozen TIMEOUTS object and spread it into all six projects. One object is what keeps A1-A6 from drifting apart again; six hand-written triples is how they got here
A8 retry, set nowhere 0 keep 0 — see "Do not raise"

Group B — Testing Library asyncUtilTimeout (the largest unpinned surface)

# Site Current Recommend
B1 No configure({ asyncUtilTimeout }) anywhere. clients/web/src/test/setup.ts is the unit project's only setupFiles entry 1000 default, governing 638 waitFor( + 150 findBy* sites in the unit project configure({ asyncUtilTimeout: 5000 }) in src/test/setup.ts — one call, 788 sites. 5x because a contended happy-dom render is the worst-measured case, and because this becomes the binding constraint on every unit async assertion once A1 lands
B2 Same for the storybook project: 12 waitFor( + 68 findBy* across 123 story files, no configure() 1000 default decide deliberately — Storybook runs in real Chromium and A3 already gives the test 15000, so the 1000ms async default is the tighter bound there too

Trade-off to state in the PR: a genuinely-failing assertion then takes 5s rather than 1s to report. Only failing assertions pay it, and only once each — the cost is on red runs, which are already the slow path.

Group C — Budgets that can never be observed

# Site Current Enclosing Recommend
C1 clients/web/src/components/elements/AppRenderer/AppRenderer.test.tsx:572 waitFor(…, { timeout: 5000 }) unit testTimeout 5000 exactly #2292's shape, still live. Subsumed by A1; then delete the argument
C2 clients/web/src/App.test.tsx:4300 per-test { timeout: 20000 } unit testTimeout 5000 keep until A1 lands, then reduce or delete — its comment names the 5s default as the reason it exists
C3 clients/web/src/components/elements/JsonObjectInput/JsonObjectInput.stories.tsx:217 { timeout: 10000 } storybook testTimeout 15000 keep — correctly sized against its project since #2292, and the model for the rest
C4 clients/cli/__tests__/e2e.test.ts:39 hand-rolled setTimeout(… 15000) that SIGTERMs the child and rejects "E2E CLI timed out" cli testTimeout 15000 two bounds at the identical value, so which one fires is arbitrary and the two produce different messages. Lower the in-test timer (it owns the useful diagnostic and the child kill) or raise the project budget above it — but they must not be equal

Group D — Per-suite restatements that hide the shared value

# Sites Current Recommend
D1 22 trailing }, 30_000); across 9 files under clients/web/src/test/integration/ restate the integration project's own testTimeout: 30000 verbatim delete the argument and let A2 govern, so one shared raise moves all 22. Restatements are how a project default stops being the thing anyone edits
D2 clients/cli/__tests__/oauth-interactive.test.ts:107,158,250}, 30_000); raises cli's 15000 keep — genuinely different work (interactive OAuth)
D3 .../integration/mcp/inspectorClient-oauth-remote-mid-session-e2e.test.ts:209,643,734}, 15_000); lowers A2's 30000 by half, no stated reason delete the argument, or say why lower
D4 .../integration/auth/node/file-lock.test.ts — 3x 90_000, 1x 60_000, 2x 30_000, 3x 20_000; secret-store-selection.test.ts:71860_000 explicit keep the 60s/90s ones — real cross-process lock contention, and they should keep saying so. Delete the three 20_000, which lower A2

Group E — Harness poll helpers with their own budgets

# Site Budget Recommend
E1 clients/tui/__tests__/App.test.tsx:624 POLL_TRIES = 100 x 25ms tick 2500ms keep — already raised twice (#1742, #1942) and its comment explains why a high ceiling is free (a poll exits the instant the predicate holds)
E2 clients/tui/__tests__/ResourcesTab.test.tsx:34 tries = 25 x tick() (8 x 4ms) ~800ms raise to match E1's shape. Same renderer, same coverage instrumentation, one third of the budget, for no stated reason. This is the next #1942
E3 clients/tui/__tests__/logger.test.ts:13 waitForFile(timeoutMs = 2000), 20ms poll 2000ms raise — it waits on a real filesystem write
E4 clients/web/src/test/renderWithMantine.tsx:116 DEFAULT_SETTLE_MS, and the settleMs each caller passes 500ms fixed sleep — 3 live call sites, all at HEADER_ANIM_MS + 200 = 500 Do not scale the window. Raise the slack term only — see E4 detail below

E4 detail — the transition auto-settle is a fixed sleep with a silent failure mode

This one does not behave like anything else in the enumeration and is worth reading before it is touched. AGENTS.md says to read the helper's long comment before changing anything about it; this is the result of doing that.

What it is. settleTransitions(ms) is await new Promise(r => setTimeout(r, ms)) inside act (renderWithMantine.tsx:178-180), armed automatically by renderWithMantineTransitions and consumed by an import-registered afterEach. It drains the queued rAF → React commit → terminal setTimeout chain of an in-flight Mantine transition while window is still alive, so the settling setState lands on a live tree instead of throwing a post-teardown ReferenceError: window is not defined (#1760 / #1786).

Current exposure is small and fully enumerated:

Site Budget
ViewHeader.test.tsx:243, :316, :389 settleMs: TRANSITION_SETTLE_MS = HEADER_ANIM_MS + 200 = 500ms
renderWithMantine.tsx:116 DEFAULT_SETTLE_MS 500ms — currently no caller; every call site passes settleMs explicitly, and settleTransitions has no direct caller outside the helper. It is live as a contract, dead as a value

The derivation is correct — I checked the way it could have been wrong. TRANSITION_SETTLE_MS is built from the component's exported HEADER_ANIM_MS (300), so bumping the animation moves the settle, which is exactly the right shape. The plausible bug would have been the 150ms stagger: ViewHeader staggers the incoming cell by half the duration, which would make the longest chain 450ms and leave 50ms of slack rather than 200. It does not, because no enterDelay/exitDelay is passed to any of the four Transitions — the stagger is a CSS animation-delay (App.css:349, inspector-fade-slide-in 300ms ease 150ms both) and a CSS animation schedules no JS timer. The longest JS chain really is 300ms and the slack really is 200ms. Keep the derived-from-the-component pattern; it is the model the rest of the repo should copy.

Why it must not be scaled like a project budget. It is a fixed sleep, not a timeout: it always waits the full window, on every passing run, and never exits early. It therefore belongs with the ~95 sleeps in "Do not raise" and not with Groups A–C.

Why it is nevertheless load-sensitive, unlike those 95. The window is wall-clock; the work it drains is CPU-bound (rAF callbacks, a React commit, the terminal timer). Under 6–8x oversubscription the 200ms slack term is what gets eaten, not the 300ms that tracks the animation. So the two halves of the 500 have completely different relationships to load, and scaling the sum confuses them.

Why it cannot be found by re-running, which is what makes it worth pre-empting. Every other budget in this issue fails loudly, at its own site. An insufficient settle fails nothing there — the test's assertions have already passed. What happens instead is one of:

  1. setup.ts's leaked-timer net (CI flake: leaked Mantine transition timer fails the whole run (env="test" does not suppress it, contrary to AGENTS.md) #1984) cancels the still-pending frame/timer, so the transition's terminal setState never lands and nothing reports it; or
  2. on a path that net does not close, an uncaught post-teardown ReferenceError: window is not defined fails the whole run from an arbitrary innocent file — the Flaky test teardown: Mantine Transition timer fires after happy-dom window teardown (window is not defined) #1760 shape, which took a dedicated investigation to attribute the first time.

So it is invisible to "did the suite go red", and no amount of re-running localizes it. That asymmetry is the argument for spending wall-clock here that would not be justified for an ordinary sleep.

Recommendation

  • Do not raise the 500 as a unit. Split it: name the slack term as a shared constant in renderWithMantine.tsx (e.g. RAF_SLACK_MS) instead of a bare + 200 at the call site, and size that for load — 200 → 500. The animation-derived term stays untouched and keeps tracking HEADER_ANIM_MS.
  • Redefine DEFAULT_SETTLE_MS in terms of the same RAF_SLACK_MS, so the fallback and the derived call sites cannot drift apart the way A1–A6 did.
  • Cost, stated plainly: 3 tests x 300ms ≈ 0.9s added to every passing unit run. That is a real cost and the opposite call from the one made for the 95 ordinary sleeps — justified here only because this failure is silent and displaced rather than loud and local. Do not generalize it.
  • A condition wait is not available and should not be re-litigated. The helper's comment already records why: a completed enter transition leaves no DOM signal to waitFor, which is the whole reason the sleep exists. Say so in the PR so the next reader does not re-derive it.
  • Leave both container.isConnected liveness checks exactly as they are. They are synchronous structural assertions about hook ordering (cleanup() running before this settle finishes), not timing budgets — load does not affect them, and a "make it load-friendly" pass must not touch them.
  • Update the stale comment at renderWithMantine.tsx:168-169 in the same change: it justifies the fake-timer guard as throwing "rather than a 5s test-timeout hang", and that 5s is A1's unchosen default. Raising A1 makes the number wrong and makes a hypothetical deadlock 3x slower to report.

Group C check, for completeness: the auto-settle runs in an afterEach, so it is bounded by hookTimeout, not testTimeout — the one place in this enumeration where a hook budget actually governs an await. 500ms against today's 10000ms default is comfortably inside, and A1's proposed 30000 keeps it so. No "budget inside an equal budget" problem here. No action.

Group F — Smoke and script budgets (npm run smoke, local:gate stages)

# Site Current Recommend
F1 scripts/lib/announced-child.mjs:48-49 timeoutMs = 30_000, pollMs = 250 assess — bounds test-server readiness for every smoke that needs one
F2 scripts/lib/prod-web-server.mjs:192 waitForReady({ attempts: 120, intervalMs: 500 }) 60s likely keep — cold dist/ boot, generous already
F3 scripts/lib/render-smoke.mjs:31-36 DEFAULTS timeoutMs 15_000, surviveMs 2_000, exitGraceMs 5_000, drainMs 500 the model to copy — named constants, documented, and env-overridable (SMOKE_TUI_TIMEOUT_MS etc.) through normalizeMs. timeoutMs: 15_000 for a full launcher + Ink boot is the tight one
F4 scripts/lib/mcp-app-flow.mjs:177-179 goto 30_000, connect 45_000, ready 45_000 assess
F5 scripts/lib/deep-link-connect.mjs:60-61 goto 30_000, connect 45_000 assess; deliberately mirrors F4 — keep them shared, don't fork
F6 Playwright locator budgets across smoke-web-{tabs,elicitation,browser}.mjs 15 sites: 9x 30_000, 3x 45_000, 3x 15_000, 1x 5_000 (waitForLoadState("networkidle")) hoist to named constants in one place. Three files, three independently-chosen scales, no config to raise
F7 scripts/lib/pty.mjs:133 spawnSync(…, { timeout: 5000 }) 5000 assess — probes for script(1); a 5s bound on a process spawn under 8x load is thin
F8 scripts/verify-build-gate.mjs:257 10 * 60_000 keep — bounds a full Vite build

Group G — CI

# Site Current Recommend
G1 No timeout-minutes on any job or step in .github/workflows/main.yml (5 jobs: build, coverage, publish, publish-github-container-registry, and the release path) 360 min default set an explicit per-job budget — as a hung-job guard, not as a flake fix. GitHub runners are not the contended machine, and nothing in the evidence above implicates CI. sdk-watch.yml:119 already does this (timeout-minutes: 20) and is the precedent

Do not raise

A blanket increase is not the deliverable. These stay as they are:

  • Vitest retry, currently unset (0) in all six projects. A retry converts a load-induced red into a silent green, which is exactly the signal this issue exists to preserve — and it would re-open test-stability: eliminate timeout flakiness in the web test suite (zero-flake requirement) #1596 by hiding real races behind a second attempt. local:gate is the only pre-push gate; a retry there hides defects on the one check that catches them.
  • The transition auto-settle's animation-derived term (the HEADER_ANIM_MS half of the 500). Only the slack term moves; see the E4 detail. Scaling the 500 as a unit confuses a term that tracks a component constant with a term that tracks machine load.
  • The ~95 fixed sleeps in test files (setTimeout(r, N) with no condition — 10 at 400ms, 8 at 150ms, plus a long tail). These are not timeouts. Scaling them slows every passing run on an idle machine and still races on a loaded one. Replace one with a condition wait when it actually flakes; otherwise leave it. Flaky web test: ServerImportJsonModal debounce guard fails under a full parallel run #2250 is the precedent for the replace-don't-scale call.
  • file-lock.test.ts's 60s/90s budgets and verify-build-gate.mjs's 10min. Real cross-process contention and a real Vite build. They are explicit, they say why, and a shared default must never silently lower them.
  • A blanket raise of A1's testTimeout beyond 15000. 3x the default covers the measured load; past that, a genuinely hung unit test costs the whole budget to discover, 342 files over.
  • G1 as a flake remedy. Worth setting as a hung-job guard, but not justified by this evidence and not what makes this issue red.

Where the fix belongs

Shared config, committed, one place per gate. Per-developer advice ("run fewer sessions") is out of scope: parallel worktrees are the intended working mode here, and the local-dev and pre-push-gate skills already treat a worktree as the normal place to work.

  • vitest.shared.mts — a new exported frozen TIMEOUTS (A7), spread into all six project configs. clients/launcher/vitest.config.ts must start importing the shared module to get it. A per-suite }, 30000) on the handful of currently-observed sites would fix those and leave every future file exposed, which is how five of six projects drifted in the first place.
  • clients/web/src/test/setup.ts — one configure({ asyncUtilTimeout }) call reaching 788 sites (B1). Decide B2 for the storybook project at the same time.
  • scripts/lib/ — F6's 15 scattered Playwright budgets become named constants beside F3's DEFAULTS, which already has the right shape.
  • Per-suite only where the work is genuinely different: D2, D4's 60s/90s, F8. Those should keep stating their own budget and their reason.
  • Then delete the restatements (D1, D3, D4's 20_000s, C1) so the shared value is the only one anyone has to edit.

Optional escape hatch, deliberately secondary. A single INSPECTOR_TEST_TIMEOUT_SCALE (default 1) multiplied into TIMEOUTS, for someone running six sessions. normalizeMs in scripts/lib/render-smoke.mjs already establishes the pattern for reading a duration from the environment safely (Number("") is 0, Number("typo") is NaN, and setTimeout fires immediately on both). Do not ship this first. A scale factor everyone permanently pins to 3 is a raise with extra steps and an un-reviewable committed default. Commit the chosen values; add the knob only if a measured case needs it.

Cross-worktree concurrency control considered and not proposed here. The oversubscription is across runs in different worktrees, which no in-repo config can observe. An advisory lease on local:gate would fix it at the source and is a larger design call — file separately if wanted. One trap to design around if it is attempted: two sessions each running a naive "wait until the machine is clear" loop deadlock on each other, since neither ever clears.

Regression guard

Add scripts/verify-test-timeouts.mjs with a sibling scripts/verify-test-timeouts.test.mjs, wired into validate alongside the other verify:* guards. It must:

  1. Resolve each of the six Vitest projects and assert the resolved testTimeout / hookTimeout / teardownTimeout — not merely that the keys are absent from some other block. Resolved, because that is the number a test actually gets.
  2. Assert clients/web/src/test/setup.ts configures asyncUtilTimeout.
  3. Assert retry is unset or 0 everywhere, so the "do not raise" decision above is enforced rather than remembered.

Per AGENTS.md, the guard must be observed to FAIL against the unfixed config before it is trusted, and the PR body must say so. That is trivially arranged here: today it resolves testTimeout to 5000 in three of six projects and hookTimeout to 10000 in five of six.

Keep the filename exactly verify-test-timeouts.test.mjsnode --test silently skips a file its glob misses and still exits 0.

Acceptance

  • Every budget in Groups A-C is a value stated in a committed config with a comment saying what it covers, or is explicitly recorded here as "keep, default is correct".
  • The redundant restatements in Group D are deleted, so raising a project budget moves every site under it.
  • npm run local:gate passes on this machine with two other sessions running their own gate concurrently — the condition the current budgets fail.
  • verify:test-timeouts is in validate, and was observed red before the fix.
  • No test was skipped, no retry was added, and no fixed sleep was scaled.

Follow-up work identified but not in scope

Group F's assess rows and G1 need a read that this survey did not do. Fold in the cheap ones; file the rest. (Group E4 was read — see its detail section above.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

testsUnit, smoke (e2e), Storybookv2Issues and PRs for v2

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions