Windows: the suite's failure count tracks machine load, not the tree (216 of 227 failures are 5s timeouts on a green commit)

On Windows, the suite's result depends on how loaded the machine is, not on the tree. Three full runs of the same commit (cd4e65b) on the same box produced 0, 3 and 227 failures. The 227-failure run contains no product defect I can find — 216 of the 227 are Test timed out in 5000ms.

The three runs

Run	passed	failed	failed suites	exit
A (idle box)	4396	0	0	1
B	4393	3	6	1
C (loaded)	4035	227	144	1

Failure breakdown for run C, from its --reporter=json report:

216  Error: Test timed out in 5000ms.
  5  Error: Timed out after 10000ms
  2  Error: Timed out waiting for response init
  1  Error: Timed out after 12000ms
  1  Error: Hook timed out in 10000ms.
  1  Error: EPERM, Permission denied: ...\Temp\codegraph-visibility-3DiTWG
  1  AssertionError: expected 79425392 to be less than or equal to 67108864

225 of 227 are timeouts. Its stderr additionally holds 119 EPERM temp-directory teardown errors and one Error: Worker exited unexpectedly (tinypool).

Environment: Windows 11 Pro, 32 logical cores, 128 GB RAM, Node (not Bun), vitest run --maxWorkers=8.

Two specific things

1. __tests__/wal-heal.test.ts:165 fails on a wall-clock deadline, not on the behaviour it tests.

const deadline = Date.now() + 30_000;
while (walSize() > WAL_HEAL_THRESHOLD_BYTES && Date.now() < deadline) {
  await new Promise((r) => setTimeout(r, 200));
}
expect(walSize()).toBeLessThanOrEqual(WAL_HEAL_THRESHOLD_BYTES);

open() kicks the heal off in the background and the test polls for 30s. Under load the heal does not finish inside that window, and the assertion then reports a size (79,425,392 against the 64 MiB threshold) that reads like a real regression — the same byte count in two separate runs, which is exactly what makes it look deterministic rather than flaky. It is a timeout wearing an assertion's clothes. Waiting on a signal from the heal itself, rather than a wall clock, would make it honest on a slow box.

2. A fully green run still exits non-zero.

Run A is the interesting one: success: true, 0 failed tests, 0 failed suites, 4396 passed — and vitest exits 1, because a pool worker died after the last test finished. Any CI or automation that trusts the exit code over the JSON report treats a completely clean suite as a failure. I hit this in my own tooling; flagging it because the report and the exit code disagreeing is worth knowing about independently of the timeouts.

Suggestion

The 5s default testTimeout is the dominant factor — it is what turns a busy Windows box into 216 failures. Raising testTimeout (and hookTimeout) on win32, or lowering the default maxWorkers there, would make the suite report the tree instead of the load. The EPERM teardown class is separate, and #1717 covers it.

Happy to attach the three full JSON reports if useful.

Correction: the load attribution in this issue was not controlled, and one of my two specific points was overstated

I opened this with three runs of the same commit producing 0, 3 and 227 failures, and attributed the spread to machine load. Colleagues working on the same host have since shown that at least three variables were uncontrolled across those runs, and that the intervention which actually removes the dominant failure class is not the one I suggested. Correcting the record, and separating what still stands from what does not.

What was uncontrolled

No antivirus exclusions were in force. The box had zero AV exclusions for the test temp directories or the working trees until after those runs. On-access scanning touched every file the suite created. (Even now the exclusions are partial — see below.)
A background supervisor loop was spawning processes throughout. Roughly 22 bun and 19 git child processes per 40 seconds, from unrelated local tooling, fixed separately since.
%TEMP% had accumulated a very large uncleaned working set from the suite itself. See the separate finding below.

None of these were measured or held constant. "Machine load" was the variable I could see, not the variable I isolated.

The better-controlled measurement

With some AV exclusions applied and arms interleaved (parallel, serial, parallel), on the same host:

arm	wall	failed	`EPERM` lines
parallel	43s	37	50
serial	385s	36	48
parallel	44s	36	48

Parallel median 43.5s, spread 43–44s; serial n=1.

Two conclusions, and I want to be careful about the strength of each:

Zero 5s timeouts in every arm, including the first, which ran while other work was active on the box. That it held under load is what makes it track the exclusions rather than the quiet.
Serializing does not help. 36 failures vs 37, the same EPERM count, at 9x the wall time. My suggestion to lower maxWorkers on win32 is not supported — worker concurrency is not what produces the timeouts.

Limits of this measurement, stated plainly — there are four, and they matter:

It was run in a different checkout on a different branch from the three runs in the issue body, and its residual ~36 failures are that tree's own (a missing UI build — viewer assets are missing, codegraph ui exited with 1 before serving). So this is not a 227 → 36 result and I am not claiming one.
The exclusions were partial. %TEMP%\codegraph-* — the ~350,000-file pile described below, by a wide margin the largest scan target on the box — was not excluded. Only the sync tooling's scratch directories and working trees were. So these arms measured a partially excluded box, not a clean one.
The arms were not run on an idle box either. They overlapped a separate full gate run on the same host. Their failure totals therefore carry load in them and should not be quoted as clean numbers.
Serial arm is n=1.

Because of the last two points, the claim to take from this table is the timeout class, not the totals: 37/36/36 are contended numbers, whereas zero Test timed out in 5000ms in all three arms is a categorical result that contention can only have made harder to achieve. What this supports is narrow: the failure class that dominated every run in this issue disappears once the scanning is partially relieved, even under load, and does not respond to serialization at all.

The honest restatement of the headline is therefore: not "the failure count tracks machine load", but "the failure count tracked uncontrolled on-access antivirus scanning of the suite's temp and working files, which we could not separate from load." The 5s testTimeout is still the amplifier that converts slow I/O into 216 failures, so the suggestion to raise testTimeout/hookTimeout on win32 stands on its own merits — but raising it is a mitigation, and the scanning was the cause.

A third contributor: the suite leaks temp directories at scale

A colleague found %TEMP% on this host holding 49,646 codegraph-* directories, of which 44,260 are codegraph-resolution-test-*, going back four days with nothing cleaned up. A 200-directory sample measured 44 MB and 1,413 files; extrapolated that is roughly 11 GB and ~350,000 files. I want to flag that the total is an extrapolation from a sample, not a count. The per-run leak rate has since been measured at 169 directories per run, correcting an earlier static-analysis estimate of 262; at that rate the accumulated total re-derives as roughly 262 runs over five days.

This matters here because all of it was created by test runs, none of it is removed, and all of it sat inside the AV scan path during every run reported in this issue — including the three interleaved arms above, since that path is still not excluded. It is a load source that was present in, and growing across, every measurement in this thread.

It is now filed separately as #1777, with a sharper root cause than "mkdtempSync without cleanup". In __tests__/resolution.test.ts the afterEach is:

if (cg) { cg.destroy(); } else if (fs.existsSync(tempDir)) { fs.rmSync(tempDir, { recursive: true }); }

rmSync sits on the else, and cg is declared in the describe scope and never reset — so the first test that assigns cg poisons every subsequent afterEach, and destroy() closes the database without touching the directory. 32 of that file's 294 it( blocks precede the assignment and 262 follow it. The measured leak is 169 directories per run rather than the 262 that static count predicts, so not every following block reaches the poisoned path — the mechanism holds, the magnitude was over-estimated, and #1777 carries the corrected figure. Confirmed present on upstream main at 195888d, not only on our fork. Credit for both the find and the diagnosis goes to a colleague on this host; #1777 has the measured counts.

On my two specific points

Point 1 — __tests__/wal-heal.test.ts:165 — stands. A 30s wall-clock poll that reports a byte count when it expires is a timeout wearing an assertion's clothes regardless of why the box was slow. Waiting on a signal from the heal would make it honest.

Point 2 — a green run exiting non-zero — the observation stands, the cause I implied does not. The report/exit-code disagreement is real and reproducible: success: true, 0 failed tests, 0 failed suites, 4396 passed, exit 1.

I was about to attribute that specific run to a V8 crash and I cannot, so I will not. Here is the actual state of the evidence:

A V8 compiler-zone OOM — distinct from a JS heap OOM, and notably not reachable by --max-old-space-size — is reproducible. It is now filed separately as Windows: vitest pool worker killed by a V8 compiler-zone OOM, invisible in the JSON report #1779, since it is a distinct failure mode rather than a claim about these measurements. It has now appeared in two separate full runs on this host, with a byte-identical native stack, most recently in a run that also had the partial antivirus exclusions in place. Since the exclusions did not prevent it, it is not a scanning artifact; whatever the timeout class turns out to be, this is a different failure mode. The stack:
```
#
# Fatal process out of memory: Zone
#
----- Native stack trace -----
 1: 00007FF6F95A495D node::MultiIsolatePlatform::DisposeIsolate+5341
 2: 00007FF6FAA12006 v8::base::FatalOOM+54
 3: 00007FF6FA29F5E5 v8::Function::NewInstance+245
 4: 00007FF6F9C19576 v8::SnapshotCreator::GetIsolate+56566
...
13: 00007FF6FAA722EB v8::internal::compiler::CompilationDependencies::FieldTypeDependencyOffTheRecord+192907
...
17: 00007FF6FAA8519F v8::platform::DefaultJobHandle::UpdatePriority+591
18: 00007FF6F95A081E node::GetNodeReport+90430
19: 00007FF6FA2F46D6 uv_thread_detach+150
```
Frames 13 and 17 put this on a background optimizing-compiler job thread, which is why it is a Zone OOM rather than a heap one. Both runs' stderr also carries Error: Worker exited unexpectedly from tinypool at tinypool/dist/index.js:118:30.

This is invisible in the JSON report, and that is the point. A worker dying is not a test failure, so it appears in no testResults entry; vitest recycles the worker and carries on. The most recent occurrence sat inside a run that reported 4163 passed / 59 failed — a report that looks internally consistent while a worker had in fact been killed. I initially classified that run from its JSON alone and missed the crash entirely, which is a good illustration of the hazard: the report cannot be used to establish that a run was healthy. That is the same mechanism as the exit-code disagreement below.
The green run (A), the one that exits 1 over a clean report, has no stderr preserved — my tooling only began streaming vitest's stderr to disk after that run — so I cannot say it died the same way. All that survives for it is a truncated tail containing vitest's own WARNING: Some tests are still running when generating the JSON report. This is likely an internal bug in Vitest. That is consistent with a worker dying or hanging at teardown, and it is not proof of the zone OOM.

So: the zone OOM is reproducible and survives the exclusions, but whether it also explains run A specifically is unproven — A has no stderr, and I am not going to assert a cause I cannot show. What remains solid, and is purely mechanical, is that a run whose JSON report is internally green can still exit 1, so anything trusting the exit code over the report treats a clean suite as a failure — while a run whose report looks fine may equally have lost a worker. Both directions of that disagreement are untouched by antivirus, by load, and by the temp-directory leak. That pair is what I would most like fixed.

I can attach the three full JSON reports and run C's complete stderr if either would help.

Amended shortly after posting. Two claims were weakened because they were stronger than the evidence: the antivirus exclusions are partial (the large %TEMP%\codegraph-* pile is still scanned), and the three interleaved arms were themselves run on a contended box, so their failure totals are not clean numbers — the zero-timeout result is unaffected by both. One claim was strengthened: the compiler-zone OOM has since recurred in a further run, with an identical stack and with the partial exclusions in place, so it is reproducible rather than a single observation.

test(wal-heal): await the heal open() started instead of a wall clock (…

Status update from today, taking your two points and the correction in turn:

wal-heal.test.ts wall clock (point 1): fixed in test(wal-heal): await the heal open() started instead of a wall clock #2056. The test now awaits the heal pass that open() started instead of polling the file for 30s, so a slow heal can no longer show up as a size assertion. It still fails at once if open() stops kicking off the heal (checked by removing that call).
EPERM teardown class: fix: make the full test suite pass on Windows #2053 fixed the test-side causes: graphs are closed before fixture dirs are removed, and serve --mcp is spawned with the runtime flags and awaited to exit. The long-standing mcp-initialize / mcp-roots teardown failures are gone on our Windows 11 ARM VM.
Load sensitivity is real on our side too. A full run of current main (e63fe2e) today, while the host machine was heavily loaded (load average 10–38 from other work), gave 5,247 passed, 8 failed, 64 skipped (882s). Every failure was a 5s/15s timeout, an EPERM/EBUSY temp-dir teardown, or daemon-lock-refresh's new sharing-violation test, whose PowerShell handle holder can outlast the 375ms retry window when the VM is starved. Rerunning the failing files alternately on e63fe2e and the commit before today's merges gives the same failures on both, so they come from timing, not from the tree.

On point 2 (a green run exiting 1 because a pool worker died after the last test): we haven't reproduced that. Do you still see it on current main?

Leaving this open while we look at the remaining timing-sensitive tests.

fix(windows): make load-sensitive tests wait on real signals; longer …

fix(indexing): reject indexAll after a failed store instead of spinni…

More data for the "remaining timing-sensitive tests", from a second Windows box, with a set of changes that took it to zero failures in three consecutive full runs. Measured on our Windows-only fork, which tracks your main (ported through 7c01910) and runs your test suite largely unchanged; the causes below are in code we share with you.

What we see

On Windows 11 x64 (12 logical CPUs, Node 24.13, vitest 2.1.9), a plain npx vitest run fails
10-18 tests per run, different ones each time. Every failing file passes when re-run on its own or with
--maxWorkers=2 --minWorkers=1. Two causes, both environmental:

EPERM deleting a test's temp folder. Almost every suite removes its project in afterEach with
fs.rmSync(dir, { recursive: true, force: true }) straight after closing the index (for example
git-index-currency.test.ts:37, mcp-status-freshness.test.ts:32, sync.test.ts:272). On Windows a file
stays locked for a moment after close (a git child still exiting, SQLite letting go of WAL/SHM, Defender
scanning what was just written), and under load that moment grows. There is no retry, so the delete
throws EPERM, Permission denied: \\?\C:\Users\...\Temp\cg-git-currency-XXXX and fails the test. About
360 call sites pass no maxRetries; the 14 that do never failed.
Timeouts. Vitest defaults to one fork per CPU minus one (11 here), and each git-driven suite starts git,
Node children and parse workers of its own, so the CPU is heavily oversubscribed. Tests near the 5 s default
time out at random (wal-deferral, worktree-detection, sync "committed DELETE", oversize-file-not-read).
mpeg-ts-not-typescript › "a tracked TypeScript file that becomes a clip..." takes about 5.5 s on this
machine with nothing else running, so it fails on its own. A test that times out also leaves its body
running while afterEach deletes its folder, which then fails a second time with a misleading EPERM.

Measurements (full `npx vitest run`, same machine)

Configuration	Failed tests per run	Duration
before the changes (3 runs)	17, 18, 10	340-350 s, 301 s
+ `rmSync` retries + 6 forks (3 runs)	1, 2, 3	382-395 s
+ 20 s test / 30 s hook timeout on Windows (3 runs)	0, 0, 0	426-660 s (slowed by an experiment running alongside)

What fixed it for us

All three changes are Windows-only, so macOS/Linux behaviour and timing are unchanged:

Retry the cleanup in one place, not 360. Your new __tests__/setup-home-sandbox.ts (Running the test suite on Windows rewrites the developer's real ~/.claude/settings.json #2275) is the
natural home: give fs.rmSync a default of
maxRetries: 10, retryDelay: 100 when the caller chose none (we carry exactly this in that file). Node's own retry covers exactly
EBUSY/EPERM/ENOTEMPTY. Patch the CommonJS fs and call module.syncBuiltinESMExports() so every
import * as fs from 'fs' sees it:
```
if (process.platform === 'win32') {
  const cjsFs = createRequire(__filename)('fs') as typeof fs;
  const rmSync = cjsFs.rmSync;
  cjsFs.rmSync = ((target: fs.PathLike, options?: fs.RmOptions) =>
    rmSync(target, options?.maxRetries === undefined ? { ...options, maxRetries: 10, retryDelay: 100 } : options)) as typeof fs.rmSync;
  syncBuiltinESMExports();
}
```
Cap the forks in vitest.config.mts: poolOptions.forks gets maxForks: 6, minForks: 1 on win32.
(minForks is needed too, or a small selection fails "minThreads and maxThreads must not conflict" and runs
nothing.)
Longer default timeouts on win32: testTimeout: 20_000, hookTimeout: 30_000. Per-test timeouts still win.

AGENTS.md says the full suite passes on the Windows 11 ARM64 VM; we suspect the difference is load (12 threads
running 11 forks versus a smaller VM), not the architecture.

One more Windows trap we hit while chasing this, in case it saves you time: index-daemon-rebuild's
expect(isProcessAlive(pid)).toBe(false) after a rebuild failed about once in 60 runs under load. We
instrumented it: stopDaemonAt was right every time, and Windows had already recycled the daemon's PID
to a vitest worker or conhost.exe. We now check that the daemon's named pipe (its socketPath) is
gone instead of the bare PID.

Phase 2: reliability and crash safety

test(sync): compare synced graphs with an index rebuilt from scratch (#…

Thanks for checking. I haven't re-run this on current main, so I can't say whether it still happens. The run that exited 1 over a clean report was on Windows with Node, and I only started saving vitest's stderr after it, so I also can't say whether that worker died from the compiler-zone out-of-memory crash in #1779 or something else. I have no reproduction to offer right now. If it recurs with stderr saved, I'll add it here.

test: stop index-daemon-rebuild leaving a suspended daemon on Windows (…

Thanks for the careful measurements. The specific problems you found are fixed on main:

the wal-heal wall clock (test(wal-heal): await the heal open() started instead of a wall clock #2056);
worker deaths from the V8 zone crash, which also caused the "green run exits 1" (test(vitest): run pool workers with the WASM runtime flags (#1779) #2046);
the temp-dir leak (test(resolution): close graphs and connections, and always remove the temp project #1786);
the Windows teardown and timing problems (fix: make the full test suite pass on Windows #2053, fix(windows): make load-sensitive tests wait on real signals; longer pid-file retry (#1773) #2078, fix(indexing): reject indexAll after a failed store instead of spinning (#1773) #2079, test: wait for what holds an MCP test's dir before removing it #2624, test: stop index-daemon-rebuild leaving a suspended daemon on Windows #2629).

A full suite on Windows 11 now passes 8,301 of 8,301 and exits 0, matching the JSON report, with no worker deaths and no EPERM/EBUSY teardown errors. A saturated machine can still push a few tests past their 5-second timeouts, which is expected of wall-clock timeouts, so we're closing this. Please open a new issue for any failure that shows up on an idle machine.

Navigation Menu

Repository navigation

Windows: the suite's failure count tracks machine load, not the tree (216 of 227 failures are 5s timeouts on a green commit) #1773

The three runs

Two specific things

Suggestion

Correction: the load attribution in this issue was not controlled, and one of my two specific points was overstated

What was uncontrolled

The better-controlled measurement

A third contributor: the suite leaks temp directories at scale

On my two specific points

What we see

Measurements (full `npx vitest run`, same machine)

What fixed it for us

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Windows: the suite's failure count tracks machine load, not the tree (216 of 227 failures are 5s timeouts on a green commit) #1773

Description

The three runs

Two specific things

Suggestion

Activity

bompus commented on Sep 8, 2026

Correction: the load attribution in this issue was not controlled, and one of my two specific points was overstated

What was uncontrolled

The better-controlled measurement

A third contributor: the suite leaks temp directories at scale

On my two specific points

colbymchenry commented on Sep 27, 2026

ijbranch commented on Oct 2, 2026

What we see

Measurements (full npx vitest run, same machine)

What fixed it for us

bompus commented on Oct 11, 2026

colbymchenry commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Measurements (full `npx vitest run`, same machine)