Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
28915ee
test(oracle-gaps): a slow owner's kill reports as it does when it doe…
KyleAMathews Nov 14, 2026
9412eac
test(oracle-gaps): RED for the work budget, a kill past Vitest's limi…
KyleAMathews Nov 14, 2026
b71be91
test(oracle-gaps): size the work witness's loops so two copies pass t…
KyleAMathews Nov 15, 2026
54c1645
test(oracle-gaps): the model no longer predicts a timeout from a wall…
KyleAMathews Nov 15, 2026
4b063c8
fix(oracle-gaps): Vitest's per-test limit is a backstop gap, and the …
KyleAMathews Nov 15, 2026
72db011
fix(oracle-gaps): no wall-clock cut decides an outcome
KyleAMathews Nov 15, 2026
b377f9e
test(oracle-gaps): the timer witness waits on one timer, with no load…
KyleAMathews Nov 15, 2026
7aeae86
Merge origin/main into fix/gap-cli-deterministic-cuts
KyleAMathews Nov 15, 2026
e78f30f
test(oracle-gaps): RED for the outcome rule: a spin ends the mutant's…
KyleAMathews Nov 15, 2026
a3ee642
fix(oracle-gaps): the outcome rule: a clock or count cut ends the mut…
KyleAMathews Nov 15, 2026
32e60aa
fix(oracle-gaps): refined outcome rule: count and state cuts end thei…
KyleAMathews Nov 15, 2026
245a317
test(oracle-gaps): pass the history's backstop to cliArgs (plan was n…
KyleAMathews Nov 15, 2026
4c25a6e
fix(oracle-gaps): a context close past the 60 s backstop is a loud ga…
KyleAMathews Nov 15, 2026
59c4d4f
test(oracle-gaps): the H1 backstop witness never finishes and never g…
KyleAMathews Nov 15, 2026
d974b03
fix(oracle-gaps): each timeout names its cut: backstop, spin, no prog…
KyleAMathews Nov 15, 2026
f640d1c
test(oracle-gaps): the fingerprint records the constructed-only field…
KyleAMathews Nov 15, 2026
bfca29a
docs(oracle-gaps): the usage, README, index header and states name th…
KyleAMathews Nov 15, 2026
da1bf12
docs(oracle-gaps): the README and the changeset describe the backstop…
KyleAMathews Nov 15, 2026
9585d79
Merge origin/main into fix/gap-cli-deterministic-cuts
KyleAMathews Nov 15, 2026
d0df191
fix(oracle-gaps): a constructed backstop takes back as the drawn null…
KyleAMathews Nov 15, 2026
efc169c
test(oracle-gaps): the fingerprint helpers do not shadow Vitest's test
KyleAMathews Nov 15, 2026
78347cb
test(oracle-gaps): a constructed backstop of null is the full limit i…
KyleAMathews Nov 15, 2026
784e056
test(oracle-gaps): the held-state witness: a copy waiting on an unref…
KyleAMathews Nov 15, 2026
2c83b2f
fix(oracle-gaps): the idle rule is state only: no CPU or lateness con…
KyleAMathews Nov 15, 2026
8ec9d45
test(oracle-gaps): the Windows shards list the unref'd timer history
KyleAMathews Nov 15, 2026
24daca2
test(oracle-gaps): the e2e shard list names the 16 s wait and the unr…
KyleAMathews Nov 15, 2026
4768203
test(oracle-gaps): a slow wait past the horizon is no progress, not a…
KyleAMathews Nov 16, 2026
d1ff18b
test(oracle-gaps): the model's slow-wait comment cites the M entry by…
KyleAMathews Nov 16, 2026
91c15ca
test(oracle-gaps): RED for idle counting only the test's I/O
KyleAMathews Nov 16, 2026
a9df3f4
test(oracle-gaps): the runner-write witness reads its cut from the re…
KyleAMathews Nov 16, 2026
1bb0b98
fix(oracle-gaps): idle counts only the test's own I/O
KyleAMathews Nov 16, 2026
9aa6ab4
Merge remote-tracking branch 'origin/main' into fix/gap-cli-determini…
KyleAMathews Nov 16, 2026
8183c12
test(oracle-gaps): readGapLine reads a timeout's cause, judged agains…
KyleAMathews Nov 16, 2026
325aeb1
test(oracle-gaps): every run of an I/O-owner history sets its hooks
KyleAMathews Nov 16, 2026
df71570
Merge origin/main (9aed105, #103) into fix/gap-cli-deterministic-cuts
KyleAMathews Nov 16, 2026
66e4ddd
test(oracle-gaps): RED for timers judged by ownership
KyleAMathews Nov 16, 2026
f349f4a
test(oracle-gaps): determinism compares stderr gap lines as a sorted …
KyleAMathews Nov 16, 2026
fec5d43
fix(oracle-gaps): timers keep a copy busy only when the test started …
KyleAMathews Nov 16, 2026
6d35fed
test(oracle-gaps): RED for spin, no progress and the work budget as k…
KyleAMathews Nov 16, 2026
6d9cbfb
fix(oracle-gaps): a spin, no progress and the work budget kill by a b…
KyleAMathews Nov 16, 2026
27b3e43
test(oracle-gaps): RED for count-only kills; draw the deterministic-c…
KyleAMathews Nov 16, 2026
7f6b40b
fix(oracle-gaps): a spin or the work budget kills only by a count tha…
KyleAMathews Nov 16, 2026
6488cbb
Merge origin/main (fac63df, #61) into fix/gap-cli-deterministic-cuts
KyleAMathews Nov 16, 2026
88c0173
test(oracle-gaps): an authored copy never spins; a setup error beside…
KyleAMathews Nov 16, 2026
bafc072
test(oracle-gaps): RED for an authored mutant whose owner hangs
KyleAMathews Nov 16, 2026
d046d1b
fix(oracle-gaps): an authored mutant's copy is cut for no progress too
KyleAMathews Nov 16, 2026
9ecef3f
Merge origin/main (2552d6a, #121 and #128) into fix/gap-cli-determini…
KyleAMathews Nov 16, 2026
e54f769
test(oracle-gaps): the wrong-cause calibration runs the clocked spin'…
KyleAMathews Nov 16, 2026
a07e6e9
ci(oracles): rebalance the Linux e2e fixed-seed shards (P12)
KyleAMathews Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .changeset/gap-cli-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
'@tanstack/oracles': minor
---

The `oracle-gaps` CLI bounds a mutant's confirming runs, reports long runs, stops the processes it started, and writes nothing beside the tests.
The `oracle-gaps` CLI bounds a mutant's copies by counts, not by wall-clock limits, reports long runs, stops the processes it started, and writes nothing beside the tests.

- Each mutant has a budget for the runs that confirm its cut-off copies: the larger of the new `--mutant-limit` (30 s by default) and ten times its owner files' unmutated time. The confirmations run in the owner files' order, each with at most what the mutant's earlier confirmations left, so the outcome depends only on the recorded runs, not on their order or on reuse through the daemon. A mutant whose confirmations pass the budget is a `timeout` gap. With the default limits a single owner's confirmation always fits; the budget binds when several owners need long confirmations, or when `--hang-limit` is above it, which the run notes. To give one copy more time with `--hang-limit`, raise `--mutant-limit` too.
- No wall-clock limit decides an outcome. A copy that spins past its call budget, makes no progress, or whose work passes the mutant's work budget (the larger of 100,000 and ten times its owner files' unmutated work, counted in site evaluations, not in time) is killed by the built-in law `bounded work` or `no progress` on its test's route. A copy makes no progress when none of its test's own timers is due before its backstop and none of its test's own I/O is pending; timers and I/O the runner keeps for itself, such as Vitest's task-update timer or a pending write that forwards the test's console output on Windows, do not keep it busy. Only the backstop (`--hang-limit`, 60 s by default, plus 3 s per second of the file's time) is a gap: the mutant is a `timeout`, named `(cause: backstop)` on its gap line, and no later copy runs. `--mutant-limit` and `--first-limit` are retired: they are checked and noted, and change nothing.
- While one copy of a test file runs for 30 s or more, the CLI prints `busy: <mutant id> in <test file> for <n>s` every 30 s (`baseline` for a run with no mutant). Through `--daemon` the heartbeats arrive when the check ends.
- The check runs in a process group of its own, with every Vitest worker and every process a test starts, except one started detached in a session of its own. On SIGINT or SIGTERM the CLI signals the group, kills what is left 2 s later, and exits 130 or 143. Before, a SIGTERM left `forks` workers running. On Windows the run is not supervised.
- The runner's copies of the test files exist only in memory. Vitest's read of a copy for its per-file pragmas (`@vitest-environment`, `@module-tag`) is served its text, so a copy runs in its file's environment. The capture setup and Vitest configs are in a session directory of its own while it runs, and one that a stopped run left is removed by the next run. Before, the copies and configs sat beside the tests and the package's config, where a Vitest run started during a check found them and then failed when they were removed.
43 changes: 33 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,25 @@ agent.
While one copy of a test file runs for 30 s or more, the CLI prints
`oracle-gaps: busy <mutant id> in <test file> for <n>s` every 30 s. A run of many fast
copies prints none, and a check through `--daemon` prints its heartbeats
only when the check ends. The runs that confirm a mutant's cut-off copies
share one budget: the larger of `--mutant-limit` (30 s by default) and ten
times the owner files' unmutated time. They run in the owner files'
order, each with at most what its mutant's earlier confirmations left. A
mutant whose confirmations pass the budget is a `timeout` gap. With the
default limits one owner's confirmation always fits; the budget binds
when several owners need long confirmations, or when `--hang-limit` is
above it, which the run notes. To give one copy more time with
`--hang-limit`, raise `--mutant-limit` too.
only when the check ends. No wall-clock limit decides an outcome. A copy
is cut in one of three ways: by its call count, when it runs past its
call budget (`spin`); by its state, when it makes no progress (`no progress`:
none of its test's own timers is due before its backstop, and none of its
test's own I/O is pending, while timers and I/O the runner keeps for itself,
such as Vitest's task-update timer or a write that forwards the test's
console output, do not count); or by the backstop, when it runs past `--hang-limit` (60 s by default) plus
3 s per second of its file's unmutated time (`backstop`). A spin and no
progress are kills: the built-in law `bounded work` or `no progress` fails
on the route of the test that was running, so on a reproducible route the
mutant is `killed`, and on another route the next owner's copy still runs.
The copies of one mutant share a work budget, counted in site evaluations
and not in time: the larger of 100,000 and ten times their owner files'
unmutated work. The copy whose work passes it fails `bounded work` on the
routes of its tests that did that work, and no later copy runs. Only the
backstop is a gap: the mutant is a `timeout`, with `(cause: backstop)` on
its gap line and `detail` in `results.json`, and no later copy runs. A kill
in an earlier owner still decides. `--first-limit` and `--mutant-limit` are retired:
they are checked and noted, and change nothing.

The report, on stdout and in `summary.md`, leads with the baseline: each
test file's result with its test counts (ordinary files too), and for a
Expand Down Expand Up @@ -108,7 +118,7 @@ agent.
| `baseline` | `oracle-gaps: baseline pass tests/a.test.ts: 3 tests`, `oracle-gaps: baseline fail tests/b.test.ts: 1 of 2 tests failed` |
| `baseline` (decisive, before any mutant) | `oracle-gaps: baseline all 4 oracle files pass`, or `oracle-gaps: baseline 1 of 4 oracle files failed (provisional)` |
| `blocked` | `oracle-gaps: blocked <file> fails with no mutant active at <test>, so the mutants it owns wait for it: …` |
| `gap` | `oracle-gaps: gap <kind> L<line> <id> <change>` |
| `gap` | `oracle-gaps: gap <kind> L<line> <id> <change>`, and for a `timeout` ` (cause: <cause>)` |
| `progress` | `oracle-gaps: progress judged 25 of 120 mutants: 7 gaps so far, 18 not gaps` |
| `busy` | `oracle-gaps: busy <mutant id or baseline> in <test file> for <n>s` |
| `note` | `oracle-gaps: note …`: the `--hang-limit` note, Vite's own messages (`note vite: …`), ordinary files left out, the provisional count, unowned counts, the results' path, and where the tests' console output went (`note test output in <path>`) |
Expand Down Expand Up @@ -166,6 +176,19 @@ agent.
watchdog has polled twice. The directory holds a `.begun` marker per batch and
a `.reached` or `.unreached` marker per hold (`.unreached` after 60 s with no
two polls).
- `ORACLE_GAPS_TEST_SAMPLES` names a directory where each worker's idle
sampler writes, to `<pid>-<threadId>`, how many times it has judged the
running test. A witness holds its own I/O open until that count rises, so
the hold is a count and not a time.
- `ORACLE_GAPS_TEST_RUNNER_WRITE` names a directory. Through each test, the
runner itself holds one write pending on a local pipe whose reader never
reads, as a write that forwards a test's output can stay pending on
win32. A copy that goes idle while that write is pending writes an
`idle-<pid>-<threadId>` marker there.
- `ORACLE_GAPS_TEST_RUNNER_TIMER` names a directory. Through each test, the
runner itself keeps one 100 ms interval running, made through the global
`setInterval` as Vitest's task-update timer is. A copy that goes idle while
it runs writes an `idle-<pid>-<threadId>` marker there.

A few endings print no done line, or one more line: a SIGKILL of the CLI,
or a signal that comes while Node is still starting the CLI, before its
Expand Down
35 changes: 23 additions & 12 deletions packages/oracles/oracle-gaps/args.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,18 @@ export const usage = `Usage: oracle-gaps --package <dir> --source <src/x.ts,...>
[--env-prefix <name>] [--daemon] [--ordinary] [--seed <n>] [--mutants <dir>]
[--out <dir>] [--only <id,id>] [--diff-base <rev>] [--test-timeout <ms>]
[--sample <n>] [--contexts <n>] [--pool threads|forks] [--config <file>]
[--first-limit <ms>] [--hang-limit <ms>] [--mutant-limit <ms>]
[--hang-limit <ms>] [--first-limit <ms>] [--mutant-limit <ms>]
oracle-gaps --stop-daemon --package <dir>
oracle-gaps --help

Defaults: --env-prefix ORACLE; --test-timeout 65000 (Vitest's per-test limit,
in ms); --hang-limit 60000 (the base of each copy's backstop, in ms, plus 3 s
per second of its file's unmutated time); --contexts the machine's parallelism
minus 2, at most 16; --pool threads (forks when a package has a native
addon); --sample and --only off; --out a directory under the package's state.
Retired: --first-limit and --mutant-limit are checked and print a note, and
change nothing. A copy has no first budget, and a mutant's work budget bounds it.

The paths in --source and --config are relative to --package. Without
--config, the run loads the Vitest config that Vitest discovers for the
package: in its directory, or else in the nearest parent directory that
Expand Down Expand Up @@ -210,17 +218,20 @@ function optionsOf(args, cwd) {
sample: args.sample ? Number(args.sample) : undefined,
// A mutant can make a test hang. Oracles that run in milliseconds do not
// need Vitest's default limits, which would make each hang cost seconds.
testTimeout: Number(args[`test-timeout`] ?? 15000),
// The base of each mutant run's time limit, in milliseconds; the limit
// adds three times the file's unmutated duration (see limitFor).
hangLimit: Number(args[`hang-limit`] ?? 10000),
// The base of a mutant run's first, short budget; a copy past it runs
// again alone with the full limit before it can be a timeout.
firstLimit: Number(args[`first-limit`] ?? 2000),
// The floor of a mutant's budget, which the runs that confirm its
// cut-off copies share (check.mjs, `budgetOf`): 30 s by default, room
// for three confirmations at the default hang limit.
mutantLimit: Number(args[`mutant-limit`] ?? 30000),
// Vitest's per-test limit, above the copies' backstop (`hangLimit` plus
// five seconds, so a copy that waits for its backstop is cut by it first).
testTimeout: Number(args[`test-timeout`] ?? 65000),
// The base of each mutant run's backstop time limit, in milliseconds; the
// limit adds three times the file's unmutated duration (see limitFor). A
// backstop reports a gap and never decides a kill (deterministic cuts,
// option A, 2026-11-14).
hangLimit: Number(args[`hang-limit`] ?? 60000),
// The `--first-limit` option is retired (option A): it is validated and
// noted, and changes nothing.
retiredFirstLimit: args[`first-limit`] !== undefined,
// `--mutant-limit` is retired too (deterministic cuts, option A): the work
// budget bounds a mutant, so it is validated and noted, and changes nothing.
retiredMutantLimit: args[`mutant-limit`] !== undefined,
// Vitest's worker pool. Threads start much faster than child processes;
// a package with a native addon in its tree runs with `forks`.
pool: args.pool ?? defaultPool(pkg),
Expand Down
Loading
Loading