Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
28915ee
test(oracle-gaps): a slow owner's kill reports as it does when it doe…
KyleAMathews Nov 14, 2026
9412eac
test(oracle-gaps): RED for the work budget, a kill past Vitest's limi…
KyleAMathews Nov 14, 2026
b71be91
test(oracle-gaps): size the work witness's loops so two copies pass t…
KyleAMathews Nov 15, 2026
54c1645
test(oracle-gaps): the model no longer predicts a timeout from a wall…
KyleAMathews Nov 15, 2026
4b063c8
fix(oracle-gaps): Vitest's per-test limit is a backstop gap, and the …
KyleAMathews Nov 15, 2026
72db011
fix(oracle-gaps): no wall-clock cut decides an outcome
KyleAMathews Nov 15, 2026
b377f9e
test(oracle-gaps): the timer witness waits on one timer, with no load…
KyleAMathews Nov 15, 2026
7aeae86
Merge origin/main into fix/gap-cli-deterministic-cuts
KyleAMathews Nov 15, 2026
e78f30f
test(oracle-gaps): RED for the outcome rule: a spin ends the mutant's…
KyleAMathews Nov 15, 2026
a3ee642
fix(oracle-gaps): the outcome rule: a clock or count cut ends the mut…
KyleAMathews Nov 15, 2026
32e60aa
fix(oracle-gaps): refined outcome rule: count and state cuts end thei…
KyleAMathews Nov 15, 2026
245a317
test(oracle-gaps): pass the history's backstop to cliArgs (plan was n…
KyleAMathews Nov 15, 2026
4c25a6e
fix(oracle-gaps): a context close past the 60 s backstop is a loud ga…
KyleAMathews Nov 15, 2026
59c4d4f
test(oracle-gaps): the H1 backstop witness never finishes and never g…
KyleAMathews Nov 15, 2026
d974b03
fix(oracle-gaps): each timeout names its cut: backstop, spin, no prog…
KyleAMathews Nov 15, 2026
f640d1c
test(oracle-gaps): the fingerprint records the constructed-only field…
KyleAMathews Nov 15, 2026
bfca29a
docs(oracle-gaps): the usage, README, index header and states name th…
KyleAMathews Nov 15, 2026
da1bf12
docs(oracle-gaps): the README and the changeset describe the backstop…
KyleAMathews Nov 15, 2026
9585d79
Merge origin/main into fix/gap-cli-deterministic-cuts
KyleAMathews Nov 15, 2026
d0df191
fix(oracle-gaps): a constructed backstop takes back as the drawn null…
KyleAMathews Nov 15, 2026
efc169c
test(oracle-gaps): the fingerprint helpers do not shadow Vitest's test
KyleAMathews Nov 15, 2026
78347cb
test(oracle-gaps): a constructed backstop of null is the full limit i…
KyleAMathews Nov 15, 2026
784e056
test(oracle-gaps): the held-state witness: a copy waiting on an unref…
KyleAMathews Nov 15, 2026
2c83b2f
fix(oracle-gaps): the idle rule is state only: no CPU or lateness con…
KyleAMathews Nov 15, 2026
8ec9d45
test(oracle-gaps): the Windows shards list the unref'd timer history
KyleAMathews Nov 15, 2026
24daca2
test(oracle-gaps): the e2e shard list names the 16 s wait and the unr…
KyleAMathews Nov 15, 2026
4768203
test(oracle-gaps): a slow wait past the horizon is no progress, not a…
KyleAMathews Nov 16, 2026
d1ff18b
test(oracle-gaps): the model's slow-wait comment cites the M entry by…
KyleAMathews Nov 16, 2026
91c15ca
test(oracle-gaps): RED for idle counting only the test's I/O
KyleAMathews Nov 16, 2026
a9df3f4
test(oracle-gaps): the runner-write witness reads its cut from the re…
KyleAMathews Nov 16, 2026
1bb0b98
fix(oracle-gaps): idle counts only the test's own I/O
KyleAMathews Nov 16, 2026
e4f5c9f
scratch: run only the Windows jobs, for #99's idle-ownership check (n…
KyleAMathews Nov 16, 2026
9aa6ab4
Merge remote-tracking branch 'origin/main' into fix/gap-cli-determini…
KyleAMathews Nov 16, 2026
8183c12
test(oracle-gaps): readGapLine reads a timeout's cause, judged agains…
KyleAMathews Nov 16, 2026
325aeb1
test(oracle-gaps): every run of an I/O-owner history sets its hooks
KyleAMathews Nov 16, 2026
a0b0d27
Merge branch 'fix/gap-cli-deterministic-cuts-b' into scratch/dc-own-i…
KyleAMathews Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .changeset/gap-cli-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
'@tanstack/oracles': minor
---

The `oracle-gaps` CLI bounds a mutant's confirming runs, reports long runs, stops the processes it started, and writes nothing beside the tests.
The `oracle-gaps` CLI bounds a mutant's copies by counts, not by wall-clock limits, reports long runs, stops the processes it started, and writes nothing beside the tests.

- Each mutant has a budget for the runs that confirm its cut-off copies: the larger of the new `--mutant-limit` (30 s by default) and ten times its owner files' unmutated time. The confirmations run in the owner files' order, each with at most what the mutant's earlier confirmations left, so the outcome depends only on the recorded runs, not on their order or on reuse through the daemon. A mutant whose confirmations pass the budget is a `timeout` gap. With the default limits a single owner's confirmation always fits; the budget binds when several owners need long confirmations, or when `--hang-limit` is above it, which the run notes. To give one copy more time with `--hang-limit`, raise `--mutant-limit` too.
- A mutant's copies share a work budget, counted in site evaluations and not in time: the larger of 100,000 and ten times their owner files' unmutated work. A mutant past it is a `timeout` gap. No wall-clock limit decides an outcome. A copy cut by its call count (`spin`) or by no progress ends only itself, a copy cut by the backstop (`--hang-limit`, 60 s by default, plus 3 s per second of its file's time) ends the mutant's later copies, and every timeout names its cause in `results.json` and on its gap line. A copy makes no progress when no timer of it is due before its backstop and none of its test's own I/O is pending; I/O the runner makes for itself, such as a pending write that forwards the test's console output on Windows, does not keep it busy. `--mutant-limit` and `--first-limit` are retired: they are checked and noted, and change nothing.
- While one copy of a test file runs for 30 s or more, the CLI prints `busy: <mutant id> in <test file> for <n>s` every 30 s (`baseline` for a run with no mutant). Through `--daemon` the heartbeats arrive when the check ends.
- The check runs in a process group of its own, with every Vitest worker and every process a test starts, except one started detached in a session of its own. On SIGINT or SIGTERM the CLI signals the group, kills what is left 2 s later, and exits 130 or 143. Before, a SIGTERM left `forks` workers running. On Windows the run is not supervised.
- The runner's copies of the test files exist only in memory. Vitest's read of a copy for its per-file pragmas (`@vitest-environment`, `@module-tag`) is served its text, so a copy runs in its file's environment. The capture setup and Vitest configs are in a session directory of its own while it runs, and one that a stopped run left is removed by the next run. Before, the copies and configs sat beside the tests and the package's config, where a Vitest run started during a check found them and then failed when they were removed.
8 changes: 8 additions & 0 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ jobs:
# 21 minutes when they ran as one job. Each job should stay well under the
# 10-minute limit; pr-shards.mjs records how each suite is split.
test:
if: false
name: Test
runs-on: ubuntu-latest
steps:
Expand All @@ -46,6 +47,7 @@ jobs:
- run: pnpm run test:publint
- run: pnpm run test:sherif
test-eslint-plugin:
if: false
name: Test eslint-plugin-oracles
runs-on: ubuntu-latest
steps:
Expand All @@ -60,6 +62,7 @@ jobs:
- run: pnpm run build
- run: pnpm --filter @tanstack/eslint-plugin-oracles run test
test-oracle-framework:
if: false
name: Framework self-oracle (${{ matrix.shard }}/8)
runs-on: ubuntu-latest
strategy:
Expand All @@ -81,6 +84,7 @@ jobs:
env:
SHARD: ${{ matrix.shard }}
test-oracle-gaps-fast:
if: false
name: Gap report oracles (${{ matrix.shard }}/3)
runs-on: ubuntu-latest
strategy:
Expand All @@ -106,6 +110,7 @@ jobs:
env:
SHARD: ${{ matrix.shard }}
test-oracle-gaps-end-to-end:
if: false
name: Gap report end-to-end oracle (fixed seed, ${{ matrix.shard }}/3)
runs-on: ubuntu-latest
strategy:
Expand All @@ -126,6 +131,7 @@ jobs:
env:
SHARD: ${{ matrix.shard }}
test-vitest-5:
if: false
name: Test on Vitest 5 (${{ matrix.suite }})
# The package's vitest peer range includes 5; this job is the evidence.
# The unit tests and each self-oracle shard are their own jobs.
Expand Down Expand Up @@ -159,6 +165,7 @@ jobs:
env:
SUITE: ${{ matrix.suite }}
preview:
if: false
name: Publish preview
# Only a commit that passes every check gets an installable preview.
needs:
Expand Down Expand Up @@ -190,6 +197,7 @@ jobs:
# npm token, so npm trusted publishing does not apply.
- run: pnpm dlx pkg-pr-new@0.0.92 publish --pnpm './packages/*' # zizmor: ignore[use-trusted-publishing]
zizmor:
if: false
name: zizmor
runs-on: ubuntu-latest
steps:
Expand Down
36 changes: 26 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,22 @@ agent.
While one copy of a test file runs for 30 s or more, the CLI prints
`oracle-gaps: busy <mutant id> in <test file> for <n>s` every 30 s. A run of many fast
copies prints none, and a check through `--daemon` prints its heartbeats
only when the check ends. The runs that confirm a mutant's cut-off copies
share one budget: the larger of `--mutant-limit` (30 s by default) and ten
times the owner files' unmutated time. They run in the owner files'
order, each with at most what its mutant's earlier confirmations left. A
mutant whose confirmations pass the budget is a `timeout` gap. With the
default limits one owner's confirmation always fits; the budget binds
when several owners need long confirmations, or when `--hang-limit` is
above it, which the run notes. To give one copy more time with
`--hang-limit`, raise `--mutant-limit` too.
only when the check ends. No wall-clock limit decides an outcome. A copy
is cut in one of three ways: by its call count, when it runs past its
call budget (`spin`); by its state, when it makes no progress (`no progress`:
no timer is due before its backstop, and none of its test's own I/O is
pending, while I/O the runner makes for itself, such as forwarding the
test's console output, does not count); or by the backstop, when it runs past `--hang-limit` (60 s by default) plus
3 s per second of its file's unmutated time (`backstop`). A spin or no
progress ends only its own copy, and a backstop ends the mutant's later
copies. The copies of one mutant share a work budget, counted in site
evaluations and not in time: the larger of 100,000 and ten times their owner
files' unmutated work. Past it, the mutant is a `timeout`. A timeout names
its cause in `results.json` (`detail`) and on its gap line, as
`(cause: <cause>)`, where the cause is the first cut in owner order, or
`work budget`. A kill in another owner decides a mutant unless a backstop
ended its copies first. `--first-limit` and `--mutant-limit` are retired:
they are checked and noted, and change nothing.

The report, on stdout and in `summary.md`, leads with the baseline: each
test file's result with its test counts (ordinary files too), and for a
Expand Down Expand Up @@ -106,7 +113,7 @@ agent.
| `baseline` | `oracle-gaps: baseline pass tests/a.test.ts: 3 tests`, `oracle-gaps: baseline fail tests/b.test.ts: 1 of 2 tests failed` |
| `baseline` (decisive, before any mutant) | `oracle-gaps: baseline all 4 oracle files pass`, or `oracle-gaps: baseline 1 of 4 oracle files failed (provisional)` |
| `blocked` | `oracle-gaps: blocked <file> fails with no mutant active at <test>, so the mutants it owns wait for it: …` |
| `gap` | `oracle-gaps: gap <kind> L<line> <id> <change>` |
| `gap` | `oracle-gaps: gap <kind> L<line> <id> <change>`, and for a `timeout` ` (cause: <cause>)` |
| `progress` | `oracle-gaps: progress judged 25 of 120 mutants: 7 gaps so far, 18 not gaps` |
| `busy` | `oracle-gaps: busy <mutant id or baseline> in <test file> for <n>s` |
| `note` | `oracle-gaps: note …`: the `--hang-limit` note, Vite's own messages (`note vite: …`), ordinary files left out, the provisional count, unowned counts, the results' path, and where the tests' console output went (`note test output in <path>`) |
Expand Down Expand Up @@ -157,6 +164,15 @@ agent.
watchdog has polled twice. The directory holds a `.begun` marker per batch and
a `.reached` or `.unreached` marker per hold (`.unreached` after 60 s with no
two polls).
- `ORACLE_GAPS_TEST_SAMPLES` names a directory where each worker's idle
sampler writes, to `<pid>-<threadId>`, how many times it has judged the
running test. A witness holds its own I/O open until that count rises, so
the hold is a count and not a time.
- `ORACLE_GAPS_TEST_RUNNER_WRITE` names a directory. Through each test, the
runner itself holds one write pending on a local pipe whose reader never
reads, as a write that forwards a test's output can stay pending on
win32. A copy that goes idle while that write is pending writes an
`idle-<pid>-<threadId>` marker there.

A few endings print no done line, or one more line: a SIGKILL of the CLI,
or a signal that comes while Node is still starting the CLI, before its
Expand Down
35 changes: 23 additions & 12 deletions packages/oracles/oracle-gaps/args.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,18 @@ export const usage = `Usage: oracle-gaps --package <dir> --source <src/x.ts,...>
[--env-prefix <name>] [--daemon] [--ordinary] [--seed <n>] [--mutants <dir>]
[--out <dir>] [--only <id,id>] [--diff-base <rev>] [--test-timeout <ms>]
[--sample <n>] [--contexts <n>] [--pool threads|forks] [--config <file>]
[--first-limit <ms>] [--hang-limit <ms>] [--mutant-limit <ms>]
[--hang-limit <ms>] [--first-limit <ms>] [--mutant-limit <ms>]
oracle-gaps --stop-daemon --package <dir>
oracle-gaps --help

Defaults: --env-prefix ORACLE; --test-timeout 65000 (Vitest's per-test limit,
in ms); --hang-limit 60000 (the base of each copy's backstop, in ms, plus 3 s
per second of its file's unmutated time); --contexts the machine's parallelism
minus 2, at most 16; --pool threads (forks when a package has a native
addon); --sample and --only off; --out a directory under the package's state.
Retired: --first-limit and --mutant-limit are checked and print a note, and
change nothing. A copy has no first budget, and a mutant's work budget bounds it.

The paths in --source and --config are relative to --package. Without
--config, the run loads the Vitest config that Vitest discovers for the
package: in its directory, or else in the nearest parent directory that
Expand Down Expand Up @@ -210,17 +218,20 @@ function optionsOf(args, cwd) {
sample: args.sample ? Number(args.sample) : undefined,
// A mutant can make a test hang. Oracles that run in milliseconds do not
// need Vitest's default limits, which would make each hang cost seconds.
testTimeout: Number(args[`test-timeout`] ?? 15000),
// The base of each mutant run's time limit, in milliseconds; the limit
// adds three times the file's unmutated duration (see limitFor).
hangLimit: Number(args[`hang-limit`] ?? 10000),
// The base of a mutant run's first, short budget; a copy past it runs
// again alone with the full limit before it can be a timeout.
firstLimit: Number(args[`first-limit`] ?? 2000),
// The floor of a mutant's budget, which the runs that confirm its
// cut-off copies share (check.mjs, `budgetOf`): 30 s by default, room
// for three confirmations at the default hang limit.
mutantLimit: Number(args[`mutant-limit`] ?? 30000),
// Vitest's per-test limit, above the copies' backstop (`hangLimit` plus
// five seconds, so a copy that waits for its backstop is cut by it first).
testTimeout: Number(args[`test-timeout`] ?? 65000),
// The base of each mutant run's backstop time limit, in milliseconds; the
// limit adds three times the file's unmutated duration (see limitFor). A
// backstop reports a gap and never decides a kill (deterministic cuts,
// option A, 2026-11-14).
hangLimit: Number(args[`hang-limit`] ?? 60000),
// The `--first-limit` option is retired (option A): it is validated and
// noted, and changes nothing.
retiredFirstLimit: args[`first-limit`] !== undefined,
// `--mutant-limit` is retired too (deterministic cuts, option A): the work
// budget bounds a mutant, so it is validated and noted, and changes nothing.
retiredMutantLimit: args[`mutant-limit`] !== undefined,
// Vitest's worker pool. Threads start much faster than child processes;
// a package with a native addon in its tree runs with `forks`.
pool: args.pool ?? defaultPool(pkg),
Expand Down
Loading
Loading