Skip to content

fix(oracle-gaps): no wall-clock cut decides an outcome - #99

Draft
KyleAMathews wants to merge 49 commits into
mainfrom
fix/gap-cli-deterministic-cuts
Draft

KyleAMathews wants to merge 49 commits into
mainfrom
fix/gap-cli-deterministic-cuts

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Deterministic cuts for the gap CLI. No wall-clock cut decides an outcome.

  • A copy runs until one of these happens:
    • it finishes;
    • it spins past its call budget;
    • it makes no progress;
    • its counted work passes the mutant's work budget;
    • it reaches the backstop.
  • A spin, no progress and the work budget are kills by the built-in laws bounded work and no progress, on the route of the test that ran.
  • A spin or a budget overrun kills only when the counts that decide it cannot depend on time.
  • Only the backstop, a wall-clock cut, is a gap (timeout).
  • Idle is decided by state and by ownership. A timer or an I/O handle or request keeps a copy busy only when the test's own code started it. Runner timers and runner I/O never count.

Rules (v21.2 IDs; register entries in quotes)

  • R49, R50, R3, R23, I4, I15, G6, O13, K1, K15 — M-17 "Deterministic cuts, option A" and M-24 "Spin, no progress and the work budget are kills".
  • R19a, K20, K21 — M "Count-only kills" (2026-11-16).
  • R51 — M-21 "Idle is decided by state only", M-22 "Idle counts only the test's I/O", and C-41 "Idle timers by ownership too".
  • R4, P4 — M "Randomness wins" (2026-11-16) and M "Main's determinism failures".
  • G43 — C "Test-only hooks".
  • The determinism comparison — C "Progress lines in determinism" and the C entry under P10 and P11.
  • C-38 (5) — the backstop band, the one declared wall-clock exception (K21).
  • P3, P5, P7, P9, P10, P12.

Changes

Idle (capture-setup.mjs)

  • An async_hooks init hook records which code started each timer, handle and request. A resource is the test's in either case:
    • the test's or production's code is on the stack when Node creates it;
    • the resource that triggered it is the test's.
  • Node triggers a write by the stream it writes to. So the runner's write on its own channel (Vitest forwarding output) stays the runner's, even when the test's console.log caused it.
  • The sampler keeps a copy busy only in these cases:
    • one of the test's timers is due inside the horizon;
    • one of the test's handles or requests is active.
  • Deleted:
    • the name list (quiet);
    • the listed-timer count and its beforeEach base;
    • the per-worker map of wrapped timers;
    • the setImmediate wrapper.
  • Vite's module runner is now machinery (found by the runner-timer witness). It evaluates the setup file, so the sampler's own interval had read as the test's.

Kills (classify.mjs, check.mjs)

  • A spin row and an idle row name the test that was running, its route, and whether its count could have followed the clock by then.
  • A copy cut by its call count or by no progress fails the built-in law on that route.
  • The copy whose counted work passes the work budget fails bounded work on the routes of its counted tests.
  • On a reproducible route the mutant is killed, and its later copies do not run. On another route the result is random-only or unregistered.
  • An authored mutant's owner files now get the same idle detection (R50 names no exception). Before, an authored hang ran to the backstop. An authored copy counts no calls, so it never spins.
  • A spin whose count could follow the clock names no test, so it stays a gap: timeout, cause spin.
  • The budget charges only rows that are deterministic in the unmutated runs and not clocked in the mutant's run.
  • Only the backstop gives (cause: backstop).

Determinism

readGapLine (#115)

  • It reads (cause: <cause>) and judges it equal to detail.
  • New calibration: a stdout gap line names another cause. It is bound to the history a clocked copy that spins, because a timeout line is now rare in generated histories.

Draws (M "Randomness wins")

  • Now drawn: hangMs, slowMs, loops, timerWait and pulse. Each set includes the values that the constructed histories use, so a constructed history takes back as a drawn one.
  • pendingIo switches test-only hooks, so it stays a constant (the hook exception).
  • The draws shift every later field, so the fingerprint lists are recorded anew.
  • A test that loops declares no work bound, because the model judges no work in a generated test.
  • The classify oracle draws two new effect kinds: a run cut by its call count, and a run cut by no progress.

Merges

Windows shard 7: the diagnosis

The failing history is a declaration fault: brokenOwner. It has a native addon, so it runs in the forks pool. The diagnostic was run 43781834633, job 133020579174, on scratch/dc-win-idle-log at 37b6359.

  • The copies the backstop cut (.oracle-gaps-*-a.test.ts) listed a SimpleWriteWrap, 4 × PipeWrap and 2 × Timeout. They showed io: true, base 1 and no timer due.
  • Other copies listed io: false, 4 × PipeWrap and 2 × Timeout or Immediate. They showed base 1, with timers due at 0 or −1 ms.
  • The unmutated tests/a.test.ts also had a sample with a SimpleWriteWrap and io: true. That put file a in the set of I/O files, so no copy of file a was checked for idleness. Every hang of file a then ran to the backstop. In shard 2 this also hit M163 and M165 (w_hangkill).
  • A SimpleWriteWrap is a pending stream write. On POSIX a small pipe write completes at once.
  • The IPC write that Vitest's worker makes is triggered by the channel's own pipe handle. That was measured on POSIX with a 4 MB write. So under the ownership rule, the write is the runner's.
  • The diagnostic printed only the first 12 samples, one per copy, so it does not show a hanging copy's samples through the window. The fix was then checked on Windows directly.

Evidence

Law

  • No wall-clock cut decides an outcome (R49, I15).
  • A spin past the call budget, no progress, and counted work past the work budget are built-in law kills on the owner's route (R50, R3, R23, I4, G6).
  • A count kills only when it does not depend on time (R19a, K20).
  • A copy is idle when no timer and no I/O handle or request that the test started can let it progress inside its horizon (R51).

Reach

Constructed histories, each with a fixed-seed, a random-seed and a determinism unit. All reach their states on macOS and on the pool.

History Reaches How the state is held
a copy waiting on a pipe it opened a copy waiting on I/O it opened killed the test's own pipe read is the only active I/O, released after 20 sampler samples (a count)
a copy beside the runner's pending write … cut as no progress the runner holds a 4 MiB pipe write pending; marker required
a copy beside the runner's pending timer … cut as no progress a ref'd test timer past the horizon beside a 100 ms runner interval; marker required
a clocked copy that spins a spin whose count could follow the clock timed out —
an authored mutant whose owner hangs an authored mutant killed by no progress —
  • The classify oracle adds killed by bounded work, killed by no progress, a random-only kill by a built-in law and a spin whose count could follow the clock timed out.

Reach lost and gained at the fixed seed

Fixed-seed units, fec5d43 against d046d1b:

  • e2e oracle (66 units, then 69):
  • Classify oracle:
    • Lost: the measured authored agreement-only-kill and authored unregistered-kill, not required.
    • Gained: the four states above.

RED

  • Idle ownership (91c15ca): a copy waiting on a pipe it opened cut as no progress, owed killed. a copy beside the runner's pending write: cut by backstop, owed no progress.
  • Timers (66e4ddd): a copy beside the runner's pending timer: cut by backstop, owed no progress.
  • Kills (6d35fed): on the pool, every outcome class, brokenOwner, the work budget, the runner write and the runner timer give timeout, owed killed. The classify oracle's three new histories give timeout.
  • Count-only (27b3e43): a clocked copy that spins: delete-call#0: killed, owed timeout. The classify clocked spin: a bounded-work kill, owed timeout (spin).
  • Authored hang (bafc072): the authored mutant a1 is timeout, owed killed. It was found as random seed −313313556, a declaration fault: apiLessTypescript, path 0:1:1:1.
  • readGapLine (merge with test(oracle-gaps): judge stdout gap lines against results.json, by id #115): does not read as a kind, a line, an id, and a change.

GREEN (pool)

Mutants

Each one fails the oracle.

Mutant Where it was run Result
I/O: count every request (the old name rule) macOS fails the pipe witness and the runner-write witness
I/O: count every active resource macOS fails the runner-write witness
I/O: count no I/O macOS fails the pipe witness
Timers: count every listed Timeout macOS fails the runner-timer witness
Timers: ignore owner macOS fails the runner-write and runner-timer witnesses
Timers: ignore the due time macOS fails the runner-write and runner-timer witnesses
Timers: drop the timer check (mutants 2 and 2b collapse into it) macOS fails both timer witnesses
Spin, no progress and the work budget reported as timeouts pool, 111c7c0 fails 8 classify units and 5 e2e units
A spin kills by count whatever the clock pool, 8dbe646 fails a clocked copy that spins
An authored copy without idle detection (bafc072, the RED) macOS fails an authored mutant whose owner hangs

Weight

Minified, with gzip −9 bytes in parentheses. "d1ff18b" is this PR's head before this work.

File main fac63df d1ff18b head
capture-setup.mjs 15,281 (6,233) 15,135 (6,170) 16,349 (6,631)
check.mjs 37,208 (14,824) 35,569 (14,225) 38,124 (15,157)
classify.mjs 6,781 (2,773) 6,780 (2,785) 7,049 (2,882)
  • Most of the capture-setup.mjs growth is the three test-only hooks (about 1,000 bytes).
  • The ownership rule is about +100 bytes net, after the deletions.
  • This is all test-time code; nothing reaches a user's bundle.

Async hooks cost

  • A Vitest benchmark in forks: chained setImmediate +4–40%, nextTick +30 ms per 100,000, fs.stat +13–76 ms per 5,000. Synchronous code is unchanged.
  • The hook evaluates no instrumented site. The work-law witnesses pass unchanged.

Limits

Process

  • Commit a9df3f4 was made with -c core.hooksPath=.husky by mistake, which is a hook bypass, so its pre-commit hook did not run. Its one file, packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs, passes eslint and prettier unchanged. It was pushed, so it is not amended.
  • The merge commit 9aa6ab4 has no Co-authored-by trailer.

Test plan

This pull request and its description were written by Isaac.

KyleAMathews and others added 8 commits November 14, 2026 16:37
…s not wait

Determinism under load (2026-11-13): a run's outcome must not depend on
how long a run took. A constructed history, "a slow owner's kill and
another owner's kill", has one mutant that file a's random-route test
and file b's fixed-seed test both kill. Its driver also runs the slow
twin: the same package, where a's test waits 3 s past its first budget
wherever the output differs and then judges as before. The law compares
the two reports, `randomAlso` included.

On main, a's slow copy is cut at its first budget and queued for a
confirmation, b's kill decides the mutant first, and a's random failure
is never recorded: the twin reports `randomAlso: false` where the package
reports `true`. `slowTwin` is constructed-only (no generated history
draws it; `histories` takes it back without it).

Co-authored-by: Isaac <no-reply@databricks.com>
…t, and a timer under load

Deterministic cuts, option A (register M, 2026-11-14): no wall-clock cut
decides an outcome, so a mutant's copies are judged by count, not by time.
Three laws are added to the end-to-end oracle, each RED on the parent
commit before any production change.

- The budget witness w_slowpair becomes a work-budget witness: its two
  owners' tests call the function PAIR_LOOPS times where their output
  differs. Each copy stays inside its call budget, the two together pass
  the mutant's work budget (WORK_FLOOR), so the mutant is a timeout by
  count, whatever the clock says. The model judges the budget in owner
  order, a kill before the budget runs out still deciding.
- A slow kill whose test waits 16 s, past Vitest's 15 s per-test limit, is
  killed: no limit that the CLI leaves to Vitest decides the outcome.
- A copy whose test starts a timer, spins under load, then waits for the
  timer, is killed and never cut as idle (the idle law is state, not time).

Constructed-only test fields (slowMs, loops, busy) are held by the drawn
test record at neutral constants and taken back to them by the histories'
map, so no generated history draws them (generator rule, 2026-11-14). The
constructed histories are appended at the end of the list, and the slow
owner's history moved there from the middle of it.

Co-authored-by: Isaac <no-reply@databricks.com>
…he work budget

The first RED (9412eac) lets each copy of w_slowpair make 37,500 calls
beyond its first. At three site evaluations a call that is 112,500 per copy,
past the 100,000 call budget, so main cuts each copy by count and the mutant
is a timeout on either side; the work law is not reached. At 30,000 calls a
copy makes 60,000 to 90,000 site evaluations, inside its budget, and the two
copies together pass the work budget, so main must report survived.

Co-authored-by: Isaac <no-reply@databricks.com>
…-clock budget

Register M entry "Deterministic cuts, option A" (2026-11-14): no wall-clock
cut decides an outcome; a mutant's budget is a work count, and the wall-clock
budget is only a backstop that reports a gap. The model's wall-clock clause
(two budgets: a slow owner file's confirmations past --mutant-limit predict a
timeout) is removed as the repair the M entry requires. This is not a model
change to match production: the clause's authority is the retired confirmation
path, and the M entry retires it.

Consequence, a RED on main: the constructed history "a slow kill past Vitest's
15 s limit" (a 16 s wait where its output differs) now predicts `killed`, and
main reports `timeout` for it, because main's confirmation is cut by the
mutant budget at 15 s (diagnostic on a scratch branch: the copy's outcome is
"run time limit", not Vitest's "Test timed out").

Also: a test that awaits its busy witness is async (the emitter marks it so),
so the baseline loads.

Co-authored-by: Isaac <no-reply@databricks.com>
…states are listed apart

Register M entry "Deterministic cuts, option A" (2026-11-14), with the
coordinator's ruling 3 (fallback): a test that Vitest's own limit ends
("Test timed out") is reported as a backstop gap, like a timed-out run. It
is `timeout` after any kill, and it never keeps a kill from deciding. The
previous rule read a Vitest timeout only when every failing test had one,
so a timed-out test beside a verdict-less failure was `crashed`, and a
timed-out test beside an unregistered kill was not a timeout at all. The
`every` branch and the `!timeout` filter are now redundant and removed.

Law: a run's outcome does not depend on how long a test took (determinism
under load); a clock limit may only report a gap.
RED: the classify oracle on the parent classify.mjs (HEAD before this commit,
built from the same fixtures) fails with "mutant m0: unregistered-kill, owed
timeout", and the parent's crashed/timeout split fails the same law.
Mutant: timeout checked before the kills (scratch copy): the classify oracle
fails, "mutant m0: timeout, owed killed"; the mutant is rejected.
GREEN: the classify oracle passes (276 tests). The model counts a timeout
message only in a file that loaded, as production does (a file that fails to
load gives no tests).

The e2e oracle's states "a slow owner's kill reported as when it did not
wait", "a slow kill past fifteen seconds killed" and "a copy waiting on a
timer under load killed" are listed as separate: they have their own
histories, so "every outcome class" must not enter them.

Co-authored-by: Isaac <no-reply@databricks.com>
Register M entry "Deterministic cuts, option A" (2026-11-14). A copy runs
until it finishes, goes idle, or spins; the only wall-clock limit is a
backstop that reports a gap and never decides a kill.

- No first budget and no confirmation: a copy runs once, to its end, under
  the backstop limit (hang-limit plus three seconds per second of the file's
  unmutated time). The cut marking and the confirming run are removed.
- G1 becomes a work budget: a mutant's copies, summed in their owner files'
  report order, may do at most max(WORK_FLOOR, 10 x the owners' unmutated
  calls) site evaluations (WORK_FLOOR = CALL_FLOOR, 100,000). Past it, the
  copy that crosses and every later copy is a timeout, so a kill among them
  does not count. This is deterministic because the work is a count; the
  owner order is the report's, not the completion order.
- The wall-clock mutant budget (--mutant-limit) is a backstop checked after
  all copies: no kill and elapsed over budget is a timeout; it never decides a
  kill.
- Defaults (coordinator rulings, 2026-11-14): --hang-limit 60 s, --mutant-limit
  180 s, --test-timeout 65 s (hang-limit plus 5 s). Cost to state: an unmutated
  hang that no CLI limit cuts now fails after 65 s, not 15 s; under option A an
  idle hang is cut by `no progress` in about 500 ms, so only a busy hang with no
  instrumented site waits for the backstop.
- --first-limit is validated and retired: it prints a note and changes nothing.
- A test that Vitest's own limit ends is a backstop gap (classify.mjs; its
  RED and mutant are in the prior commit, 4b063c8).

Correction to 4b063c8's message: that message said the parent's crashed/timeout
split fails the classify oracle. I did not run that case. The verified RED is
the one in 4b063c8's body: the parent gives "unregistered-kill" where the model
owes "timeout".

Co-authored-by: Isaac <no-reply@databricks.com>
… in the test

The counter-witness (register, 2026-11-13 "Determinism under load"; the
2026-11-14 entry "Deterministic cuts, option A") was a copy whose test spun
for 600 ms and then waited 2 s on a timer. Its spin kept event-loop
utilization above 5%, which alone keeps a copy from looking idle, so the
timer check never mattered: mutant 2 (elapsed-only idle, the timer checks
disabled) passed the witness. The witness now waits on one timer due 3 s
out, inside the horizon, with nothing else in the test. A copy with its
timer pending is never idle; only that check stands between it and an idle
cut. A loaded variant, if wanted, is a CPU hog outside the worker.

The state is "a copy waiting on a timer killed"; the constructed field is
timerWait (a constructed-only test field, taken back to null; no generated
test draws it).

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

# Conflicts:
#	packages/oracles/oracle-gaps/gap-report-model-oracle.mjs
#	packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
#	packages/oracles/scripts/pr-shards.mjs
@changeset-bot

changeset-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: a07e6e9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes changesets to release 1 package
Name Type
@tanstack/oracles Minor

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

KyleAMathews and others added 18 commits November 14, 2026 23:08
… copies

Register C entry "Deterministic cuts, outcome rule (under M option A)",
2026-11-14: a copy cut by a clock or by its call count ends the mutant's
copies; no later copy runs. The model now predicts that: a copy that spins
(a count) stops the owner files after it, so a later owner's kill does not
count and the outcome is a timeout.

Witness: a constructed history whose one function spins in the mutant (the
until loop of w_spin) with two owners, a and b. Owner a's copy spins first;
owner b's judging test would kill. Production (main at this commit) runs
b's copy after a's spin and reports killed. The model owes timeout.

RED: on this commit, the new witness fails (the CLI reports killed where the
model owes timeout; pool run dc-rule-red-*).
Limits: the model's rule covers copies cut by a count; a wall-clock cut
(the backstop and Vitest's limit) is a backstop gap under the same rule, and
its own witness is pending (finding 4, the H1 witness).

Co-authored-by: Isaac <no-reply@databricks.com>
…ant, uncharged

Register C entry "Deterministic cuts, outcome rule (under M option A)",
2026-11-14: only counts and states decide a mutant's outcome.

- A copy cut by a clock or by its call count (its own timeout, or Vitest's
  "Test timed out") is never charged, and it ends the mutant's copies: no
  later copy runs, and a kill among later copies does not count (review B2).
- A copy's work is the calls of its deterministic, unclocked tests
  (`baseline.work`, classify.mjs), and the owner files' most-working tests
  set the budget and the call budget (review B3).
- The mutant-level elapsed-seconds backstop is removed: it summed cached
  units' stored seconds, so a daemon reuse could differ from a cold run
  (review finding 3). Each copy's own wall cut remains the backstop. Whether
  --mutant-limit still has a role is open (it now only affects identity).
- Model: a copy cut by a count or a clock ends the owner files (no later file
  runs), so the model owes the same outcome.

Witness: a constructed history with two owners, each making 30,000 extra
calls; owner a's copy is under the budget, owner b's would kill but passes
the budget, so the mutant is a timeout by count. The reversed-sum mutant
(review H2, 968bfd7) is the mutant it must fail.

Not witnessed here: a spin in an earlier owner ahead of a kill in a later
one. A spin witness was tried first and removed: the later owner runs the same
spinning function, so it spins too and never kills, and the witness passed on
the parent. Finding 4's backstop witness (H1) is pending.

Co-authored-by: Isaac <no-reply@databricks.com>
…r copy, wall cuts end the mutant

Register C entry "Deterministic cuts, outcome rule (under M option A)", as
refined by the coordinator: a copy ending in "no progress" (a state) or a spin
past its call budget (a count) is a deterministic timeout for that copy. It is
charged nothing (the rows of a cut copy are not kept, and the per-copy call
budget already bounds it), and the next owner's copy still runs, so a later
kill decides. Only a wall-clock cut (the CLI backstop, or Vitest's "Test timed
out") ends the remaining copies, as a backstop gap that is never charged.

- Model: runsOf tags each cut with its kind (wall, count, state); judge ends
  the owner files only on a wall cut.
- Check: spin and idle copies settle with spin or idle set; cutByTime covers
  wall cuts only.
- --mutant-limit is retired (validated, noted, changes nothing); the driver no
  longer passes it.
- Constructed history "a backstop cut ends the copies ahead of a later kill":
  owner a's copy waits past a 2 s backstop; owner b would kill, and does not
  run. The state "a backstop cut ends the copies ahead of a later kill" is
  entered.

Open: "a declaration fault: noTypescript" (shard 3) streams 161 gap lines where
the model expects 159; a per-mutant check is pending. It is a model and CLI
disagreement, not a duplicate line (the scratch diagnostic found no repeated
line).

Co-authored-by: Isaac <no-reply@databricks.com>
…ot in scope there)

Co-authored-by: Isaac <no-reply@databricks.com>
…p, never a silent one

Review finding 5 (register L8). The 10 s close waits in closeContexts and in
replace continued silently after a close was still pending. Now each close
waits up to the 60 s backstop; a close still pending is an error that names
the backstop gap, and no replacement is warmed while the old context is still
closing. Closes that finish are unchanged.

Witness: none. A pending close needs a context whose close never settles, which
costs 60 s per run; it is a limit of this change (the coverage is by inspection
of closeWithin and its two callers).

Co-authored-by: Isaac <no-reply@databricks.com>
…oes idle

Owner a's copy now awaits a promise that never resolves, while a repeating
timer stays due inside its horizon, so the copy is never idle and no site
spins. Only the backstop cuts it, on every machine. The witness no longer
depends on whether a 12 s wait lands inside the backstop band (register
2026-11-14, outcome rule item 5).

Co-authored-by: Isaac <no-reply@databricks.com>
…ress, or work budget

A copy's cut is now recorded as its cause and carried into the mutant's timeout
detail, results.json, and the gap line (cause: ...). The model predicts the
cause of each timeout: a backstop cut (a wall-clock limit), a spin past the
copy's call budget (a count, outcome rule item 4), no progress (a state), or
the mutant's work budget. The model had no per-copy call budget, so the
work-budget history's 4-sites-per-call mutant was a spin and not a work-budget
timeout; the model now predicts that. ORACLE_GAPS_PROFILE rows carry each
copy's cause, its file's measured time, its limit and its duration.

RED: the e2e oracle, with the cause predicted and production unchanged, fails
on the backstop history (its timeout is cut by run time limit).

Co-authored-by: Isaac <no-reply@databricks.com>
…s as constants

The deterministic-cuts constructed fields are constants in every drawn record,
so no draw changes: the tests line hashes the records without them and is
main's (e9029433a23d1f10). slowMs, loops, timerWait and pulse each get a line
after the others, with its neutral value checked on every generated test, and
hangMs is a drawn constant (null where generated) with its own line. The
constructed histories #99 adds are appended after main's 26, by name and index.

Co-authored-by: Isaac <no-reply@databricks.com>
…e current cuts

--help and the index header give the defaults (--test-timeout 65000, --hang-limit
60000) and mark --first-limit and --mutant-limit retired. The README, the
changeset and the e2e law 4 text describe the backstop, the call and work
counts, and the cause each timeout names. Stale states and names are renamed
(confirmations, the first budget, the 15 s limit): a slow kill after a 16 s
wait, and a mutant's copies past its work budget. MUTANT_LIMIT_MS, unused, is
removed. The determinism twin is described as a determinism check: it passes
on its parent, so the outcome rule's RED is the classify oracle's.

Co-authored-by: Isaac <no-reply@databricks.com>
… and counts

The README's gap section and the changeset's bullet describe the counts and the
backstop, the cause each timeout names (on the gap line too), and the retired
options, in place of the confirmation budget. The classify oracle's gap-line
law (in bfca29a) accepts a timeout's cause suffix.

Co-authored-by: Isaac <no-reply@databricks.com>
Resolves the constructed list (main's 29 histories first, then the five #99 histories at 29 to 33), the drawn record (main's pauseEngages and warmReplaced, then hangMs), the take-back, the limits text (main's Windows notes, with #99's work-budget wording), and the shard lists (#99's histories in shard 4, main's test.projects line in shard 5). The fingerprint canary passes.

Co-authored-by: Isaac <no-reply@databricks.com>
…; the Windows shards list the new histories

The constructed backstop (hangMs) is a drawn constant now, so its take-back returns
the drawn value (null), which the arbitrary can take back. The fixed-seed shards
of main are kept as they are: the new histories fall into the Linux rest shard,
and the Windows partition, which is exact, lists them in shard 10.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…n the CLI arguments

Co-authored-by: Isaac <no-reply@databricks.com>
…'d timer

An unref'd timer is not among the process's active resources, so only the copy's
own pending-timer record keeps it busy (the idle rule, held state, M 2026-11-15).
The witness is a constructed history of its own; the reach state is recorded.

Co-authored-by: Isaac <no-reply@databricks.com>
…dition

A copy is idle when no counted timer is due within its horizon, no timer is listed
beyond its test's base, and no I/O handle is open (M entry 2026-11-15). Sample
lateness and the event loop's utilization no longer reset the window: a busy
synchronous or microtask loop blocks the sampler, so it cannot look idle. The
held-state witnesses (a copy waiting on a timer, ref'd or unref'd) keep the
copy busy, and the mutants that ignore a due timer are killed.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…ef'd timer history

The e2e shard check named the removed history 'a slow kill past Vitest's 15 s limit'
and had no shard for the unref'd timer history, so CI's test:oracle-gaps:shards
failed. The check now partitions the e2e tests exactly.

Co-authored-by: Isaac <no-reply@databricks.com>
@pkg-pr-new

pkg-pr-new Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@99
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@99

commit: a07e6e9

… wall-clock cut

With the idle rule decided by state (M entry, 2026-11-15), a timer due beyond the
copy's horizon does not keep it busy, so it is idle and cut as no progress. The
model's wall-clock cut for such a wait was stale; the load experiment's 12 s copies
gave the same outcome on three repeats under a concurrent run (the model owed a
backstop cut).

Co-authored-by: Isaac <no-reply@databricks.com>
…sult's detail

Co-authored-by: Isaac <no-reply@databricks.com>
@KyleAMathews

Copy link
Copy Markdown
Contributor Author

#115 adds a reader for stdout gap lines to the e2e oracle (readGapLine in gap-report-oracle.property.test.mjs). It reads oracle-gaps: gap <kind> L<line> <id> <change> and requires the line to end after its change. This PR's ' (cause: )' suffix on timeout lines will therefore fail as "does not read as a kind, a line, an id, and a change", under greppable stream. It fails loudly; it does not misread. Whichever of #99 and #115 merges second should extend readGapLine to read the suffix. The limit is recorded in the e2e oracle's limits.

KyleAMathews added a commit that referenced this pull request Nov 16, 2026
…e daemon checks too

Review of #115:
- F1: one calibration per rejection branch of the stdout gap-line check:
  a dropped line (the #102 symptom), another change, another kind, an
  unreadable line, and a line for a mutant that is not a gap, beside the
  duplicate and the wrong line number.
- F4: each completed check through the daemon (every step's check and the
  check a mid-run edit overlaps) lists its gaps on stdout against its own
  results.json, at checkpoint incremental, with a calibration there.
- F2, F3: the oracle's limits record which runs are judged, the loose
  reading of a cut (the classify oracle's law 8 owns the exact cut), and
  #99's cause suffix, which fails loudly until readGapLine reads it. A cut
  now keeps a non-empty part.

Co-authored-by: Isaac <no-reply@databricks.com>
M "Idle counts only the test's I/O" (2026-11-15). The idle sampler no
longer reads I/O from resource names (the `quiet` list). It records, once
per worker, which code started each handle and request (async_hooks init):
the test's when the test's or production's code is on the stack then, or
when the resource that triggered it is the test's. Node triggers a write
by the stream it writes to, so the runner's own pending write (Vitest
forwarding output over its IPC pipe on Windows, the SimpleWriteWrap of
Windows shard 7) is the runner's, and keeps neither a copy busy nor a file
in the I/O set. The same rule holds on every OS.

The test's own pipe read keeps its copy busy, where the name list took a
PipeWrap as the runner's. Both witnesses of the previous commit pass.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
…#115)

* test(oracle-gaps): judge stdout gap lines against results.json, by id

The e2e oracle's greppable-stream law compared the number of stdout gap
lines with the model's number of gaps, and ran before the classification
law. A mutant whose outcome differed from the model's was then reported
as "the stdout has N gap lines for N±1 gaps" under the stream law, not at
its site. #102, #87 and the Windows "every outcome class" failures were
all of this kind: in each, stdout, stderr and results.json agree.

The stream law now compares the stdout with the run's own results.json:
each reported gap has exactly one gap line under its id, with its kind,
line and change, and no other gap line is printed. It runs after the
report completeness law, so model agreement stays with classification.
Two calibrations calibrate the new comparison: one gap printed twice and
another dropped (the count holds), and one line with another line number.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): calibrate each stdout gap-line rejection, and judge daemon checks too

Review of #115:
- F1: one calibration per rejection branch of the stdout gap-line check:
  a dropped line (the #102 symptom), another change, another kind, an
  unreadable line, and a line for a mutant that is not a gap, beside the
  duplicate and the wrong line number.
- F4: each completed check through the daemon (every step's check and the
  check a mid-run edit overlaps) lists its gaps on stdout against its own
  results.json, at checkpoint incremental, with a calibration there.
- F2, F3: the oracle's limits record which runs are judged, the loose
  reading of a cut (the classify oracle's law 8 owns the exact cut), and
  #99's cause suffix, which fails loudly until readGapLine reads it. A cut
  now keeps a non-empty part.

Co-authored-by: Isaac <no-reply@databricks.com>

---------

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 2 commits November 15, 2026 22:34
…stic-cuts-b

# Conflicts:
#	packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
…t results.json

#115 (merged as eec7f77) reads each stdout gap line against results.json and
fails loudly on any text after the change, so #99's ` (cause: <cause>)` suffix
on timeout lines did not read: on the merge, "a copy beside the runner's
pending write" fails with `does not read as a kind, a line, an id, and a
change`. #99 lands second, so it extends the reader: a timeout's line names its
cause after its change, equal to results.json's `detail`, and no other line
names one. The calibration "a stdout gap line names another cause" swaps one
timeout line's cause (backstop and no progress); the stream law rejects it.

Co-authored-by: Isaac <no-reply@databricks.com>
The random-seed history of "a copy waiting on a pipe it opened" also runs a
partial check, which ran without ORACLE_GAPS_TEST_SAMPLES: its child never saw
the count rise, so the pipe read held the copy busy until the backstop and the
partial run judged a timeout where the full run killed (law: baseline lead).
The hooks now ride on the history's shared run options, so the partial run,
the slow twin and the stability rerun set them too; their directories are
removed with the scratch package.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 4 commits November 15, 2026 23:27
#103's constructed histories (51 to 55) come first; #99's follow them, its
deterministic-cuts histories and then the two I/O-owner histories. #103's
history fields (testRoot, loadPlugin) come before #99's hangMs, in the draw
order and in the fingerprint, and the take-back reads all of them as their
drawn constants.

Co-authored-by: Isaac <no-reply@databricks.com>
C (2026-11-16, under both idle M entries): a timer keeps a copy busy only
when the test's code started it and it is due inside the horizon; runner
timers never count. New constructed history, appended last: "a copy beside
the runner's pending timer". The test waits on a ref'd timer due long past
its horizon while the runner keeps its own 100 ms interval running through
the test (test-only hook ORACLE_GAPS_TEST_RUNNER_TIMER, made through the
global setInterval as Vitest's task-update timer is), held by state. The
model owes a timeout by no progress.

With the current rule (the listed-timer count against its beforeEach base,
and every wrapped timer due inside the horizon), the copy is busy until the
backstop: `if-false#0: its timeout is cut by backstop, owed no progress`.

Co-authored-by: Isaac <no-reply@databricks.com>
…multiset

The order of streamed lines between runs is not contract (C, "Progress lines
in determinism", 2026-11-15, and the C entry under P10 and P11). The e2e
oracle's cli observation recorded the stderr gap lines in stream order, so
two runs of a history whose copies finish in another order (two contexts:
"two batches each read a capture row while its worker wrote it", and the
witness-module histories) recorded different traces. The observation now
sorts them and keeps duplicates. No law reads their order; #90 brings the
general sort to the determinism comparison.

Co-authored-by: Isaac <no-reply@databricks.com>
…them

C (2026-11-16, under the M entries "Idle is decided by state only" and
"Idle counts only the test's I/O"): a timer counts only when the test's code
started it, by the same async_hooks record as I/O, and it is due inside the
copy's horizon. Runner timers never count.

- The hook also keeps each live timer and immediate. Its due time comes from
  Node's own fields, against the sampler interval's due time.
- Deleted: the listed-timer count and its beforeEach base, the per-worker
  map of wrapped timers and its setImmediate wrapper. The global setTimeout
  and setInterval only tell the clock now, as the other clock paths do.
- Vite's module runner is machinery: it evaluates this file, so the
  sampler's own interval (and anything the runner starts while it loads a
  setup file) had Vite's frames on its stack and read as the test's.

"a copy beside the runner's pending timer" passes, and so do the I/O and
timer witnesses.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 2 commits November 16, 2026 13:42
…ills

M (2026-11-16, "Spin, no progress and the work budget are kills"), restoring
v21.1 K1 and P3 (R23, G6): a copy that spins past its call budget, makes no
progress, or whose counted work passes the mutant's work budget is killed by
the built-in law (bounded work or no progress) on the route of the test that
was running, or, for the budget, of the copy's tests that did counted work.
Only a backstop cut stays a gap. The count and state cuts and the owner
order are unchanged.

- e2e model: `judge` owes those kills, and a kill's `builtIn` law (null for
  a behavioral law); the check compares it, so a copy wrongly cut as no
  progress beside a behavioral kill is still caught. Reach states renamed
  for the kills; `timeout` now comes from the backstop history.
- classify oracle: two effect kinds, a run cut by its call count (`runSpin`)
  and by no progress (`runIdle`), drawn (M "Randomness wins", appended last),
  with three constructed histories (fixed seed and random campaign).

On the current CLI both fail: the classify oracle's constructed histories
give `timeout` with cause `spin` or `no progress` where the model owes the
built-in kill.

Co-authored-by: Isaac <no-reply@databricks.com>
…uilt-in law

M (2026-11-16, "Spin, no progress and the work budget are kills"), restoring
v21.1 K1 and P3 (R23, G6, K15). The cuts, their order and the owner order are
unchanged; only the outcome changes.

- capture-setup: a spin row and an idle row name the test the cut ends and
  the route it registered.
- check: a copy cut by its call count or by no progress, in a batch or alone,
  carries that test (`builtIn`); the copy whose counted work passes the
  mutant's work budget carries its tests that did counted work.
- classify: such a unit fails `bounded work` or `no progress` on each named
  route, so on a reproducible route the mutant is killed (and its later copies
  do not run); on another route it is random-only or unregistered. Only a
  backstop cut is a timeout (`cause: backstop`).

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 9 commits November 16, 2026 14:24
…uts fields

M (2026-11-16, "Count-only kills", settling v21.2 K20 and K21): a spin past
the call budget and the work budget kill only when time-independent counts
decide them (v21.2 R19a). A copy whose count could follow the clock is never
killed by count; it ends as a gap.

- e2e model: a spin whose test read the clock, made a timer or held I/O
  before the cut (`clocked`) is a timeout by its spin, and its copy ends
  alone; the call budget is per test, as the capture counts it; and only
  rows whose count does not follow the clock are charged to the work budget.
- e2e oracle: "a clocked copy that spins" (appended last): a test that reads
  `Date.now` before production, under the mutant that makes the until loop
  never end. Owed a timeout with cause spin.
- classify oracle: a spin whose test's row is clocked is a gap (`cutOf`),
  with the constructed history "a clocked copy cut by its call count".

M (2026-11-16, "Randomness wins"): `hangMs`, `slowMs`, `loops`, `timerWait`
and `pulse` are drawn, appended where they were; `pendingIo` switches
test-only hooks, so it stays a constant. The draws shift every later field,
so the fingerprint lists are recorded anew. A test that loops declares no
work bound, since the model judges no work in a generated test.

On the kills GREEN (6d9cbfb) both new histories fail: the classify oracle's
clocked spin is a bounded-work kill where a timeout by spin is owed, and the
e2e history reports `delete-call#0: killed, owed timeout`.

Co-authored-by: Isaac <no-reply@databricks.com>
…t ignores the clock

M (2026-11-16, "Count-only kills"; v21.2 K20, R19a, R50, G6).

- capture-setup: a cut row says whether the test had read the clock, made a
  timer or held I/O by then (`clocked`).
- check: a spin whose row is clocked names no test, so it stays a gap; the
  work budget charges only rows that are deterministic in the unmutated runs
  and unclocked in the mutant's run, and its bounded-work verdicts name only
  those rows.
- classify: a spin or work-budget unit with no named test is a timeout (a
  gap), whose detail is its cause.

"a clocked copy that spins" and the classify oracle's clocked spin pass.

Co-authored-by: Isaac <no-reply@databricks.com>
#61's constructed history ("a batch whose run rejects") comes before #99's,
and its field rejectBatch before #99's hangMs; rejectBatch switches a
test-only hook, so it stays a constant. Its env rides on the history's shared
run options with #99's hooks. In runBatch, #61's try/finally (retire the
batch's copies, write the retention line) wraps #99's cut handling: the spin
rows, the built-in verdicts of cut copies, and the backstop's cause. The
context replacement keeps #99's bounded close (closeWithin) with #61's
registry swap. The fingerprint lists are recorded anew (M "Randomness wins").

Co-authored-by: Isaac <no-reply@databricks.com>
… a hang

Two failures of the fixed-seed shard 3 at 6488cbb, from the new rules and the
new draws:

- "a daemon pause that cannot engage": with `loops` drawn, a generated test
  that loops made the model cut an authored mutant's copy by its call budget
  (`unregistered-kill`, a bounded-work kill on an unregistered route). An
  authored copy is not instrumented, so it counts no calls and never spins
  (the model's own rule in runsOf: "An authored mutant's copy is not
  instrumented, so it counts no work"); the CLI says not-judged. The model now
  spins only an operator mutant's copy.
- "a load failure beside a hang": its hang is a kill by no progress now (M,
  2026-11-16), so the state "a setup error beside a timeout under one
  budget" becomes "a setup error beside a hang under one budget", reached by
  a no-progress kill in the same batch.

Co-authored-by: Isaac <no-reply@databricks.com>
v21.2 R50 kills a copy that makes no progress, and names no exception for
an authored mutant. The CLI runs an authored mutant's owner files with no
idle detection, so its hang runs to the backstop: found as the random-seed
counterexample `random seed: a declaration fault: apiLessTypescript` (seed
-313313556, path 0:1:1:1: `the authored mutant a1 is timeout, owed
unregistered-kill`).

New constructed history, appended last: "an authored mutant whose owner
hangs" (one fixed-seed owner test that hangs where the authored edit changes
its output). It fails now: `the authored mutant a1 is timeout, owed killed`.

Co-authored-by: Isaac <no-reply@databricks.com>
v21.2 R50: a copy that makes no progress is killed by the built-in law, and
an authored mutant's copy is no exception. Its owner files now run with the
same idle detection as an operator copy's (none where an unmutated test of
the file did I/O), with its horizon; it still counts no calls, so it has no
call budget. "an authored mutant whose owner hangs" passes, and so does the
replay of seed -313313556 (`a declaration fault: apiLessTypescript`).

Co-authored-by: Isaac <no-reply@databricks.com>
…stic-cuts

#121's and #128's constructed histories come before #99's. The rejected
batch's `outBreak` and #128's `foreign` ride on the history's shared run
options beside #99's hook env. capture-setup's `hit` keeps #121's probe-site
skip beside #99's cut rows.

Co-authored-by: Isaac <no-reply@databricks.com>
…s history

With spin, no progress and the work budget as kills, a timeout line is a
backstop or a clocked spin, which no generated history is likely to print, so
the calibration's candidate search ran past Vitest's 900 s limit. It now
binds the history "a clocked copy that spins", whose timeout line names
`spin`, and it skips a run that recorded no gap lines (a rejected batch), as
it read `stdoutGaps` before checking the exit.

Co-authored-by: Isaac <no-reply@databricks.com>
On CI at e54f769 the third Linux shard took 12:31 (P12: every job under 10
minutes); on the pool it took 480 s against 197 s and 217 s. noTypescript,
the daemon pause that cannot engage and the reset check move to shard 1, and
brokenOwner and the 16 s slow kill to shard 2: about 300 s each on the pool.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant