Skip to content

scratch: #99 idle ownership, Windows only (close after reading) - #119

Closed
KyleAMathews wants to merge 36 commits into
mainfrom
scratch/dc-own-io-win
Closed

KyleAMathews wants to merge 36 commits into
mainfrom
scratch/dc-own-io-win

Conversation

@KyleAMathews

Copy link
Copy Markdown
Contributor

Scratch run for #99: the Windows jobs only, at #99's local head 1bb0b98 plus a narrowed workflow. Not for merge; closed after the run.

This pull request and its description were written by Isaac.

KyleAMathews and others added 30 commits November 14, 2026 16:37
…s not wait

Determinism under load (2026-11-13): a run's outcome must not depend on
how long a run took. A constructed history, "a slow owner's kill and
another owner's kill", has one mutant that file a's random-route test
and file b's fixed-seed test both kill. Its driver also runs the slow
twin: the same package, where a's test waits 3 s past its first budget
wherever the output differs and then judges as before. The law compares
the two reports, `randomAlso` included.

On main, a's slow copy is cut at its first budget and queued for a
confirmation, b's kill decides the mutant first, and a's random failure
is never recorded: the twin reports `randomAlso: false` where the package
reports `true`. `slowTwin` is constructed-only (no generated history
draws it; `histories` takes it back without it).

Co-authored-by: Isaac <no-reply@databricks.com>
…t, and a timer under load

Deterministic cuts, option A (register M, 2026-11-14): no wall-clock cut
decides an outcome, so a mutant's copies are judged by count, not by time.
Three laws are added to the end-to-end oracle, each RED on the parent
commit before any production change.

- The budget witness w_slowpair becomes a work-budget witness: its two
  owners' tests call the function PAIR_LOOPS times where their output
  differs. Each copy stays inside its call budget, the two together pass
  the mutant's work budget (WORK_FLOOR), so the mutant is a timeout by
  count, whatever the clock says. The model judges the budget in owner
  order, a kill before the budget runs out still deciding.
- A slow kill whose test waits 16 s, past Vitest's 15 s per-test limit, is
  killed: no limit that the CLI leaves to Vitest decides the outcome.
- A copy whose test starts a timer, spins under load, then waits for the
  timer, is killed and never cut as idle (the idle law is state, not time).

Constructed-only test fields (slowMs, loops, busy) are held by the drawn
test record at neutral constants and taken back to them by the histories'
map, so no generated history draws them (generator rule, 2026-11-14). The
constructed histories are appended at the end of the list, and the slow
owner's history moved there from the middle of it.

Co-authored-by: Isaac <no-reply@databricks.com>
…he work budget

The first RED (9412eac) lets each copy of w_slowpair make 37,500 calls
beyond its first. At three site evaluations a call that is 112,500 per copy,
past the 100,000 call budget, so main cuts each copy by count and the mutant
is a timeout on either side; the work law is not reached. At 30,000 calls a
copy makes 60,000 to 90,000 site evaluations, inside its budget, and the two
copies together pass the work budget, so main must report survived.

Co-authored-by: Isaac <no-reply@databricks.com>
…-clock budget

Register M entry "Deterministic cuts, option A" (2026-11-14): no wall-clock
cut decides an outcome; a mutant's budget is a work count, and the wall-clock
budget is only a backstop that reports a gap. The model's wall-clock clause
(two budgets: a slow owner file's confirmations past --mutant-limit predict a
timeout) is removed as the repair the M entry requires. This is not a model
change to match production: the clause's authority is the retired confirmation
path, and the M entry retires it.

Consequence, a RED on main: the constructed history "a slow kill past Vitest's
15 s limit" (a 16 s wait where its output differs) now predicts `killed`, and
main reports `timeout` for it, because main's confirmation is cut by the
mutant budget at 15 s (diagnostic on a scratch branch: the copy's outcome is
"run time limit", not Vitest's "Test timed out").

Also: a test that awaits its busy witness is async (the emitter marks it so),
so the baseline loads.

Co-authored-by: Isaac <no-reply@databricks.com>
…states are listed apart

Register M entry "Deterministic cuts, option A" (2026-11-14), with the
coordinator's ruling 3 (fallback): a test that Vitest's own limit ends
("Test timed out") is reported as a backstop gap, like a timed-out run. It
is `timeout` after any kill, and it never keeps a kill from deciding. The
previous rule read a Vitest timeout only when every failing test had one,
so a timed-out test beside a verdict-less failure was `crashed`, and a
timed-out test beside an unregistered kill was not a timeout at all. The
`every` branch and the `!timeout` filter are now redundant and removed.

Law: a run's outcome does not depend on how long a test took (determinism
under load); a clock limit may only report a gap.
RED: the classify oracle on the parent classify.mjs (HEAD before this commit,
built from the same fixtures) fails with "mutant m0: unregistered-kill, owed
timeout", and the parent's crashed/timeout split fails the same law.
Mutant: timeout checked before the kills (scratch copy): the classify oracle
fails, "mutant m0: timeout, owed killed"; the mutant is rejected.
GREEN: the classify oracle passes (276 tests). The model counts a timeout
message only in a file that loaded, as production does (a file that fails to
load gives no tests).

The e2e oracle's states "a slow owner's kill reported as when it did not
wait", "a slow kill past fifteen seconds killed" and "a copy waiting on a
timer under load killed" are listed as separate: they have their own
histories, so "every outcome class" must not enter them.

Co-authored-by: Isaac <no-reply@databricks.com>
Register M entry "Deterministic cuts, option A" (2026-11-14). A copy runs
until it finishes, goes idle, or spins; the only wall-clock limit is a
backstop that reports a gap and never decides a kill.

- No first budget and no confirmation: a copy runs once, to its end, under
  the backstop limit (hang-limit plus three seconds per second of the file's
  unmutated time). The cut marking and the confirming run are removed.
- G1 becomes a work budget: a mutant's copies, summed in their owner files'
  report order, may do at most max(WORK_FLOOR, 10 x the owners' unmutated
  calls) site evaluations (WORK_FLOOR = CALL_FLOOR, 100,000). Past it, the
  copy that crosses and every later copy is a timeout, so a kill among them
  does not count. This is deterministic because the work is a count; the
  owner order is the report's, not the completion order.
- The wall-clock mutant budget (--mutant-limit) is a backstop checked after
  all copies: no kill and elapsed over budget is a timeout; it never decides a
  kill.
- Defaults (coordinator rulings, 2026-11-14): --hang-limit 60 s, --mutant-limit
  180 s, --test-timeout 65 s (hang-limit plus 5 s). Cost to state: an unmutated
  hang that no CLI limit cuts now fails after 65 s, not 15 s; under option A an
  idle hang is cut by `no progress` in about 500 ms, so only a busy hang with no
  instrumented site waits for the backstop.
- --first-limit is validated and retired: it prints a note and changes nothing.
- A test that Vitest's own limit ends is a backstop gap (classify.mjs; its
  RED and mutant are in the prior commit, 4b063c8).

Correction to 4b063c8's message: that message said the parent's crashed/timeout
split fails the classify oracle. I did not run that case. The verified RED is
the one in 4b063c8's body: the parent gives "unregistered-kill" where the model
owes "timeout".

Co-authored-by: Isaac <no-reply@databricks.com>
… in the test

The counter-witness (register, 2026-11-13 "Determinism under load"; the
2026-11-14 entry "Deterministic cuts, option A") was a copy whose test spun
for 600 ms and then waited 2 s on a timer. Its spin kept event-loop
utilization above 5%, which alone keeps a copy from looking idle, so the
timer check never mattered: mutant 2 (elapsed-only idle, the timer checks
disabled) passed the witness. The witness now waits on one timer due 3 s
out, inside the horizon, with nothing else in the test. A copy with its
timer pending is never idle; only that check stands between it and an idle
cut. A loaded variant, if wanted, is a CPU hog outside the worker.

The state is "a copy waiting on a timer killed"; the constructed field is
timerWait (a constructed-only test field, taken back to null; no generated
test draws it).

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

# Conflicts:
#	packages/oracles/oracle-gaps/gap-report-model-oracle.mjs
#	packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
#	packages/oracles/scripts/pr-shards.mjs
… copies

Register C entry "Deterministic cuts, outcome rule (under M option A)",
2026-11-14: a copy cut by a clock or by its call count ends the mutant's
copies; no later copy runs. The model now predicts that: a copy that spins
(a count) stops the owner files after it, so a later owner's kill does not
count and the outcome is a timeout.

Witness: a constructed history whose one function spins in the mutant (the
until loop of w_spin) with two owners, a and b. Owner a's copy spins first;
owner b's judging test would kill. Production (main at this commit) runs
b's copy after a's spin and reports killed. The model owes timeout.

RED: on this commit, the new witness fails (the CLI reports killed where the
model owes timeout; pool run dc-rule-red-*).
Limits: the model's rule covers copies cut by a count; a wall-clock cut
(the backstop and Vitest's limit) is a backstop gap under the same rule, and
its own witness is pending (finding 4, the H1 witness).

Co-authored-by: Isaac <no-reply@databricks.com>
…ant, uncharged

Register C entry "Deterministic cuts, outcome rule (under M option A)",
2026-11-14: only counts and states decide a mutant's outcome.

- A copy cut by a clock or by its call count (its own timeout, or Vitest's
  "Test timed out") is never charged, and it ends the mutant's copies: no
  later copy runs, and a kill among later copies does not count (review B2).
- A copy's work is the calls of its deterministic, unclocked tests
  (`baseline.work`, classify.mjs), and the owner files' most-working tests
  set the budget and the call budget (review B3).
- The mutant-level elapsed-seconds backstop is removed: it summed cached
  units' stored seconds, so a daemon reuse could differ from a cold run
  (review finding 3). Each copy's own wall cut remains the backstop. Whether
  --mutant-limit still has a role is open (it now only affects identity).
- Model: a copy cut by a count or a clock ends the owner files (no later file
  runs), so the model owes the same outcome.

Witness: a constructed history with two owners, each making 30,000 extra
calls; owner a's copy is under the budget, owner b's would kill but passes
the budget, so the mutant is a timeout by count. The reversed-sum mutant
(review H2, 968bfd7) is the mutant it must fail.

Not witnessed here: a spin in an earlier owner ahead of a kill in a later
one. A spin witness was tried first and removed: the later owner runs the same
spinning function, so it spins too and never kills, and the witness passed on
the parent. Finding 4's backstop witness (H1) is pending.

Co-authored-by: Isaac <no-reply@databricks.com>
…r copy, wall cuts end the mutant

Register C entry "Deterministic cuts, outcome rule (under M option A)", as
refined by the coordinator: a copy ending in "no progress" (a state) or a spin
past its call budget (a count) is a deterministic timeout for that copy. It is
charged nothing (the rows of a cut copy are not kept, and the per-copy call
budget already bounds it), and the next owner's copy still runs, so a later
kill decides. Only a wall-clock cut (the CLI backstop, or Vitest's "Test timed
out") ends the remaining copies, as a backstop gap that is never charged.

- Model: runsOf tags each cut with its kind (wall, count, state); judge ends
  the owner files only on a wall cut.
- Check: spin and idle copies settle with spin or idle set; cutByTime covers
  wall cuts only.
- --mutant-limit is retired (validated, noted, changes nothing); the driver no
  longer passes it.
- Constructed history "a backstop cut ends the copies ahead of a later kill":
  owner a's copy waits past a 2 s backstop; owner b would kill, and does not
  run. The state "a backstop cut ends the copies ahead of a later kill" is
  entered.

Open: "a declaration fault: noTypescript" (shard 3) streams 161 gap lines where
the model expects 159; a per-mutant check is pending. It is a model and CLI
disagreement, not a duplicate line (the scratch diagnostic found no repeated
line).

Co-authored-by: Isaac <no-reply@databricks.com>
…ot in scope there)

Co-authored-by: Isaac <no-reply@databricks.com>
…p, never a silent one

Review finding 5 (register L8). The 10 s close waits in closeContexts and in
replace continued silently after a close was still pending. Now each close
waits up to the 60 s backstop; a close still pending is an error that names
the backstop gap, and no replacement is warmed while the old context is still
closing. Closes that finish are unchanged.

Witness: none. A pending close needs a context whose close never settles, which
costs 60 s per run; it is a limit of this change (the coverage is by inspection
of closeWithin and its two callers).

Co-authored-by: Isaac <no-reply@databricks.com>
…oes idle

Owner a's copy now awaits a promise that never resolves, while a repeating
timer stays due inside its horizon, so the copy is never idle and no site
spins. Only the backstop cuts it, on every machine. The witness no longer
depends on whether a 12 s wait lands inside the backstop band (register
2026-11-14, outcome rule item 5).

Co-authored-by: Isaac <no-reply@databricks.com>
…ress, or work budget

A copy's cut is now recorded as its cause and carried into the mutant's timeout
detail, results.json, and the gap line (cause: ...). The model predicts the
cause of each timeout: a backstop cut (a wall-clock limit), a spin past the
copy's call budget (a count, outcome rule item 4), no progress (a state), or
the mutant's work budget. The model had no per-copy call budget, so the
work-budget history's 4-sites-per-call mutant was a spin and not a work-budget
timeout; the model now predicts that. ORACLE_GAPS_PROFILE rows carry each
copy's cause, its file's measured time, its limit and its duration.

RED: the e2e oracle, with the cause predicted and production unchanged, fails
on the backstop history (its timeout is cut by run time limit).

Co-authored-by: Isaac <no-reply@databricks.com>
…s as constants

The deterministic-cuts constructed fields are constants in every drawn record,
so no draw changes: the tests line hashes the records without them and is
main's (e9029433a23d1f10). slowMs, loops, timerWait and pulse each get a line
after the others, with its neutral value checked on every generated test, and
hangMs is a drawn constant (null where generated) with its own line. The
constructed histories #99 adds are appended after main's 26, by name and index.

Co-authored-by: Isaac <no-reply@databricks.com>
…e current cuts

--help and the index header give the defaults (--test-timeout 65000, --hang-limit
60000) and mark --first-limit and --mutant-limit retired. The README, the
changeset and the e2e law 4 text describe the backstop, the call and work
counts, and the cause each timeout names. Stale states and names are renamed
(confirmations, the first budget, the 15 s limit): a slow kill after a 16 s
wait, and a mutant's copies past its work budget. MUTANT_LIMIT_MS, unused, is
removed. The determinism twin is described as a determinism check: it passes
on its parent, so the outcome rule's RED is the classify oracle's.

Co-authored-by: Isaac <no-reply@databricks.com>
… and counts

The README's gap section and the changeset's bullet describe the counts and the
backstop, the cause each timeout names (on the gap line too), and the retired
options, in place of the confirmation budget. The classify oracle's gap-line
law (in bfca29a) accepts a timeout's cause suffix.

Co-authored-by: Isaac <no-reply@databricks.com>
Resolves the constructed list (main's 29 histories first, then the five #99 histories at 29 to 33), the drawn record (main's pauseEngages and warmReplaced, then hangMs), the take-back, the limits text (main's Windows notes, with #99's work-budget wording), and the shard lists (#99's histories in shard 4, main's test.projects line in shard 5). The fingerprint canary passes.

Co-authored-by: Isaac <no-reply@databricks.com>
…; the Windows shards list the new histories

The constructed backstop (hangMs) is a drawn constant now, so its take-back returns
the drawn value (null), which the arbitrary can take back. The fixed-seed shards
of main are kept as they are: the new histories fall into the Linux rest shard,
and the Windows partition, which is exact, lists them in shard 10.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…n the CLI arguments

Co-authored-by: Isaac <no-reply@databricks.com>
…'d timer

An unref'd timer is not among the process's active resources, so only the copy's
own pending-timer record keeps it busy (the idle rule, held state, M 2026-11-15).
The witness is a constructed history of its own; the reach state is recorded.

Co-authored-by: Isaac <no-reply@databricks.com>
…dition

A copy is idle when no counted timer is due within its horizon, no timer is listed
beyond its test's base, and no I/O handle is open (M entry 2026-11-15). Sample
lateness and the event loop's utilization no longer reset the window: a busy
synchronous or microtask loop blocks the sampler, so it cannot look idle. The
held-state witnesses (a copy waiting on a timer, ref'd or unref'd) keep the
copy busy, and the mutants that ignore a due timer are killed.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…ef'd timer history

The e2e shard check named the removed history 'a slow kill past Vitest's 15 s limit'
and had no shard for the unref'd timer history, so CI's test:oracle-gaps:shards
failed. The check now partitions the e2e tests exactly.

Co-authored-by: Isaac <no-reply@databricks.com>
… wall-clock cut

With the idle rule decided by state (M entry, 2026-11-15), a timer due beyond the
copy's horizon does not keep it busy, so it is idle and cut as no progress. The
model's wall-clock cut for such a wait was stale; the load experiment's 12 s copies
gave the same outcome on three repeats under a concurrent run (the model owed a
backstop cut).

Co-authored-by: Isaac <no-reply@databricks.com>
… name

Co-authored-by: Isaac <no-reply@databricks.com>
Two constructed histories, appended after the others (M "Idle counts only the
test's I/O", 2026-11-15):

- "a copy waiting on a pipe it opened": the test reads the stdout of a child
  it unrefs, so only its own pipe read is active. The child writes once the
  copy's sampler has judged the test 20 more times (a count, not a time). The
  mutant is owed killed.
- "a copy beside the runner's pending write": the test writes to the console
  and waits on an unref'd timer past its horizon, while the runner holds its
  own pipe write pending through the test. The mutant is owed a timeout by no
  progress.

Two test-only hooks hold those states: ORACLE_GAPS_TEST_SAMPLES (the
sampler's count) and ORACLE_GAPS_TEST_RUNNER_WRITE (the runner's pending
write, with an idle marker for reach). The idle rule itself is unchanged
here, so both histories fail: the test's pipe read is cut as no progress,
and the runner's write keeps its copy busy until the backstop.

Co-authored-by: Isaac <no-reply@databricks.com>
…sult's detail

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 2 commits November 15, 2026 21:32
M "Idle counts only the test's I/O" (2026-11-15). The idle sampler no
longer reads I/O from resource names (the `quiet` list). It records, once
per worker, which code started each handle and request (async_hooks init):
the test's when the test's or production's code is on the stack then, or
when the resource that triggered it is the test's. Node triggers a write
by the stream it writes to, so the runner's own pending write (Vitest
forwarding output over its IPC pipe on Windows, the SimpleWriteWrap of
Windows shard 7) is the runner's, and keeps neither a copy busy nor a file
in the I/O set. The same rule holds on every OS.

The test's own pipe read keeps its copy busy, where the name list took a
PipeWrap as the runner's. Both witnesses of the previous commit pass.

Co-authored-by: Isaac <no-reply@databricks.com>
…ot for the PR)

Co-authored-by: Isaac <no-reply@databricks.com>
@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: a0b0d27

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes changesets to release 1 package
Name Type
@tanstack/oracles Minor

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

KyleAMathews and others added 2 commits November 15, 2026 22:34
…stic-cuts-b

# Conflicts:
#	packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
…t results.json

#115 (merged as eec7f77) reads each stdout gap line against results.json and
fails loudly on any text after the change, so #99's ` (cause: <cause>)` suffix
on timeout lines did not read: on the merge, "a copy beside the runner's
pending write" fails with `does not read as a kind, a line, an id, and a
change`. #99 lands second, so it extends the reader: a timeout's line names its
cause after its change, equal to results.json's `detail`, and no other line
names one. The calibration "a stdout gap line names another cause" swaps one
timeout line's cause (backstop and no progress); the stream law rejects it.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 2 commits November 15, 2026 22:43
The random-seed history of "a copy waiting on a pipe it opened" also runs a
partial check, which ran without ORACLE_GAPS_TEST_SAMPLES: its child never saw
the count rise, so the pipe read held the copy busy until the backstop and the
partial run judged a timeout where the full run killed (law: baseline lead).
The hooks now ride on the history's shared run options, so the partial run,
the slow twin and the stability rerun set them too; their directories are
removed with the scratch package.

Co-authored-by: Isaac <no-reply@databricks.com>
@KyleAMathews

Copy link
Copy Markdown
Contributor Author

Read: Windows run 43805903246 at a0b0d27 (#99 local head 325aeb1 plus a Windows-only workflow). Closing the scratch PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant