Repository navigation
scratch: #99 idle ownership, Windows only (close after reading) - #119
Closed
KyleAMathews wants to merge 36 commits into
Closed
KyleAMathews wants to merge 36 commits into
KyleAMathews wants to merge 36 commits into
Conversation
…s not wait Determinism under load (2026-11-13): a run's outcome must not depend on how long a run took. A constructed history, "a slow owner's kill and another owner's kill", has one mutant that file a's random-route test and file b's fixed-seed test both kill. Its driver also runs the slow twin: the same package, where a's test waits 3 s past its first budget wherever the output differs and then judges as before. The law compares the two reports, `randomAlso` included. On main, a's slow copy is cut at its first budget and queued for a confirmation, b's kill decides the mutant first, and a's random failure is never recorded: the twin reports `randomAlso: false` where the package reports `true`. `slowTwin` is constructed-only (no generated history draws it; `histories` takes it back without it). Co-authored-by: Isaac <no-reply@databricks.com>
…t, and a timer under load Deterministic cuts, option A (register M, 2026-11-14): no wall-clock cut decides an outcome, so a mutant's copies are judged by count, not by time. Three laws are added to the end-to-end oracle, each RED on the parent commit before any production change. - The budget witness w_slowpair becomes a work-budget witness: its two owners' tests call the function PAIR_LOOPS times where their output differs. Each copy stays inside its call budget, the two together pass the mutant's work budget (WORK_FLOOR), so the mutant is a timeout by count, whatever the clock says. The model judges the budget in owner order, a kill before the budget runs out still deciding. - A slow kill whose test waits 16 s, past Vitest's 15 s per-test limit, is killed: no limit that the CLI leaves to Vitest decides the outcome. - A copy whose test starts a timer, spins under load, then waits for the timer, is killed and never cut as idle (the idle law is state, not time). Constructed-only test fields (slowMs, loops, busy) are held by the drawn test record at neutral constants and taken back to them by the histories' map, so no generated history draws them (generator rule, 2026-11-14). The constructed histories are appended at the end of the list, and the slow owner's history moved there from the middle of it. Co-authored-by: Isaac <no-reply@databricks.com>
…he work budget The first RED (9412eac) lets each copy of w_slowpair make 37,500 calls beyond its first. At three site evaluations a call that is 112,500 per copy, past the 100,000 call budget, so main cuts each copy by count and the mutant is a timeout on either side; the work law is not reached. At 30,000 calls a copy makes 60,000 to 90,000 site evaluations, inside its budget, and the two copies together pass the work budget, so main must report survived. Co-authored-by: Isaac <no-reply@databricks.com>
…-clock budget Register M entry "Deterministic cuts, option A" (2026-11-14): no wall-clock cut decides an outcome; a mutant's budget is a work count, and the wall-clock budget is only a backstop that reports a gap. The model's wall-clock clause (two budgets: a slow owner file's confirmations past --mutant-limit predict a timeout) is removed as the repair the M entry requires. This is not a model change to match production: the clause's authority is the retired confirmation path, and the M entry retires it. Consequence, a RED on main: the constructed history "a slow kill past Vitest's 15 s limit" (a 16 s wait where its output differs) now predicts `killed`, and main reports `timeout` for it, because main's confirmation is cut by the mutant budget at 15 s (diagnostic on a scratch branch: the copy's outcome is "run time limit", not Vitest's "Test timed out"). Also: a test that awaits its busy witness is async (the emitter marks it so), so the baseline loads. Co-authored-by: Isaac <no-reply@databricks.com>
…states are listed apart
Register M entry "Deterministic cuts, option A" (2026-11-14), with the
coordinator's ruling 3 (fallback): a test that Vitest's own limit ends
("Test timed out") is reported as a backstop gap, like a timed-out run. It
is `timeout` after any kill, and it never keeps a kill from deciding. The
previous rule read a Vitest timeout only when every failing test had one,
so a timed-out test beside a verdict-less failure was `crashed`, and a
timed-out test beside an unregistered kill was not a timeout at all. The
`every` branch and the `!timeout` filter are now redundant and removed.
Law: a run's outcome does not depend on how long a test took (determinism
under load); a clock limit may only report a gap.
RED: the classify oracle on the parent classify.mjs (HEAD before this commit,
built from the same fixtures) fails with "mutant m0: unregistered-kill, owed
timeout", and the parent's crashed/timeout split fails the same law.
Mutant: timeout checked before the kills (scratch copy): the classify oracle
fails, "mutant m0: timeout, owed killed"; the mutant is rejected.
GREEN: the classify oracle passes (276 tests). The model counts a timeout
message only in a file that loaded, as production does (a file that fails to
load gives no tests).
The e2e oracle's states "a slow owner's kill reported as when it did not
wait", "a slow kill past fifteen seconds killed" and "a copy waiting on a
timer under load killed" are listed as separate: they have their own
histories, so "every outcome class" must not enter them.
Co-authored-by: Isaac <no-reply@databricks.com>
Register M entry "Deterministic cuts, option A" (2026-11-14). A copy runs until it finishes, goes idle, or spins; the only wall-clock limit is a backstop that reports a gap and never decides a kill. - No first budget and no confirmation: a copy runs once, to its end, under the backstop limit (hang-limit plus three seconds per second of the file's unmutated time). The cut marking and the confirming run are removed. - G1 becomes a work budget: a mutant's copies, summed in their owner files' report order, may do at most max(WORK_FLOOR, 10 x the owners' unmutated calls) site evaluations (WORK_FLOOR = CALL_FLOOR, 100,000). Past it, the copy that crosses and every later copy is a timeout, so a kill among them does not count. This is deterministic because the work is a count; the owner order is the report's, not the completion order. - The wall-clock mutant budget (--mutant-limit) is a backstop checked after all copies: no kill and elapsed over budget is a timeout; it never decides a kill. - Defaults (coordinator rulings, 2026-11-14): --hang-limit 60 s, --mutant-limit 180 s, --test-timeout 65 s (hang-limit plus 5 s). Cost to state: an unmutated hang that no CLI limit cuts now fails after 65 s, not 15 s; under option A an idle hang is cut by `no progress` in about 500 ms, so only a busy hang with no instrumented site waits for the backstop. - --first-limit is validated and retired: it prints a note and changes nothing. - A test that Vitest's own limit ends is a backstop gap (classify.mjs; its RED and mutant are in the prior commit, 4b063c8). Correction to 4b063c8's message: that message said the parent's crashed/timeout split fails the classify oracle. I did not run that case. The verified RED is the one in 4b063c8's body: the parent gives "unregistered-kill" where the model owes "timeout". Co-authored-by: Isaac <no-reply@databricks.com>
… in the test The counter-witness (register, 2026-11-13 "Determinism under load"; the 2026-11-14 entry "Deterministic cuts, option A") was a copy whose test spun for 600 ms and then waited 2 s on a timer. Its spin kept event-loop utilization above 5%, which alone keeps a copy from looking idle, so the timer check never mattered: mutant 2 (elapsed-only idle, the timer checks disabled) passed the witness. The witness now waits on one timer due 3 s out, inside the horizon, with nothing else in the test. A copy with its timer pending is never idle; only that check stands between it and an idle cut. A loaded variant, if wanted, is a CPU hog outside the worker. The state is "a copy waiting on a timer killed"; the constructed field is timerWait (a constructed-only test field, taken back to null; no generated test draws it). Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com> # Conflicts: # packages/oracles/oracle-gaps/gap-report-model-oracle.mjs # packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs # packages/oracles/scripts/pr-shards.mjs
… copies Register C entry "Deterministic cuts, outcome rule (under M option A)", 2026-11-14: a copy cut by a clock or by its call count ends the mutant's copies; no later copy runs. The model now predicts that: a copy that spins (a count) stops the owner files after it, so a later owner's kill does not count and the outcome is a timeout. Witness: a constructed history whose one function spins in the mutant (the until loop of w_spin) with two owners, a and b. Owner a's copy spins first; owner b's judging test would kill. Production (main at this commit) runs b's copy after a's spin and reports killed. The model owes timeout. RED: on this commit, the new witness fails (the CLI reports killed where the model owes timeout; pool run dc-rule-red-*). Limits: the model's rule covers copies cut by a count; a wall-clock cut (the backstop and Vitest's limit) is a backstop gap under the same rule, and its own witness is pending (finding 4, the H1 witness). Co-authored-by: Isaac <no-reply@databricks.com>
…ant, uncharged Register C entry "Deterministic cuts, outcome rule (under M option A)", 2026-11-14: only counts and states decide a mutant's outcome. - A copy cut by a clock or by its call count (its own timeout, or Vitest's "Test timed out") is never charged, and it ends the mutant's copies: no later copy runs, and a kill among later copies does not count (review B2). - A copy's work is the calls of its deterministic, unclocked tests (`baseline.work`, classify.mjs), and the owner files' most-working tests set the budget and the call budget (review B3). - The mutant-level elapsed-seconds backstop is removed: it summed cached units' stored seconds, so a daemon reuse could differ from a cold run (review finding 3). Each copy's own wall cut remains the backstop. Whether --mutant-limit still has a role is open (it now only affects identity). - Model: a copy cut by a count or a clock ends the owner files (no later file runs), so the model owes the same outcome. Witness: a constructed history with two owners, each making 30,000 extra calls; owner a's copy is under the budget, owner b's would kill but passes the budget, so the mutant is a timeout by count. The reversed-sum mutant (review H2, 968bfd7) is the mutant it must fail. Not witnessed here: a spin in an earlier owner ahead of a kill in a later one. A spin witness was tried first and removed: the later owner runs the same spinning function, so it spins too and never kills, and the witness passed on the parent. Finding 4's backstop witness (H1) is pending. Co-authored-by: Isaac <no-reply@databricks.com>
…r copy, wall cuts end the mutant Register C entry "Deterministic cuts, outcome rule (under M option A)", as refined by the coordinator: a copy ending in "no progress" (a state) or a spin past its call budget (a count) is a deterministic timeout for that copy. It is charged nothing (the rows of a cut copy are not kept, and the per-copy call budget already bounds it), and the next owner's copy still runs, so a later kill decides. Only a wall-clock cut (the CLI backstop, or Vitest's "Test timed out") ends the remaining copies, as a backstop gap that is never charged. - Model: runsOf tags each cut with its kind (wall, count, state); judge ends the owner files only on a wall cut. - Check: spin and idle copies settle with spin or idle set; cutByTime covers wall cuts only. - --mutant-limit is retired (validated, noted, changes nothing); the driver no longer passes it. - Constructed history "a backstop cut ends the copies ahead of a later kill": owner a's copy waits past a 2 s backstop; owner b would kill, and does not run. The state "a backstop cut ends the copies ahead of a later kill" is entered. Open: "a declaration fault: noTypescript" (shard 3) streams 161 gap lines where the model expects 159; a per-mutant check is pending. It is a model and CLI disagreement, not a duplicate line (the scratch diagnostic found no repeated line). Co-authored-by: Isaac <no-reply@databricks.com>
…ot in scope there) Co-authored-by: Isaac <no-reply@databricks.com>
…p, never a silent one Review finding 5 (register L8). The 10 s close waits in closeContexts and in replace continued silently after a close was still pending. Now each close waits up to the 60 s backstop; a close still pending is an error that names the backstop gap, and no replacement is warmed while the old context is still closing. Closes that finish are unchanged. Witness: none. A pending close needs a context whose close never settles, which costs 60 s per run; it is a limit of this change (the coverage is by inspection of closeWithin and its two callers). Co-authored-by: Isaac <no-reply@databricks.com>
…oes idle Owner a's copy now awaits a promise that never resolves, while a repeating timer stays due inside its horizon, so the copy is never idle and no site spins. Only the backstop cuts it, on every machine. The witness no longer depends on whether a 12 s wait lands inside the backstop band (register 2026-11-14, outcome rule item 5). Co-authored-by: Isaac <no-reply@databricks.com>
…ress, or work budget A copy's cut is now recorded as its cause and carried into the mutant's timeout detail, results.json, and the gap line (cause: ...). The model predicts the cause of each timeout: a backstop cut (a wall-clock limit), a spin past the copy's call budget (a count, outcome rule item 4), no progress (a state), or the mutant's work budget. The model had no per-copy call budget, so the work-budget history's 4-sites-per-call mutant was a spin and not a work-budget timeout; the model now predicts that. ORACLE_GAPS_PROFILE rows carry each copy's cause, its file's measured time, its limit and its duration. RED: the e2e oracle, with the cause predicted and production unchanged, fails on the backstop history (its timeout is cut by run time limit). Co-authored-by: Isaac <no-reply@databricks.com>
…s as constants The deterministic-cuts constructed fields are constants in every drawn record, so no draw changes: the tests line hashes the records without them and is main's (e9029433a23d1f10). slowMs, loops, timerWait and pulse each get a line after the others, with its neutral value checked on every generated test, and hangMs is a drawn constant (null where generated) with its own line. The constructed histories #99 adds are appended after main's 26, by name and index. Co-authored-by: Isaac <no-reply@databricks.com>
…e current cuts --help and the index header give the defaults (--test-timeout 65000, --hang-limit 60000) and mark --first-limit and --mutant-limit retired. The README, the changeset and the e2e law 4 text describe the backstop, the call and work counts, and the cause each timeout names. Stale states and names are renamed (confirmations, the first budget, the 15 s limit): a slow kill after a 16 s wait, and a mutant's copies past its work budget. MUTANT_LIMIT_MS, unused, is removed. The determinism twin is described as a determinism check: it passes on its parent, so the outcome rule's RED is the classify oracle's. Co-authored-by: Isaac <no-reply@databricks.com>
… and counts The README's gap section and the changeset's bullet describe the counts and the backstop, the cause each timeout names (on the gap line too), and the retired options, in place of the confirmation budget. The classify oracle's gap-line law (in bfca29a) accepts a timeout's cause suffix. Co-authored-by: Isaac <no-reply@databricks.com>
Resolves the constructed list (main's 29 histories first, then the five #99 histories at 29 to 33), the drawn record (main's pauseEngages and warmReplaced, then hangMs), the take-back, the limits text (main's Windows notes, with #99's work-budget wording), and the shard lists (#99's histories in shard 4, main's test.projects line in shard 5). The fingerprint canary passes. Co-authored-by: Isaac <no-reply@databricks.com>
…; the Windows shards list the new histories The constructed backstop (hangMs) is a drawn constant now, so its take-back returns the drawn value (null), which the arbitrary can take back. The fixed-seed shards of main are kept as they are: the new histories fall into the Linux rest shard, and the Windows partition, which is exact, lists them in shard 10. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…n the CLI arguments Co-authored-by: Isaac <no-reply@databricks.com>
…'d timer An unref'd timer is not among the process's active resources, so only the copy's own pending-timer record keeps it busy (the idle rule, held state, M 2026-11-15). The witness is a constructed history of its own; the reach state is recorded. Co-authored-by: Isaac <no-reply@databricks.com>
…dition A copy is idle when no counted timer is due within its horizon, no timer is listed beyond its test's base, and no I/O handle is open (M entry 2026-11-15). Sample lateness and the event loop's utilization no longer reset the window: a busy synchronous or microtask loop blocks the sampler, so it cannot look idle. The held-state witnesses (a copy waiting on a timer, ref'd or unref'd) keep the copy busy, and the mutants that ignore a due timer are killed. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…ef'd timer history The e2e shard check named the removed history 'a slow kill past Vitest's 15 s limit' and had no shard for the unref'd timer history, so CI's test:oracle-gaps:shards failed. The check now partitions the e2e tests exactly. Co-authored-by: Isaac <no-reply@databricks.com>
… wall-clock cut With the idle rule decided by state (M entry, 2026-11-15), a timer due beyond the copy's horizon does not keep it busy, so it is idle and cut as no progress. The model's wall-clock cut for such a wait was stale; the load experiment's 12 s copies gave the same outcome on three repeats under a concurrent run (the model owed a backstop cut). Co-authored-by: Isaac <no-reply@databricks.com>
… name Co-authored-by: Isaac <no-reply@databricks.com>
Two constructed histories, appended after the others (M "Idle counts only the test's I/O", 2026-11-15): - "a copy waiting on a pipe it opened": the test reads the stdout of a child it unrefs, so only its own pipe read is active. The child writes once the copy's sampler has judged the test 20 more times (a count, not a time). The mutant is owed killed. - "a copy beside the runner's pending write": the test writes to the console and waits on an unref'd timer past its horizon, while the runner holds its own pipe write pending through the test. The mutant is owed a timeout by no progress. Two test-only hooks hold those states: ORACLE_GAPS_TEST_SAMPLES (the sampler's count) and ORACLE_GAPS_TEST_RUNNER_WRITE (the runner's pending write, with an idle marker for reach). The idle rule itself is unchanged here, so both histories fail: the test's pipe read is cut as no progress, and the runner's write keeps its copy busy until the backstop. Co-authored-by: Isaac <no-reply@databricks.com>
…sult's detail Co-authored-by: Isaac <no-reply@databricks.com>
M "Idle counts only the test's I/O" (2026-11-15). The idle sampler no longer reads I/O from resource names (the `quiet` list). It records, once per worker, which code started each handle and request (async_hooks init): the test's when the test's or production's code is on the stack then, or when the resource that triggered it is the test's. Node triggers a write by the stream it writes to, so the runner's own pending write (Vitest forwarding output over its IPC pipe on Windows, the SimpleWriteWrap of Windows shard 7) is the runner's, and keeps neither a copy busy nor a file in the I/O set. The same rule holds on every OS. The test's own pipe read keeps its copy busy, where the name list took a PipeWrap as the runner's. Both witnesses of the previous commit pass. Co-authored-by: Isaac <no-reply@databricks.com>
…ot for the PR) Co-authored-by: Isaac <no-reply@databricks.com>
|
| Name | Type |
|---|---|
| @tanstack/oracles | Minor |
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…stic-cuts-b # Conflicts: # packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
…t results.json #115 (merged as eec7f77) reads each stdout gap line against results.json and fails loudly on any text after the change, so #99's ` (cause: <cause>)` suffix on timeout lines did not read: on the merge, "a copy beside the runner's pending write" fails with `does not read as a kind, a line, an id, and a change`. #99 lands second, so it extends the reader: a timeout's line names its cause after its change, equal to results.json's `detail`, and no other line names one. The calibration "a stdout gap line names another cause" swaps one timeout line's cause (backstop and no progress); the stream law rejects it. Co-authored-by: Isaac <no-reply@databricks.com>
The random-seed history of "a copy waiting on a pipe it opened" also runs a partial check, which ran without ORACLE_GAPS_TEST_SAMPLES: its child never saw the count rise, so the pipe read held the copy busy until the backstop and the partial run judged a timeout where the full run killed (law: baseline lead). The hooks now ride on the history's shared run options, so the partial run, the slow twin and the stability rerun set them too; their directories are removed with the scratch package. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scratch run for #99: the Windows jobs only, at #99's local head 1bb0b98 plus a narrowed workflow. Not for merge; closed after the run.
This pull request and its description were written by Isaac.