Skip to content

fix(oracle-gaps): close the merged-PR conformance gaps (drawn fields, F-D, cold fallback) - #131

Open
KyleAMathews wants to merge 25 commits into
mainfrom
fix/merged-conformance
Open

KyleAMathews wants to merge 25 commits into
mainfrom
fix/merged-conformance

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

This PR closes the conformance gaps that merged PRs #55, #73 and #103 left on main: v21.2 D50, D58, D59 and D61. It also closes one item from #57's findings (D68). It was revised for the v21.2 merge gate (reviews/pr131-merge-gate.md, F1 to F8). It is oracle-first throughout and merges origin/main at 2552d6a (#121, #128).

What changes

1. Randomness wins (R54, I16, R4; D59, D68; gate F1, F8). Each field that changes what a history runs is drawn, unless it switches a test-only hook on.

Field or value Group Why
warmReplaced drawn It is a scenario: it stops the warm daemon through the CLI. It acts only in resetBeforeRead, which drawn histories no longer take (see below), so only constructed histories exercise it.
testRoot, loadPlugin drawn They are fixture shapes: the config's root, and a config plugin. A root applies only to an ES module config that Vitest discovers; the plugin applies only to a config in ES module syntax.
spinData (new) drawn It is a fixture shape: file b's data-reading verdict test spins (see 2).
spinThrows (new) drawn It is a fixture shape: a spinning copy first schedules an error outside its test (see 2).
siblingDir (new) drawn It is a fixture placement: oracle-gaps-sibling-* (see 4).
tornRows, rejectBatch constant They switch on ORACLE_GAPS_TEST_TORN_ROWS and _REJECT_BATCH.
replyHeld (new) constant It switches on the daemon-check pause.
hookStops (new) constant On win32 the driver runs a history's stops only when this is set. Every win32 stop goes through ORACLE_GAPS_TEST_STOP.
steps dieMidCheck and resetBeforeRead constructed only (F1) They arm ORACLE_GAPS_TEST_PAUSE_AT and its marks. The histories map reads them out of drawn steps.
stops after and warm constructed only They switch on _HOLD_EXIT_MS and _WARM_PROBE, so a drawn one becomes run.
pauseEngages constant It only disarms the daemon pause, so it is no neutral value of its own.
  • The model no longer reads process.platform, so one seed and path give the same history on every platform (F8, R4).
  • The maps read the same draws, so no field's draws move.
  • The canary's hook test checks what the driver would switch on, by value, over 500 histories at each of 3 seeds:
    • hook fields hold their neutral values;
    • no drawn step arms a daemon pause;
    • no drawn stop switches a hook.
  • The canary failed at c58783d's model ("expected [dieMidCheck, resetBeforeRead] to not include dieMidCheck", after "hookStops undefined").

2. F-D (D50; R9, I8, I12; gate F4).

  • spinData: a mutated copy reads a data file that its file's unmutated run never reads, and its call budget cuts it.
  • The extended oracle found the batch-path defect, and the fix is 971f22e: a spun copy in a batch kept no inputs of its own. One constructor, spunRun, now serves both paths.
  • spinThrows adds an error outside any test, so the batch runs its copies alone and the alone path stores the spun run (F4).
  • That history found a second production defect, fixed in f22ef48:
    • Vitest's state keeps a run's errors outside any test for every later run (its own CLI clears them per run).
    • So after one such error, every later batch in that context ran alone, and each batch retained the errors (the bounded-retention law, errorsSet).
    • Each run now starts with none, and a batch drops its errors once its units are plain data.

3. A daemon request with no bound (D61; R39, P10/I7, I8, I15; gate F2, F5, F7).

  • The daemon sends each waiting CLI a beat (an empty line) whenever a check makes progress.
  • The bound is the socket's own idle timeout (socket.setTimeout(WAIT_MS), 60 s; F5). It is a wall-clock bound: it changes the run's path, never its outcome. It is not a count. I15 does not cover it, because it decides no kill, survival or stop.
  • On the bound, the CLI ends the daemon that holds the package's lock (the one listening on the socket it reached), prints oracle-gaps: note the package's daemon sent nothing for 60 s; this check runs cold, and runs the check cold.
  • Ending the daemon closes the second-writer race (F2): nothing else writes into the run's --out, and no later check waits for that daemon again.
  • Test hooks:
    • ORACLE_GAPS_TEST_WAIT_MARK marks the bound and pauses there.
    • The new daemon-check pause holds a daemon's first armed check before it runs, until the driver's release file exists.
    • daemon-reply honours ORACLE_GAPS_TEST_PAUSE_ARM.

4. #73 class 2 (D58).

  • siblingDir places an in-repository package in a directory beside the CLI's, whose name extends it.
  • A constructed history refuses a run there.
  • The driver judges its "inside the CLI's directory" label against relative(how.cli, dir).

Evidence

Law:

Reach (fixed seed 20261105, all three Linux shards):

  • Measured from CI 43897237634 against main 43877074448, before the gate fixes. No state that some main unit entered is now entered by no unit.
  • Arca at 52f9cdf (before the merge of fix(oracle-gaps): prefix an internal error's lines, and judge the stream law for them #128): all three fixed-seed shards pass, in 204, 283 and 324 s.
  • New required states, each entered by its constructed history, appended last:
    • a check whose daemon stopped answering ran cold
    • an edit to data that only a spun copy read changed the cold result
    • an edit to data that only a copy spun alone read changed the cold result
    • a refused run whose package lies beside the CLI's directory, under a name that extends it
  • Gained by draws: the test-root, load-plugin and F-D states are now reached by generated bases too.
  • Lost from drawn histories: the daemon-pause step states. Each is still gated by its constructed histories (an edit sequence through the daemon, the disarmed pause, the reset after a replaced warm daemon, and the held check).
  • Lost required states, each restored by pinning the constructed history that gates it:
    • a warm daemon's connection was reset, in the daemon edit sequence (warmReplaced: false);
    • edit lazy changed the cold result, in the lazy-module history (spinData: false);
    • a lazy edit changed the cold result after a batch was cut off, in the lazy-hang history (spinData: false; found by the F6 sweep);
    • the inside-the-CLI state, in badEntry (siblingDir: false);
    • the batch-path F-D state, in its history (spinThrows: false).

F6, random sweep (every random seed: unit at 52f9cdf, Arca mc-f6-rand-a and mc-f6-rand-b):

RED (parents; local, under /tmp/e2e-lock.sh):

  • Cold fallback, at 0f10264: law cold fallback, "exited 143 when the driver's backstop stopped it, and the cold run exited 0".
  • F2, at 4fbe231: law cold fallback, "the daemon ran the check whose CLI had fallen back to a cold check, and wrote into that run's --out".
  • F-D batch, at d0e32ec: law incremental equals cold, "after fsData, the daemon's report differs from the cold run's". The daemon gave timeout, and the cold run survived.
  • F-D alone, at 08e967a: law bounded retention, "retention lines break the law: retention (errorsSet)".
  • fix(oracle-gaps): Windows static and e2e failures, by class (follow-up to #58) #73 class 2: main's oracle passes the mutant on Linux and win32 (runs 43881054912 and Arca).
  • Canary, at c0f034b: fails on its own commit's model.

GREEN:

Mutants (each killed unless stated):

Code weight (gzip without line comments; test-time package, C-12, K14 open):

  • daemon.mjs: the beat, the socket timeout, ending the daemon, and the test pause.
  • check.mjs: spunRun (which removes a duplicate constructor), alive, and clearErrors.
  • index.mjs: the fallback and its note.
  • The bound uses the socket's own timeout in place of a counting interval (F5).

Limits:

This pull request and its description were written by Isaac.

KyleAMathews and others added 8 commits November 16, 2026 14:03
…ness wins)

Under the maintainer's 2026-11-16 decision, a history field that changes
what runs is drawn, unless it switches a test-only hook on. warmReplaced
(a scenario), testRoot and loadPlugin (fixture shapes) are now drawn, after
every earlier field. tornRows, pauseEngages and rejectBatch switch hooks, so
they stay constants that only constructed histories set.

A drawn root applies only to an ES module config that Vitest discovers in
the package; a drawn load plugin only to a config in ES module syntax.

Co-authored-by: Isaac <no-reply@databricks.com>
…allback

The witness for R39 that #73 left open. A constructed history's daemon holds
its reply at the test-only daemon-reply pause, and nothing ends it while the
CLI runs. The CLI owes a completed check that equals a cold run. The driver
releases the daemon only after the CLI has ended, by killing it; a CLI that
never ends is ended by the call's 120 s backstop, which the incremental check
reports as a cold-fallback violation.

replyHeld switches a test-only hook on, so it is a constant that only the
constructed history sets, appended last.

Co-authored-by: Isaac <no-reply@databricks.com>
The held check now reaches a daemon whose session is open: a warm check
starts the daemon with the reply pause disarmed (ORACLE_GAPS_TEST_PAUSE_ARM),
then the driver arms it. So the CLI's own bound, not a slow session start,
decides when it stops waiting. dieMidCheck's reply pause takes no arm.

Co-authored-by: Isaac <no-reply@databricks.com>
A check through the daemon had no bound: a daemon that accepted the request
and never answered left the CLI with no done line and no fallback (R39,
found by the merged-PR conformance check of #73).

The daemon now sends each waiting CLI a beat (an empty line) whenever one of
its checks makes progress: a streamed line, or a poll of a running copy. The
CLI's watchdog counts its polls in which nothing came; after 60 s of them
the daemon has stopped answering, and the CLI notes it and runs the check
cold, which reports what the daemon would have (incremental equals cold).

ORACLE_GAPS_TEST_WAIT_MARK now marks that bound and holds the request there,
and the daemon-reply pause waits for ORACLE_GAPS_TEST_PAUSE_ARM when it is
set, so the oracle can warm a daemon before it holds a reply.

Co-authored-by: Isaac <no-reply@databricks.com>
…(F-D)

F-D of #55 (a spun run is stored under the inputs it read) landed with no
RED and no hostile mutant (D50): no history made a cut run read a file that
its file's unmutated run does not read.

spinData, a drawn field, makes file b's verdict tests that read the data
spin on the function they test where its output differs and the data is
open. Each mutated copy that changes the output then reads a file b's
unmutated run never reads, and its call budget cuts it. A constructed
history (appended last) edits the data, closing it: the cold run reports
those mutants unjudged, so a daemon that stored the spun runs under b's
known inputs alone reuses them stale.

Co-authored-by: Isaac <no-reply@databricks.com>
…er a link root

With warmReplaced drawn, the fixed seed's base of "an edit sequence through
the daemon" drew a replaced warm daemon and lost "a warm daemon's connection
was reset", a state that history gates; it now keeps the warm daemon.

With testRoot drawn, a mid-run edit under the link root looked for its
target under tests/, where the fixture keeps no file: the driver now names
the file where the fixture keeps it (fixturePathOf).

Co-authored-by: Isaac <no-reply@databricks.com>
…t read

F-D (#55) stored a run that spun alone under the inputs its closing row
names, but a copy that spun in a batch kept no inputs of its own, so the
daemon stored it under its file's known inputs only. A copy whose mutant
reads a file that the file's unmutated run never reads was then reused after
an edit to that file, while a cold run judged it again (I8). The batch's
spun unit now carries its closing row's inputs, as every other unit does.

Co-authored-by: Isaac <no-reply@databricks.com>
…stories

Co-authored-by: Isaac <no-reply@databricks.com>
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: be3b61e7-7402-4f37-8e4f-d389c08b8cca
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: addd378

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

KyleAMathews and others added 4 commits November 16, 2026 14:33
…r in a batch

The alone path and the batch path built a spun unit separately, so the
batch path lost its inputs (fixed in the previous commit) while the alone
path kept them. Both now build it with spunRun, so the batch history that
witnesses F-D judges the one rule both paths take.

Co-authored-by: Isaac <no-reply@databricks.com>
…uctor

Co-authored-by: Isaac <no-reply@databricks.com>
…he lazy state

From #57's findings: a drawn stop after the completed line holds the run's
exit (ORACLE_GAPS_TEST_HOLD_EXIT_MS), a drawn stop while the config warms
waits at the warm probe (ORACLE_GAPS_TEST_WARM_PROBE), and on win32 every
drawn stop goes through the test stop file (ORACLE_GAPS_TEST_STOP). Under
"Randomness wins" no drawn field switches a hook on, so the histories map
reads those stops as stops while the witness spins, and drops every stop on
win32; only the halves of "every outcome class" stop with a hook. The map
reads the same draws, so no field's draws move.

The canary gains a test: at three seeds, 500 histories each, every hook
field holds its neutral value and no stop switches a hook.

Two fixed-seed failures from spinData: a flaky test of file b spun while
the model leaves flaky tests out (the spin now skips flaky tests, as a hang
does), and "an edit to a module that only mutated runs load" drew spinData
and lost "edit lazy changed the cold result", which it gates; it now pins
spinData off.

Co-authored-by: Isaac <no-reply@databricks.com>
On win32 the histories the oracle takes stop nowhere, so their stopRuns
fingerprint is no draw; the canary keeps POSIX's line and its hook test
checks win32's.

Co-authored-by: Isaac <no-reply@databricks.com>
@pkg-pr-new

pkg-pr-new Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@131
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@131

commit: addd378

KyleAMathews and others added 2 commits November 16, 2026 16:26
# Conflicts:
#	packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
…name that extends it

#73's fix class 2 (isWithin in place of a prefix test in the driver) had no
hostile mutant (D58): the driver's CLI path and package directory were
always spelled alike, so a prefix test agreed with isWithin. A drawn field,
siblingDir, places an in-repository package in a directory beside the CLI's
whose name extends it (oracle-gaps-sibling-*), and a constructed history,
appended last, refuses a run there. The driver now judges the inside label
against the package's path from the CLI's directory, so a prefix test that
attests it for the sibling fails loudly. A drive-letter case or separator
difference is win32-only and is not generated.

Co-authored-by: Isaac <no-reply@databricks.com>
@KyleAMathews
KyleAMathews marked this pull request as ready for review November 16, 2026 16:52
KyleAMathews and others added 4 commits November 16, 2026 17:37
The merge gate's F1: drawn steps still took dieMidCheck and resetBeforeRead,
which arm the daemon pause hooks (13, 6 and 9 of 500 histories at the
canary's seeds), and the canary counted pauseEngages: true as neutral. The
canary now names the hook-arming steps and stops by value, and a win32
stop's hook through a hookStops field. It fails on this commit's model
(RED), as the next commit's fix makes it pass.

Co-authored-by: Isaac <no-reply@databricks.com>
…every platform

The merge gate's F1 and F8. A drawn history no longer takes dieMidCheck or
resetBeforeRead, which arm the daemon pause hooks (v21.2 I16): the histories
map reads them out of the drawn steps, as it reads the hook stops, and the
constructed histories that gate their states keep them. The win32 stop
mapping moves out of the model: a new constant, hookStops, which only the
halves of every outcome class set, tells the driver to run a history's
stops on win32, where each goes through the test stop file. The same seed
and path now give the same history on every platform (v21.2 R4).

Co-authored-by: Isaac <no-reply@databricks.com>
…re it runs

ORACLE_GAPS_TEST_PAUSE_AT=daemon-check holds the first armed check before it
runs, until the file ORACLE_GAPS_TEST_PAUSE_RELEASE names exists: the driver
releases it, never the clock. Inert unless set. The next commit's witness
uses it to hold a daemon before it writes a run's --out (merge gate F2, F7).

Co-authored-by: Isaac <no-reply@databricks.com>
…t runs

The merge gate's F2 and F7. The held check now waits at the daemon-check
pause, before the daemon writes anything into the run's --out, so only a
real cold check gives the run its report. After the CLI has fallen back,
the driver poisons the run's results.json, releases the daemon, and runs one
more check through it. A daemon that then runs the abandoned check writes
over the poison, which the incremental check reports under cold fallback:
the run's printed summary would no longer match its --out.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 6 commits November 16, 2026 17:43
… check

The merge gate's F2: after a fallback the daemon still ran the abandoned
check and wrote into that run's --out, so the run's printed summary no
longer matched its artifacts. On the bound, the CLI now ends the daemon
that holds the package's lock (the one that listens on the socket it
reached) before it checks cold, so nothing else writes into the run's --out,
and no later check waits for that daemon again.

F5: the bound is the socket's own idle timeout (setTimeout), a wall-clock
bound that changes the run's path and never its outcome (v21.2 I15, R39),
in place of a counting interval. F3 is recorded in the Limits with issue
#135.

Co-authored-by: Isaac <no-reply@databricks.com>
…side its test

The merge gate's F4: a batch with an error outside any test runs each copy
alone, and a spun copy's unit then comes from runUnit's spunRun, which no
history reached. A drawn field, spinThrows, makes a copy that spins on open
data first schedule an error that fires after its test, and a constructed
history (appended last) edits the data under it. The batch-path history
pins spinThrows off, so both paths stay witnessed.

Co-authored-by: Isaac <no-reply@databricks.com>
Found by the F-D alone-path history (merge gate F4): Vitest's state keeps a
run's errors outside any test for every later run (its own CLI clears them
per run), so after one such error every later batch in that context read it
as its own and ran its copies alone, and each batch retained the errors
(the bounded-retention law, errorsSet, failed at 08e967a). Each run now
starts with none, and a batch drops its errors once its units are plain
data.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…ore a rewritten --out

Co-authored-by: Isaac <no-reply@databricks.com>
# Conflicts:
#	packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
#	packages/oracles/scripts/e2e-shards.mjs
…able

The merge gate's F6 sweep: random seed 2039724679 gave "a lazy module
loaded before a hang in the same batch" a base that drew spinData, whose
spin pre-empts the lazy-gated verdict, so it lost "a lazy edit changed the
cold result after a batch was cut off", which it gates. It now pins
spinData off, as the other lazy history does.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant