Skip to content

test(oracle-gaps): two checks at once are recorded by role, not by process order (#82) - #126

Open
KyleAMathews wants to merge 7 commits into
mainfrom
fix/daemon-edit-sequence-race
Open

KyleAMathews wants to merge 7 commits into
mainfrom
fix/daemon-edit-sequence-race

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Refs #92: this resolves its edit-sequence instance (the second failure in #82, "an edit sequence through the daemon"). #92 stays open for its other instance, #57's single-CLI history a daemon check with the default output directory, which this race cannot cause (comment). #82 stays open for its hang history (observation 18, "reach"). #98 stays open until #90's exclusion list drops the unit.

What was wrong

The history begins with two checks at once (concurrentStart). The daemon runs one check at a time, from the FIFO queue in daemon.mjs (queue = queue.then(...), and its header: "One check runs at a time"). Whichever check it runs first computes every unit, and the other reuses them. The driver recorded the pair in the order it started the two processes. The note test output line belongs to the check that ran the tests, so the trace showed which process reached the daemon first, and the determinism unit failed whenever two runs took different orders.

Option chosen: B, the model accepts both orders, with the overlap held

The order of two concurrent checks is a race between two processes. The contract does not promise it: the README promises reuse while inputs are unchanged and says nothing about which of two checks runs first. Option A would pin an order that the law leaves free. Option B pins only what the law needs:

  1. The pair is recorded by role, in a total order. The check that computed more runs comes first, as the daemon's earlier check. Ties are ordered by each check's serialized record (byRole). Each check is still judged in full, so nothing is relaxed.
  2. The later check owes unchanged reuse. It computes no run that ended without a cut, exactly as a recheck does. "One computed, the other reused" was not judged before this PR.
  3. The overlap is held, not timed (R57). A new test-only hold, ORACLE_GAPS_TEST_HOLD_OVERLAP (G43; documented in the README's test hooks), makes a daemon hold the first check it runs until it has accepted a second check request. Then it writes that count to a mark. It holds once per daemon, is released by the count and never by time, and changes no outcome. It is switched on only by a constructed-only field, holdOverlap: fc.constant(false) in the drawn record, appended last, recorded in history-fingerprint.json, and returned to false by the take-back. Only the edit sequence through the daemon sets it. The new required state is entered only when the mark shows the held overlap. Two sequential checks cannot enter it. The hold pins the overlap, not the order: either check may still be the one the daemon runs first.
  4. Runs are counted, not cut events. The driver's old measure computed − cut counted cut events, so it could go below zero and hide a run that was computed again. A cut copy confirmed alone adds a cut and no computed run (check.mjs). This needed a production counter: evidence.ran in results.json counts the units a check computed that ended without a cut. It has a changeset. The driver reads it.
  5. Each check's own output directory is named <out> in its stream records, and only as a whole path. A sibling whose name begins with it (out-daemon-2-3 beside out-daemon-2) keeps its own name.
  6. New law: a completed check through the daemon names its own report in its done line (… report <package>/<out>/summary.md).

Proposed register entry (C; the coordinator records it)

Two checks at once (#126): of two checks of one state through one daemon, the one the daemon runs second computes no run that ends without a cut (law unchanged reuse, judged by the CLI's evidence.ran). Authority: the README's reuse contract ("a run is reused while its inputs are unchanged"). The law relies on an implementation fact, not a promise: the daemon serves one check at a time from the FIFO queue in daemon.mjs, and the earlier check's units are stored and flushed before its reply. Whether "one check at a time" is contract is flagged for M. It inherits C-7's exemption: tests that read the clock (t.flaky) are not judged. C-7 is not among the 15 C entries ratified on 2026-11-16 (v21.2 §12), so the exemption is pending ratification. Domain: the constructed edit sequence through the daemon, with its overlap held, and every generated history that draws concurrentStart, crashAtOnce or cliAtOnce, so random seeds judge the law on every PR (P4). Which check the daemon runs first is outside the contract, so traces record the pair by role.

v21.2 IDs this touches: I8 (incremental equals cold), I18 (determinism compares results.json and the fixed lines), R52 (progress lines are not outcome), R57 and G43 (test-only hooks; holds released by the driver or a count, never by time: C-30, now M), R9 / P9 (oracle-first), R42 / M-20 (main's determinism failures are filed), P4 (random seeds judge the new law), P10 (the done line, the new law of point 6), G42 / G34 (the daemon as a cache over the log).

Evidence

  • Law: of two checks at once on one state, the later computes no run that ends without a cut. The pair is recorded by role. A completed daemon check names its own report. The authority and scope are in the register entry above.
  • Reach: the new required state of two checks held in the daemon at once, one computed and the other reused is entered 1 time in each run of fixed seed 20261105: an edit sequence through the daemon at the head (3 of 3 runs, f82-e3-fs-{1,2,3}). The framework fails a constructed history that does not enter a state it names.
  • RED:
    • Oracle-only commit 24afe68 (R9: the driver, model and fingerprint, with no hold and no ran in the CLI): fixed seed 20261105: an edit sequence through the daemon fails: "did not enter of two checks held in the daemon at once, one computed and the other reused" (f82-e3-red-1).
    • Determinism unit, on main (fac63df, after fix(oracle-gaps): retire a batch's copies from Vitest's state #61): 8 of 10 fail at observation 4 ("incremental"). Each failure differs only in the note test output line. At 9aed105: 4 of 10. A scratch driver that runs the two checks sequentially, in opposite orders on alternate runs, fails 2 of 2 on main. The command for all three: cd packages/oracles && pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "determinism: constructed history: an edit sequence through the daemon$".
  • GREEN (head d65293e, Arca):
    • Determinism unit: 20 of 20 pass (f82-e3-green-{1..20}).
    • The same scratch driver (sequential, opposite orders, hold off) passes 2 of 2 (f82-e3-diag-{1,2}).
    • Fixed-seed unit: 3 of 3 pass. Random-seed unit random seed: an edit sequence through the daemon: 2 of 2 pass.
  • Mutant ("both checks compute"): in the daemon, a check whose request arrives while another check's connection is in flight runs against an empty store. Its results equal cold.
    • At the head, under the held overlap, it is killed 3 of 3 (f82-e3-mut-{1,2,3}): law: unchanged reuse, "the check through the daemon at start (a second check at once), of a state that nothing changed, computed 8 runs that were not cut off, owed none".
    • The hold makes the second request arrive while the first check is in flight by a count, so the kill does not depend on timing.
    • At fac63df, without the law, it survives (f82-d3-mut-main-1).
    • An earlier variant that counted only requests still queued did not engage without the hold, and the old ungated state was still entered. That is the gap R57 closes here.
  • Limits:
    • Only the constructed edit sequence holds its overlap. Generated histories with two checks at once still judge the reuse law, but their overlap is neither held nor measured, so they claim no reach.
    • The pre-existing state two checks started the daemon at once is not gated on overlap.
    • No history reaches a cut inside a pair of checks at once, so the difference between ran and computed − cut has no witness. ran is exact by construction (it counts at the three places a unit ends without a cut).
    • Determinism units run locally only (L5; ci(oracles): run the e2e oracle's random and determinism units on every PR #90 not merged), so the Arca runs above are their evidence.
    • Under the diagnostic driver the two checks run sequentially, with the hold off: a sequential pair cannot overlap.

g1-ff5 triage: filed as #124

A production counterexample observed through a complete artifact (C-31). It is not a driver timing assumption: at start the history runs one CLI, a plain runCli with no hook or hold. The daemon's results.json blocked sut.ts:M1 on a failing tests/a.test.ts baseline that the cold run passed. The cause is not determined. #124 now has the full counterexample history (M "Replay by history").

CI

Run 43919288577 at d65293e is green overall.

Weight

Code in the test-time package only, so zero shipped bytes (C "Zero shipped bytes"):

  • daemon.mjs +20 lines: the hold.
  • check.mjs +4 counters and a comment.

Tests and docs:

  • gap-report-oracle.property.test.mjs +196/−37.
  • The model's constant field.
  • The fingerprint.
  • The README hook text.
  • A changeset.

This pull request and its description were written by Isaac.

KyleAMathews and others added 3 commits November 16, 2026 13:29
…er one reuse

Two checks at once reach one daemon, which runs one check at a time in the
order their requests arrive. That order is a race between two processes and
no part of the contract, so the driver recorded the pair by which process it
started first, and the determinism unit of "an edit sequence through the
daemon" saw the test-output note on either check (#82, #92).

The driver now records the pair by what each check did: the check that
computed more runs comes first. The later check of one state owes the reuse
law that a recheck owes (unchanged reuse): it computes no run that was not
cut off, where no test reads the clock. A new reach state, "of two checks at
once, one computed and the other reused", is required with law 9's states.

Co-authored-by: Isaac <no-reply@databricks.com>
…nce-race

Co-authored-by: Isaac <no-reply@databricks.com>
The second witness of #82's race: with the pair recorded by role, a GREEN
run of the determinism unit still differed, in the done line. Each of two
checks at once gets its own output directory (out-daemon, out-daemon-2),
and the done line names it, so the trace still told which process the
daemon ran first. The stream records now name the run's own directory
<out>, as its notes already did; a line that names another run's directory
still shows it.

Co-authored-by: Isaac <no-reply@databricks.com>
@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: d65293e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@tanstack/oracles Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e5f396b9-59d1-44f3-b669-d05d71a29c24
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pkg-pr-new

pkg-pr-new Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@126
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@126

commit: d65293e

KyleAMathews and others added 4 commits November 16, 2026 17:24
…nce-race

Co-authored-by: Isaac <no-reply@databricks.com>
…e the done line

Review of #126:
- F2: the reach state for two checks at once now needs a held overlap. A
  constructed-only field, holdOverlap (fc.constant(false) in the drawn
  record, recorded in the fingerprint, set by the edit sequence through
  the daemon), arms the test-only hold ORACLE_GAPS_TEST_HOLD_OVERLAP. The
  state is entered only when the daemon reports that it held its first
  check until it had accepted the second request.
- F3: computedRuns reads the CLI's count of runs that ended without a
  cut (evidence.ran), not computed - cut, which counts cut events.
- F4: a run's own output directory is named <out> only as a whole path.
- F5: a completed check through the daemon names its own report in its
  done line.
- F6: ties between the two checks are ordered by their records.

This commit is oracle-only: against the CLI without the hold and the count,
the constructed history cannot enter its state (RED).

Co-authored-by: Isaac <no-reply@databricks.com>
…t of runs that ended without a cut

- ORACLE_GAPS_TEST_HOLD_OVERLAP (a path): a daemon holds the first check it
  runs until it has accepted a second check request, then writes that count
  to the path. Once per daemon, released by the count, never by time. It
  changes no outcome (G43, R57). Documented in the README's test hooks.
- evidence.ran in results.json: the units a check computed that ended
  without a cut. `cut` counts cut events, so computed - cut is not a count
  of runs (review of #126, F3).

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant