Skip to content

ci(oracles): run the e2e oracle's random and determinism units on every PR - #90

Open
KyleAMathews wants to merge 16 commits into
mainfrom
ci/e2e-random-determinism
Open

KyleAMathews wants to merge 16 commits into
mainfrom
ci/e2e-random-determinism

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Runs the end-to-end gap oracle's random units and determinism units on every pull request (CI-1), in blocking Linux jobs, plus a constructed-order canary. Closes #91.

Decisions and authority

  • CI-1 (register M, 2026-11-12): random seeds run on every PR. Each run draws a fresh seed (checked below).
  • P4 (register): a counterexample is filed as an issue with its replay line, and a rerun never hides it. See the counterexample below.
  • M phase-1 (register, 2026-11-12): a failure that main also has does not block. The register's text covers random counterexamples. This PR reads it as covering determinism units too, which the maintainer confirmed on 2026-11-14.
  • Phase-1 reading for determinism, confirmed (register M, 2026-11-14): a determinism failure that main also has does not block.
  • Heartbeats are progress, not outcome (register M, 2026-11-14). The determinism check compares results and fixed lines. Heartbeats are checked for shape only: each must name a mutant of its run that was still running, in a file that runs it.
  • Generator rule, mechanism clause (register, 2026-11-14): "New constructed histories are appended at the end of the constructed list, since each one samples seed+index and inserting one mid-list changes every later history."
  • P12 (register): every blocking job under 10 minutes. The witness unit is over, and it is non-blocking until P12: the witness module's determinism unit alone takes 9 to 11 minutes in CI #106 cuts its cost (coordinator ruling).

Evidence

Heartbeats: shape only (Law, Reach, RED, GREEN, Mutant, Limits)

Stream order: gap lines compared as sets (RED, GREEN, Limits)

  • Law: the order of streamed gap lines between runs is not part of the contract, and results.json is the stable record (coordinator's C entry). The stream is recorded as relations (law 15, the file's own comment), so a gap line's place in the output is not judged.
  • RED: main 4d3c8a9, 12 determinism units fail (runs ci-main4-s1 to -s4). Among them, the nine units that also fail on the merged head fail at observation 0 (cli) on stderrGaps and stdoutGaps, which were compared in print order.
  • GREEN: 4bc4eb1, 54 of 54 pass (runs ci-g6-s1 to -s4).

Constructed-order canary (Law, RED, GREEN, Mutant, Limits)

  • Law: a constructed history is built from seed + index, so a history inserted before a recorded one changes every later one (the generator rule above). The canary records the registered names in oracle-gaps/constructed-order.json and fails unless the current list begins with them.
  • Record: 51 names. Main's 30 declared entries expand to 51 registered names, because the a declaration fault: ${fault} template expands to 22 names.
  • RED: scratch commit 818d1b2 (on this branch's head) inserts a constructed history at index one. The canary fails (run ci-canary-red4, AssertionError).
  • GREEN: 947e936 passes (run ci-canary-green4, 2 of 2).
  • Mutant: the RED insertion.
  • Limits: the canary runs under Vitest 4 only, because the Vitest 5 job runs no gap oracle (its listing finds no suite in the file).

Excluded determinism units (Limits)

Two units are excluded, each with an issue:

Counterexamples filed (P4)

random seed: generated failed once in CI on 082991d (run 43688703700, shard 3/3): seed 333199054, path 1:1:1:1, "the stdout has 189 gap lines for 190 gaps". It reproduces on origin/main (5726197, twice) and on the PR head (twice), so the base has it. Under the M phase-1 rule it is filed, not blocking: #102. A second one, random seed: a declaration fault: brokenOwner (seed -1269403528), reproduced on main and on the head: #107. The printed Replay line also does not reproduce the unit (it omits --config oracle-gaps/vitest.config.mjs gap-report-oracle); that is in #102.

Job times (CI run 43725781217, head 7baf1d2)

Job totals include setup (about 15 to 30 s).

Job Job total Result
Gap report end-to-end oracle (determinism, witness module; non-blocking, #106) 7:42 passed (non-blocking)
Gap report end-to-end oracle (determinism, 2/5) 5:42 passed
Gap report end-to-end oracle (determinism, 3/5) 5:27 passed
Gap report end-to-end oracle (determinism, 4/5) 7:26 passed
Gap report end-to-end oracle (determinism, 5/5) 9:38 passed
Gap report end-to-end oracle (random seed, 1/3) 10:07 passed; over the 10-minute budget for this seed
Gap report end-to-end oracle (random seed, 2/3) 12:21 failed: a counterexample, #109 (below)
Gap report end-to-end oracle (random seed, 3/3) 7:47 passed

Two budget notes. The random shards' cost follows the drawn seed: the same shard took 6:39 and 7:28 on earlier heads and 10:07 here. The witness unit took 7:42 here and 10:11 to 11:08 on earlier heads (#106). Both are CI variance, and both need cost work before the budget holds on every seed.

Earlier runs on this branch, for the record: 082991d random 3/3 failed (#102). 4bb5c08 had all oracle jobs passing, witness 11:08, random 3/3 7:28.

Seeding (checked, not changed)

  • Each Vitest process draws a fresh campaign seed at its first random unit and logs ORACLE_SEED {...}. In CI on 082991d the three random shards logged three seeds: 1835238112, -1659558018, -330614923.

Limits

Closes #91.

This pull request and its description were written by Isaac.

…ry PR

CI-1 (random seeds run on every PR) and P12 (every job under 10 minutes).
pr-shards.mjs gets two partitions: oracle-gaps-random, in two shards, and
oracle-gaps-determinism, in four. Each shard is a longest-first packing of
one Arclet run of main at 35422e3: the random units take 624 s and the
determinism units 1,337 s, and the largest shard is about 340 s of unit
time. pr.yml runs them as blocking Linux jobs, and the preview waits for
them. The e2e-shards comment that said CI runs only the fixed-seed job is
updated.

Co-authored-by: Isaac <no-reply@databricks.com>
@changeset-bot

changeset-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e558496

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: bdd5cc47-a8e6-4a2b-946a-0976d7310537
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…h with an issue

The determinism job gates every unit except six that fail on main
(#91 heartbeat list, #92 daemon note): every outcome class, brokenOwner,
noTypescript, the witness module's edit sequence, generated 2, and the
daemon's edit sequence. Each excluded unit has its own issue (#93 to #98).
pr-shards.mjs takes an excluded list: the rest shard skips it, the partition
check fails if a shard holds an excluded name or an excluded name is not a
test, and an excluded test is in no shard on purpose.

CI ran the random shards about 1.7 times slower than the Arclet, so random
is three shards and determinism is two.

Co-authored-by: Isaac <no-reply@databricks.com>
@pkg-pr-new

pkg-pr-new Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@90
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@90

commit: 4bb5c08

KyleAMathews and others added 8 commits November 14, 2026 23:23
Maintainer decision, 2026-11-14: heartbeats are progress, not outcome. A
heartbeat's set and count depend on wall-clock timing, so the e2e oracle's
determinism check compared a timing-dependent trace and failed five units on
main (#91).

The trace now keeps no heartbeat. The driver observes the breaches of the
shape law instead: a beat names the baseline or a mutant of its run, in a
file that runs it, and a beat that follows its mutant's own streamed gap line
names a mutant that was not running. The check reports a breach as a
streaming failure. The presence obligation (a slow owner must be named in a
heartbeat) is retired with the set, and so is the model's owed-heartbeat
flag. Calibrations feed wrong beat observations to the same check.

Co-authored-by: Isaac <no-reply@databricks.com>
…ery run's beats

The stability rerun and the partial run observed their CLI runs with their
heartbeat lists still in them, so the determinism check compared timing in
two more observations. Every observed run now drops its beats, and each run
that is observed also observes its shape breaches. The check judges every
beats observation.

Co-authored-by: Isaac <no-reply@databricks.com>
The reach state "a heartbeat named a busy mutant" is derived from a beat's
timing, so determinism compared a timing-dependent observation. Under the
maintainer's 2026-11-14 decision that heartbeats are progress, the state is no
longer observed or required. #84 owns restoring its coverage through a
deterministic pause point. Recorded in the Limits.

Co-authored-by: Isaac <no-reply@databricks.com>
The reach state "a timed-out run reused after an edit" compares the cut counts
of a daemon check and a cold run. Those counts are timing, so the state
depended on wall-clock timeouts, and the determinism check failed on the
witness module's edit sequence. It is no longer observed or required, and
the witness history no longer enters it. Recorded in the Limits.

Co-authored-by: Isaac <no-reply@databricks.com>
The removed reach state was the witness history's only enters entry, so the
file failed to load. Co-authored-by: Isaac <no-reply@databricks.com>
#92)

With the heartbeat fix, the five units that failed on main pass, so the
determinism partition takes them back: the witness module's edit sequence
in its own shard, and the other 49 units in three packed shards and a rest
shard, five jobs in all. Only #92 stays excluded. The partition check now
fails on a shard name that matches no test, which the explicit names had
never been checked for.

Co-authored-by: Isaac <no-reply@databricks.com>
The e2e gap oracle's constructed histories are built from seed + index, so a
history inserted mid-list changes every later one (generator rule, register
2026-11-14: new constructed histories are appended at the end). The canary
records the registered names in oracle-gaps/constructed-order.json and fails
unless the current list begins with them.

Co-authored-by: Isaac <no-reply@databricks.com>
The Test job's e2e check failed: the retired calibration is still named, and
the two new heartbeat calibrations are in no shard.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 2 commits November 15, 2026 01:20
Co-authored-by: Isaac <no-reply@databricks.com>
The determinism check compared stderrGaps and stdoutGaps in the order the CLI
printed them. A context's gap line is printed when its copies finish, so the
order is timing. The stream's own comment already says the trace records the
stream as relations, so these two lists are sorted like the streamed list,
which is sorted. Nine determinism units failed on main (4d3c8a9) and on the
merged head for this reason.

Also the constructed-order canary lists vitest with --json=<file>, so no
package manager's stdout reaches the JSON (its Vitest 5 job failed on that).

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 2 commits November 15, 2026 02:49
… the canary's listing to Vitest 4

The determinism unit added by #73, a daemon pause that cannot engage, gives
different traces across runs (#105), so it is excluded until fixed, as the
other main failures are. The constructed-order canary lists the gap oracle
only under Vitest 4: the Vitest 5 job runs no gap oracle, and its listing finds
no suite in that file.

Co-authored-by: Isaac <no-reply@databricks.com>
The Windows unit suite failed the canary: spawning pnpm finds no program on
Windows. The listing now runs Vitest's CLI with this Node executable.

Co-authored-by: Isaac <no-reply@databricks.com>
…106)

The witness module's edit sequence takes 10 to 11 minutes in CI, over P12's
budget. It is one history and cannot be split without changing the law, so it
runs in its own continue-on-error job, owned by #106, which is not a dependency
of preview. Shard 1 of the partition is unchanged, so the partition check holds.

Co-authored-by: Isaac <no-reply@databricks.com>
The random shards took 9 to 12 minutes on some seeds (PR #90, seed-measure
runs). Each unit's declared weight is its longest measured time over six
pinned seeds, packed longest first into five shards of about 344 s of weight
each, with the fifth as the rest shard. The partition check and exactNames
hold.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
…multiset

The order of streamed lines between runs is not contract (C, "Progress lines
in determinism", 2026-11-15, and the C entry under P10 and P11). The e2e
oracle's cli observation recorded the stderr gap lines in stream order, so
two runs of a history whose copies finish in another order (two contexts:
"two batches each read a capture row while its worker wrote it", and the
witness-module histories) recorded different traces. The observation now
sorts them and keeps duplicates. No law reads their order; #90 brings the
general sort to the determinism comparison.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

determinism: the CLI's heartbeat list depends on wall-clock timing, so five determinism units fail on main

1 participant