Repository navigation
ci(oracles): run the e2e oracle's random and determinism units on every PR - #90
Open
KyleAMathews wants to merge 16 commits into
Open
KyleAMathews wants to merge 16 commits into
KyleAMathews wants to merge 16 commits into
Conversation
…ry PR CI-1 (random seeds run on every PR) and P12 (every job under 10 minutes). pr-shards.mjs gets two partitions: oracle-gaps-random, in two shards, and oracle-gaps-determinism, in four. Each shard is a longest-first packing of one Arclet run of main at 35422e3: the random units take 624 s and the determinism units 1,337 s, and the largest shard is about 340 s of unit time. pr.yml runs them as blocking Linux jobs, and the preview waits for them. The e2e-shards comment that said CI runs only the fixed-seed job is updated. Co-authored-by: Isaac <no-reply@databricks.com>
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
This was referenced Nov 15, 2026
…h with an issue The determinism job gates every unit except six that fail on main (#91 heartbeat list, #92 daemon note): every outcome class, brokenOwner, noTypescript, the witness module's edit sequence, generated 2, and the daemon's edit sequence. Each excluded unit has its own issue (#93 to #98). pr-shards.mjs takes an excluded list: the rest shard skips it, the partition check fails if a shard holds an excluded name or an excluded name is not a test, and an excluded test is in no shard on purpose. CI ran the random shards about 1.7 times slower than the Arclet, so random is three shards and determinism is two. Co-authored-by: Isaac <no-reply@databricks.com>
commit: |
Maintainer decision, 2026-11-14: heartbeats are progress, not outcome. A heartbeat's set and count depend on wall-clock timing, so the e2e oracle's determinism check compared a timing-dependent trace and failed five units on main (#91). The trace now keeps no heartbeat. The driver observes the breaches of the shape law instead: a beat names the baseline or a mutant of its run, in a file that runs it, and a beat that follows its mutant's own streamed gap line names a mutant that was not running. The check reports a breach as a streaming failure. The presence obligation (a slow owner must be named in a heartbeat) is retired with the set, and so is the model's owed-heartbeat flag. Calibrations feed wrong beat observations to the same check. Co-authored-by: Isaac <no-reply@databricks.com>
…ery run's beats The stability rerun and the partial run observed their CLI runs with their heartbeat lists still in them, so the determinism check compared timing in two more observations. Every observed run now drops its beats, and each run that is observed also observes its shape breaches. The check judges every beats observation. Co-authored-by: Isaac <no-reply@databricks.com>
The reach state "a heartbeat named a busy mutant" is derived from a beat's timing, so determinism compared a timing-dependent observation. Under the maintainer's 2026-11-14 decision that heartbeats are progress, the state is no longer observed or required. #84 owns restoring its coverage through a deterministic pause point. Recorded in the Limits. Co-authored-by: Isaac <no-reply@databricks.com>
The reach state "a timed-out run reused after an edit" compares the cut counts of a daemon check and a cold run. Those counts are timing, so the state depended on wall-clock timeouts, and the determinism check failed on the witness module's edit sequence. It is no longer observed or required, and the witness history no longer enters it. Recorded in the Limits. Co-authored-by: Isaac <no-reply@databricks.com>
The removed reach state was the witness history's only enters entry, so the file failed to load. Co-authored-by: Isaac <no-reply@databricks.com>
#92) With the heartbeat fix, the five units that failed on main pass, so the determinism partition takes them back: the witness module's edit sequence in its own shard, and the other 49 units in three packed shards and a rest shard, five jobs in all. Only #92 stays excluded. The partition check now fails on a shard name that matches no test, which the explicit names had never been checked for. Co-authored-by: Isaac <no-reply@databricks.com>
The e2e gap oracle's constructed histories are built from seed + index, so a history inserted mid-list changes every later one (generator rule, register 2026-11-14: new constructed histories are appended at the end). The canary records the registered names in oracle-gaps/constructed-order.json and fails unless the current list begins with them. Co-authored-by: Isaac <no-reply@databricks.com>
The Test job's e2e check failed: the retired calibration is still named, and the two new heartbeat calibrations are in no shard. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The determinism check compared stderrGaps and stdoutGaps in the order the CLI printed them. A context's gap line is printed when its copies finish, so the order is timing. The stream's own comment already says the trace records the stream as relations, so these two lists are sorted like the streamed list, which is sorted. Nine determinism units failed on main (4d3c8a9) and on the merged head for this reason. Also the constructed-order canary lists vitest with --json=<file>, so no package manager's stdout reaches the JSON (its Vitest 5 job failed on that). Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Nov 15, 2026
… the canary's listing to Vitest 4 The determinism unit added by #73, a daemon pause that cannot engage, gives different traces across runs (#105), so it is excluded until fixed, as the other main failures are. The constructed-order canary lists the gap oracle only under Vitest 4: the Vitest 5 job runs no gap oracle, and its listing finds no suite in that file. Co-authored-by: Isaac <no-reply@databricks.com>
The Windows unit suite failed the canary: spawning pnpm finds no program on Windows. The listing now runs Vitest's CLI with this Node executable. Co-authored-by: Isaac <no-reply@databricks.com>
…106) The witness module's edit sequence takes 10 to 11 minutes in CI, over P12's budget. It is one history and cannot be split without changing the law, so it runs in its own continue-on-error job, owned by #106, which is not a dependency of preview. Shard 1 of the partition is unchanged, so the partition check holds. Co-authored-by: Isaac <no-reply@databricks.com>
The random shards took 9 to 12 minutes on some seeds (PR #90, seed-measure runs). Each unit's declared weight is its longest measured time over six pinned seeds, packed longest first into five shards of about 344 s of weight each, with the fifth as the rest shard. The partition check and exactNames hold. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Nov 16, 2026
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
…multiset The order of streamed lines between runs is not contract (C, "Progress lines in determinism", 2026-11-15, and the C entry under P10 and P11). The e2e oracle's cli observation recorded the stderr gap lines in stream order, so two runs of a history whose copies finish in another order (two contexts: "two batches each read a capture row while its worker wrote it", and the witness-module histories) recorded different traces. The observation now sorts them and keeps duplicates. No law reads their order; #90 brings the general sort to the determinism comparison. Co-authored-by: Isaac <no-reply@databricks.com>
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs the end-to-end gap oracle's random units and determinism units on every pull request (CI-1), in blocking Linux jobs, plus a constructed-order canary. Closes #91.
pr-shards.mjs:oracle-gaps-random(3 shards) andoracle-gaps-determinism(5 shards: the witness module's edit sequence alone, three packed shards, and a rest shard). Shard 1, the witness unit, runs in its own non-blocking job (test-oracle-gaps-witness, continue-on-error, owner P12: the witness module's determinism unit alone takes 9 to 11 minutes in CI #106). It runs on every PR and is not a dependency ofpreview. The determinism partition has anexcludedlist, which has two entries (below).exactNames(from main, fix(oracle-gaps): Windows static and e2e failures, by class (follow-up to #58) #73) checks every explicit name, so a name that matches no test fails the check.pr.yml: the random and determinism jobs are blocking Linux jobs;previewwaits for them. Windows is unchanged.gap-report-oracle.property.test.mjs: determinism judges heartbeats for shape only, not as a set (maintainer decision, 2026-11-14); the stream's gap lines are compared as a set, not in print order.tests/gap-report-history-fingerprint.test.tsandoracle-gaps/constructed-order.json: the constructed-order canary.origin/main(4d3c8a9, fix(oracle-gaps): Windows static and e2e failures, by class (follow-up to #58) #73).Decisions and authority
Evidence
Heartbeats: shape only (Law, Reach, RED, GREEN, Mutant, Limits)
beats, which is empty in a correct run. Two reach states that depended on timing were removed from observation and fromrequired: "a heartbeat named a busy mutant" (reach: "a heartbeat named a busy mutant" depends on wall-clock timing and is reached 1 to 2 times per shard #84 owns restoring it) and "a timed-out run reused after an edit" (reach: a timed-out run reused after an edit depends on timeouts, so its coverage is no longer required (#90) #104).every outcome class,brokenOwner,noTypescript,an edit sequence over the witness module, andgenerated 2fail at observation 0 (cli). Runsci-e2e-red-aandci-e2e-red-b. These are the determinism: the CLI's heartbeat list depends on wall-clock timing, so five determinism units fail on main #91 units.ci-e2e-g5-a,-b), and the other 45 pass. On 4bc4eb1, all 54 determinism units pass (runsci-g6-s1to-s4), including determinism: an edit sequence through the daemon records a test-output note in one run and not the other #92.beatsobservation to the same check, and each one must fail by the streaming law: a beat in a file that does not run its mutant, a beat that follows its mutant's gap line, and a beat naming no mutant of its run. They pass (runsci-e2e-final-cal,ci-e2e-green3-cal), so the check rejects each. A hostile driver (one that skips the finished-line test) is not killed by any real history, because no real run produces such a beat. That is a limit.Stream order: gap lines compared as sets (RED, GREEN, Limits)
ci-main4-s1to-s4). Among them, the nine units that also fail on the merged head fail at observation 0 (cli) onstderrGapsandstdoutGaps, which were compared in print order.ci-g6-s1to-s4).Constructed-order canary (Law, RED, GREEN, Mutant, Limits)
seed + index, so a history inserted before a recorded one changes every later one (the generator rule above). The canary records the registered names inoracle-gaps/constructed-order.jsonand fails unless the current list begins with them.a declaration fault: ${fault}template expands to 22 names.ci-canary-red4,AssertionError).ci-canary-green4, 2 of 2).Excluded determinism units (Limits)
Two units are excluded, each with an issue:
constructed history: an edit sequence through the daemon(determinism: an edit sequence through the daemon records a test-output note in one run and not the other #92): the test-output note appears in one run and not the other. Its cause is not determined. It passed in the last two runs, so it looks intermittent.constructed history: a daemon pause that cannot engage(determinism: a daemon pause that cannot engage gives different traces across runs (intermittent, added by #73) #105): intermittent. It passed once and failed once on 4bc4eb1.Counterexamples filed (P4)
random seed: generatedfailed once in CI on 082991d (run 43688703700, shard 3/3): seed 333199054, path1:1:1:1, "the stdout has 189 gap lines for 190 gaps". It reproduces onorigin/main(5726197, twice) and on the PR head (twice), so the base has it. Under the M phase-1 rule it is filed, not blocking: #102. A second one,random seed: a declaration fault: brokenOwner(seed -1269403528), reproduced on main and on the head: #107. The printed Replay line also does not reproduce the unit (it omits--config oracle-gaps/vitest.config.mjs gap-report-oracle); that is in #102.Job times (CI run 43725781217, head 7baf1d2)
Job totals include setup (about 15 to 30 s).
Two budget notes. The random shards' cost follows the drawn seed: the same shard took 6:39 and 7:28 on earlier heads and 10:07 here. The witness unit took 7:42 here and 10:11 to 11:08 on earlier heads (#106). Both are CI variance, and both need cost work before the budget holds on every seed.
Earlier runs on this branch, for the record: 082991d random 3/3 failed (#102). 4bb5c08 had all oracle jobs passing, witness 11:08, random 3/3 7:28.
Seeding (checked, not changed)
ORACLE_SEED {...}. In CI on 082991d the three random shards logged three seeds: 1835238112, -1659558018, -330614923.Limits
test-oracle-gaps-witness(continue-on-error), so it still runs on every PR and does not block. It is not dropped. P12: the witness module's determinism unit alone takes 9 to 11 minutes in CI #106 cuts its cost, and then the job becomes blocking. That is at or over the 10-minute budget. One history cannot be split without changing the law.random seed: every outcome class, classification, seed -972199262, path 0:1), each reproduced on main. Under the M phase-1 rule each is filed, not blocking. The random 2/3 job failed on random seed: every outcome class: a classification counterexample (seed -972199262, path 0:1), on main too #109 in the last run.Closes #91.
This pull request and its description were written by Isaac.