Repository navigation
measure: random shard cost per seed (scratch, do not merge) - #110
KyleAMathews wants to merge 17 commits into
Conversation
…ry PR CI-1 (random seeds run on every PR) and P12 (every job under 10 minutes). pr-shards.mjs gets two partitions: oracle-gaps-random, in two shards, and oracle-gaps-determinism, in four. Each shard is a longest-first packing of one Arclet run of main at 35422e3: the random units take 624 s and the determinism units 1,337 s, and the largest shard is about 340 s of unit time. pr.yml runs them as blocking Linux jobs, and the preview waits for them. The e2e-shards comment that said CI runs only the fixed-seed job is updated. Co-authored-by: Isaac <no-reply@databricks.com>
…h with an issue The determinism job gates every unit except six that fail on main (#91 heartbeat list, #92 daemon note): every outcome class, brokenOwner, noTypescript, the witness module's edit sequence, generated 2, and the daemon's edit sequence. Each excluded unit has its own issue (#93 to #98). pr-shards.mjs takes an excluded list: the rest shard skips it, the partition check fails if a shard holds an excluded name or an excluded name is not a test, and an excluded test is in no shard on purpose. CI ran the random shards about 1.7 times slower than the Arclet, so random is three shards and determinism is two. Co-authored-by: Isaac <no-reply@databricks.com>
Maintainer decision, 2026-11-14: heartbeats are progress, not outcome. A heartbeat's set and count depend on wall-clock timing, so the e2e oracle's determinism check compared a timing-dependent trace and failed five units on main (#91). The trace now keeps no heartbeat. The driver observes the breaches of the shape law instead: a beat names the baseline or a mutant of its run, in a file that runs it, and a beat that follows its mutant's own streamed gap line names a mutant that was not running. The check reports a breach as a streaming failure. The presence obligation (a slow owner must be named in a heartbeat) is retired with the set, and so is the model's owed-heartbeat flag. Calibrations feed wrong beat observations to the same check. Co-authored-by: Isaac <no-reply@databricks.com>
…ery run's beats The stability rerun and the partial run observed their CLI runs with their heartbeat lists still in them, so the determinism check compared timing in two more observations. Every observed run now drops its beats, and each run that is observed also observes its shape breaches. The check judges every beats observation. Co-authored-by: Isaac <no-reply@databricks.com>
The reach state "a heartbeat named a busy mutant" is derived from a beat's timing, so determinism compared a timing-dependent observation. Under the maintainer's 2026-11-14 decision that heartbeats are progress, the state is no longer observed or required. #84 owns restoring its coverage through a deterministic pause point. Recorded in the Limits. Co-authored-by: Isaac <no-reply@databricks.com>
The reach state "a timed-out run reused after an edit" compares the cut counts of a daemon check and a cold run. Those counts are timing, so the state depended on wall-clock timeouts, and the determinism check failed on the witness module's edit sequence. It is no longer observed or required, and the witness history no longer enters it. Recorded in the Limits. Co-authored-by: Isaac <no-reply@databricks.com>
The removed reach state was the witness history's only enters entry, so the file failed to load. Co-authored-by: Isaac <no-reply@databricks.com>
#92) With the heartbeat fix, the five units that failed on main pass, so the determinism partition takes them back: the witness module's edit sequence in its own shard, and the other 49 units in three packed shards and a rest shard, five jobs in all. Only #92 stays excluded. The partition check now fails on a shard name that matches no test, which the explicit names had never been checked for. Co-authored-by: Isaac <no-reply@databricks.com>
The e2e gap oracle's constructed histories are built from seed + index, so a history inserted mid-list changes every later one (generator rule, register 2026-11-14: new constructed histories are appended at the end). The canary records the registered names in oracle-gaps/constructed-order.json and fails unless the current list begins with them. Co-authored-by: Isaac <no-reply@databricks.com>
The Test job's e2e check failed: the retired calibration is still named, and the two new heartbeat calibrations are in no shard. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The determinism check compared stderrGaps and stdoutGaps in the order the CLI printed them. A context's gap line is printed when its copies finish, so the order is timing. The stream's own comment already says the trace records the stream as relations, so these two lists are sorted like the streamed list, which is sorted. Nine determinism units failed on main (4d3c8a9) and on the merged head for this reason. Also the constructed-order canary lists vitest with --json=<file>, so no package manager's stdout reaches the JSON (its Vitest 5 job failed on that). Co-authored-by: Isaac <no-reply@databricks.com>
… the canary's listing to Vitest 4 The determinism unit added by #73, a daemon pause that cannot engage, gives different traces across runs (#105), so it is excluded until fixed, as the other main failures are. The constructed-order canary lists the gap oracle only under Vitest 4: the Vitest 5 job runs no gap oracle, and its listing finds no suite in that file. Co-authored-by: Isaac <no-reply@databricks.com>
The Windows unit suite failed the canary: spawning pnpm finds no program on Windows. The listing now runs Vitest's CLI with this Node executable. Co-authored-by: Isaac <no-reply@databricks.com>
…106) The witness module's edit sequence takes 10 to 11 minutes in CI, over P12's budget. It is one history and cannot be split without changing the law, so it runs in its own continue-on-error job, owned by #106, which is not a dependency of preview. Shard 1 of the partition is unchanged, so the partition check holds. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
…, not for merge) Co-authored-by: Isaac <no-reply@databricks.com>
Scratch measurement for PR #90's random shard sizing (P12). Closed after the measurement. Not for merge.