Skip to content

measure: random shard cost per seed (scratch, do not merge) - #110

Closed
KyleAMathews wants to merge 17 commits into
mainfrom
ci/random-seed-measure
Closed

KyleAMathews wants to merge 17 commits into
mainfrom
ci/random-seed-measure

Conversation

@KyleAMathews

Copy link
Copy Markdown
Contributor

Scratch measurement for PR #90's random shard sizing (P12). Closed after the measurement. Not for merge.

KyleAMathews and others added 16 commits November 14, 2026 21:46
…ry PR

CI-1 (random seeds run on every PR) and P12 (every job under 10 minutes).
pr-shards.mjs gets two partitions: oracle-gaps-random, in two shards, and
oracle-gaps-determinism, in four. Each shard is a longest-first packing of
one Arclet run of main at 35422e3: the random units take 624 s and the
determinism units 1,337 s, and the largest shard is about 340 s of unit
time. pr.yml runs them as blocking Linux jobs, and the preview waits for
them. The e2e-shards comment that said CI runs only the fixed-seed job is
updated.

Co-authored-by: Isaac <no-reply@databricks.com>
…h with an issue

The determinism job gates every unit except six that fail on main
(#91 heartbeat list, #92 daemon note): every outcome class, brokenOwner,
noTypescript, the witness module's edit sequence, generated 2, and the
daemon's edit sequence. Each excluded unit has its own issue (#93 to #98).
pr-shards.mjs takes an excluded list: the rest shard skips it, the partition
check fails if a shard holds an excluded name or an excluded name is not a
test, and an excluded test is in no shard on purpose.

CI ran the random shards about 1.7 times slower than the Arclet, so random
is three shards and determinism is two.

Co-authored-by: Isaac <no-reply@databricks.com>
Maintainer decision, 2026-11-14: heartbeats are progress, not outcome. A
heartbeat's set and count depend on wall-clock timing, so the e2e oracle's
determinism check compared a timing-dependent trace and failed five units on
main (#91).

The trace now keeps no heartbeat. The driver observes the breaches of the
shape law instead: a beat names the baseline or a mutant of its run, in a
file that runs it, and a beat that follows its mutant's own streamed gap line
names a mutant that was not running. The check reports a breach as a
streaming failure. The presence obligation (a slow owner must be named in a
heartbeat) is retired with the set, and so is the model's owed-heartbeat
flag. Calibrations feed wrong beat observations to the same check.

Co-authored-by: Isaac <no-reply@databricks.com>
…ery run's beats

The stability rerun and the partial run observed their CLI runs with their
heartbeat lists still in them, so the determinism check compared timing in
two more observations. Every observed run now drops its beats, and each run
that is observed also observes its shape breaches. The check judges every
beats observation.

Co-authored-by: Isaac <no-reply@databricks.com>
The reach state "a heartbeat named a busy mutant" is derived from a beat's
timing, so determinism compared a timing-dependent observation. Under the
maintainer's 2026-11-14 decision that heartbeats are progress, the state is no
longer observed or required. #84 owns restoring its coverage through a
deterministic pause point. Recorded in the Limits.

Co-authored-by: Isaac <no-reply@databricks.com>
The reach state "a timed-out run reused after an edit" compares the cut counts
of a daemon check and a cold run. Those counts are timing, so the state
depended on wall-clock timeouts, and the determinism check failed on the
witness module's edit sequence. It is no longer observed or required, and
the witness history no longer enters it. Recorded in the Limits.

Co-authored-by: Isaac <no-reply@databricks.com>
The removed reach state was the witness history's only enters entry, so the
file failed to load. Co-authored-by: Isaac <no-reply@databricks.com>
#92)

With the heartbeat fix, the five units that failed on main pass, so the
determinism partition takes them back: the witness module's edit sequence
in its own shard, and the other 49 units in three packed shards and a rest
shard, five jobs in all. Only #92 stays excluded. The partition check now
fails on a shard name that matches no test, which the explicit names had
never been checked for.

Co-authored-by: Isaac <no-reply@databricks.com>
The e2e gap oracle's constructed histories are built from seed + index, so a
history inserted mid-list changes every later one (generator rule, register
2026-11-14: new constructed histories are appended at the end). The canary
records the registered names in oracle-gaps/constructed-order.json and fails
unless the current list begins with them.

Co-authored-by: Isaac <no-reply@databricks.com>
The Test job's e2e check failed: the retired calibration is still named, and
the two new heartbeat calibrations are in no shard.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The determinism check compared stderrGaps and stdoutGaps in the order the CLI
printed them. A context's gap line is printed when its copies finish, so the
order is timing. The stream's own comment already says the trace records the
stream as relations, so these two lists are sorted like the streamed list,
which is sorted. Nine determinism units failed on main (4d3c8a9) and on the
merged head for this reason.

Also the constructed-order canary lists vitest with --json=<file>, so no
package manager's stdout reaches the JSON (its Vitest 5 job failed on that).

Co-authored-by: Isaac <no-reply@databricks.com>
… the canary's listing to Vitest 4

The determinism unit added by #73, a daemon pause that cannot engage, gives
different traces across runs (#105), so it is excluded until fixed, as the
other main failures are. The constructed-order canary lists the gap oracle
only under Vitest 4: the Vitest 5 job runs no gap oracle, and its listing finds
no suite in that file.

Co-authored-by: Isaac <no-reply@databricks.com>
The Windows unit suite failed the canary: spawning pnpm finds no program on
Windows. The listing now runs Vitest's CLI with this Node executable.

Co-authored-by: Isaac <no-reply@databricks.com>
…106)

The witness module's edit sequence takes 10 to 11 minutes in CI, over P12's
budget. It is one history and cannot be split without changing the law, so it
runs in its own continue-on-error job, owned by #106, which is not a dependency
of preview. Shard 1 of the partition is unchanged, so the partition check holds.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: f39a98d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

…, not for merge)

Co-authored-by: Isaac <no-reply@databricks.com>
@KyleAMathews
KyleAMathews deleted the ci/random-seed-measure branch November 16, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant