Skip to content

determinism: the CLI's heartbeat list depends on wall-clock timing, so five determinism units fail on main #91

Open

Description

@KyleAMathews

Summary

Five determinism units of the end-to-end gap oracle fail on main (35422e3). Each one runs a history twice and compares the two traces. The traces differ in the CLI's heartbeats list, which records which busy mutants beat. Whether a mutant beats depends on wall-clock timing, so the comparison is timing-dependent. Found by the determinism units that CI-1 moves onto pull requests (PR #90).

Failing units (main 35422e3, one Arclet, one Vitest worker)

  • determinism: constructed history: an edit sequence over the witness module
  • determinism: constructed history: every outcome class
  • determinism: constructed history: a declaration fault: brokenOwner
  • determinism: constructed history: a declaration fault: noTypescript
  • determinism: generated 2

Evidence

Each failure is at observation 0 (cli). A structural diff of the two full traces (captured with a scratch patch that prints the untruncated texts, since the message keeps 2,000 characters) differs only in $.heartbeats. For brokenOwner, one run's list includes sut.ts:M177 and the other's does not. The other units show the same kind of shift in the list.

Replay

Determinism units have no replay line. Run one unit with the project's command:

cd packages/oracles && pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "determinism: constructed history: a declaration fault: brokenOwner"

The history is printed in the failure message (History: {...}).

Why it matters

The register's "determinism under load" decision (2026-11-13) says that an oracle driver must not rely on wall-clock ordering. This comparison records a timing-dependent set, so it fails under load even when the CLI is deterministic. #84 is the reach side of the same timing (the heartbeat state's count per shard). This issue is the determinism side.

Open question for the maintainer

Either the determinism comparison should leave the heartbeat set out, since "each beat is observed as its mutant and file, not how long or how often it beat" (the law's own wording), or the beat set should be made deterministic through a test-only pause point as in #84's proposed fix. Not decided here.

Activity

  1. KyleAMathews commented on Nov 15, 2026

    @KyleAMathews
    ContributorAuthor

    Update from PR #90. The heartbeat list is no longer compared (progress, shape only, maintainer decision 2026-11-14). After merging main (#73), the five units also diverged on the order of stderrGaps and stdoutGaps, since gap lines are printed when a context finishes. That order is timing, so those two lists are now sorted, which matches the stream's own rule that the trace records relations. On 4bc4eb1, all determinism units pass on Arclet (54 of 54, including #92). The two reach states that were timing-derived ("a heartbeat named a busy mutant" and "a timed-out run reused after an edit") are no longer observed or required; that coverage is recorded in the Limits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions