Skip to content

determinism: an edit sequence through the daemon records a test-output note in one run and not the other #92

Open

Description

@KyleAMathews

Summary

The determinism unit determinism: constructed history: an edit sequence through the daemon fails on main (35422e3). Two runs of the same history differ in one daemon stream note: one run's stream includes oracle-gaps: note test output in <package>/<out>/test-output.log, and the other's does not. Found by the determinism units that CI-1 moves onto pull requests (PR #90).

Evidence

Structural diff of the two full traces (a scratch patch printed the untruncated texts), at checkpoint incremental:

  • $.daemon.stream.notes.1 exists in one run only. Both runs share --hang-limit 20000 ms is above --mutant-limit 15000 ms ... as note 0.

The cause is not determined. The note appears when a test's output is captured, so it probably depends on a timing race between the daemon's test output and the check's end. That has not been confirmed.

Replay

cd packages/oracles && pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "determinism: constructed history: an edit sequence through the daemon"

Notes

The 2,000-character prefix in the determinism message (#81) hid this difference. A diff needed the full texts. The check's law is that two runs of one history record the same trace, so either the note must be deterministic, or the comparison must leave it out with a stated reason in the coverage map.

Activity

  1. KyleAMathews commented on Nov 15, 2026

    @KyleAMathews
    ContributorAuthor

    Same symptom on PR #57's new history determinism: constructed history: a daemon check with the default output directory. Two runs of one history differ in one daemon stream note, oracle-gaps: note test output in <out>/test-output.log, present in one run and absent in the other (structural diff at checkpoint incremental, trace length 37,952 vs 38,009, the extra note is the difference).

    Mechanism confirmed in code: the note is printed only when test-output.log is non-empty at the end of the check (check.mjs), and its only writer is a synchronous appendFileSync of a test's own stdout/stderr writes (capture-setup.mjs). So its presence means a test wrote to the process's streams during that check. That part is established. What is not established is why a daemon check sometimes has such a write and sometimes not; the cause in #92 remains open.

    Per the register, this is main's defect appearing in a new history, so it does not block #57. Not excluded from the determinism unit in code; it stays a failing determinism unit until #92 is decided.

  2. KyleAMathews commented on Nov 16, 2026

    @KyleAMathews
    ContributorAuthor

    The edit-sequence instance, and how #126 resolves it. This applies only to determinism: constructed history: an edit sequence through the daemon.

    • Cause. The history starts with two checks at once (concurrentStart). The daemon runs one check at a time, in the order the requests arrive, so whichever check it runs first computes every unit and the other reuses them. Only the check that ran the tests captured console output, so the note test output line follows that check. The driver recorded the pair by the process it started first, so the trace showed which process won the race.
    • Evidence.
      • Natural RED: 8 of 10 runs at fac63df fail, and 4 of 10 at 9aed105. Each differs only in that note.
      • Forced-order RED: a scratch driver runs the two checks one after the other, in opposite orders on alternate runs. It fails 2 of 2.
    • Resolution in test(oracle-gaps): two checks at once are recorded by role, not by process order (#82) #126. The pair is recorded by role, with the check that computed more runs first and ties ordered by their records. The later check owes unchanged reuse. The pair is also held together in the daemon by a test-only hold released by a count (R57).

    Still open. The other instance in the comment above is PR #57's a daemon check with the default output directory. That history has concurrentStart: false and steps: ['test'], so it has no pair and no race, and #126's cause cannot explain it. This issue stays open for that instance. #126 says "refs #92", not "Fixes".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions