Skip to content

law 9: under load, the daemon's first check of the witness edit sequence blocked a mutant on a failing baseline that the cold run passed (ff5c021, once) #124

Open

Description

@KyleAMathews

Summary

One local run of the fixed-seed shard 1 (oracle-gaps-fixed-seed 1), on PR #61's branch at ff5c021, under a load average of about 10, failed law 9 ("incremental equals cold") in fixed seed 20261105: an edit sequence over the witness module. At the first step (start), the daemon's check reported sut.ts:M1 as blocked by tests/a.test.ts. The cold run of the same state reported it as survived (not-distinguished), owned by tests/a.test.ts. So in the daemon's first check, file a's baseline failed with no mutant active, and in the cold run it passed. The run did not reproduce.

Classification

This is an observation of production through a complete artifact (P4, "What P4 counts"): the daemon check's results.json, read after the CLI exited. It is not a driver timing assumption. At start this history runs one CLI (concurrentStart: false) through daemonChecks, which for start is a plain runCli with no test-only hook, no pause, and no hold. The driver puts the CLI in no state by wall-clock ordering.

The cause is not determined. The trace does not show the baseline line or the failing test. One candidate is a wall-clock cut that decides a baseline outcome under load. --test-timeout (15 s by default, args.mjs) becomes Vitest's testTimeout, hookTimeout and teardownTimeout for the session (check.mjs). A daemon's first check also opens its contexts. If that is the cause, it falls under "Determinism under load" (no wall-clock cut decides an outcome), next to the deterministic-cuts work (#99). This is not confirmed.

Trace

Log: /tmp/fix61-logs/g1-ff5.log on the reporting machine (run: g1-ff5 at ff5c021 ... shard oracle-gaps-fixed-seed 1, duration 536 s).

FAIL  gap-report-oracle.property.test.mjs > oracle-gaps.report > fixed seed 20261105: an edit sequence over the witness module
{ seed: 20261105, path: "0", endOnFailure: true }
Caused by: OracleViolation: oracle-gaps.report violated at checkpoint "incremental" (law: incremental equals cold).
Difference: after start, the daemon's report differs from the cold run's at src/sut.ts#devBuild:return-undefined:return process . env . NODE_ENV !== 'production'

- Expected (cold)
+ Received (daemon)
-   "changedTests": [],
-   "class": "not-distinguished",
+   "blockedBy": [ "<package>/tests/a.test.ts" ],
-   "oracleFiles": [ "<package>/tests/a.test.ts" ],
+   "oracleFiles": [],
-   "outcome": "survived",
+   "outcome": "blocked",

History (the counterexample, unshrunk): witness capture, budget: "two", contexts: 4, consoleOutput: true, concurrentStart: false, steps: ["test","helper","revert"], owners a and b, two generated functions (if > 4, push). The full record is in the log.

Replay

cd packages/oracles && pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "^oracle-gaps\.report(?: | > )fixed seed 20261105: an edit sequence over the witness module$"

It passed on the reruns. Load seems to be a factor.

Test gap

The law 9 violation message gives the first differing result, but not the daemon check's baseline lines or the failing test's error. A failing baseline that blocks a mutant only through the daemon cannot be diagnosed from the trace. Add the daemon check's baseline verdict lines (and, on a mismatch, the failing test's first error line) to the message.

Found during #82's triage.

Activity

  1. KyleAMathews commented on Nov 16, 2026

    @KyleAMathews
    ContributorAuthor

    The full counterexample history (M "Replay by history"), copied from the History: line of /tmp/fix61-logs/g1-ff5.log (seed 20261105, path "0", unshrunk):

    {"functions":[[{"kind":"if","op":">","k":4,"not":false,"quote":"'"}],[{"kind":"push","quote":"'"}]],"containers":[],"overloads":[],"owners":["a","b"],"library":[],"libraryOwners":[],"runes":[],"derived":null,"sourceSpelling":"plain","dotGen":false,"ownsSpelling":"dot","wholeFile":"none","template":false,"tests":[{"file":"a","route":"fixed seed","registered":true,"name":"route","input":5,"judges":true,"law":"L","channel":"observe","flaky":false,"judgeBy":"verdict","hangs":false,"slow":false,"shrinks":false,"ioWait":false,"rawImport":false,"cleanupThrows":false,"asProp":false,"stops":false,"then":null,"draws":false,"work":null,"timed":false,"fn":0},{"file":"b","route":"fixed seed","registered":true,"name":"route","input":1,"judges":true,"law":"L","channel":"observe","flaky":false,"judgeBy":"verdict","hangs":false,"slow":false,"shrinks":false,"ioWait":false,"rawImport":false,"cleanupThrows":false,"asProp":false,"stops":false,"then":null,"draws":false,"work":null,"timed":false,"fn":1}],"witness":"capture","authored":null,"declaration":"oracleProperty","ownerless":false,"fault":"none","stability":"none","ordinary":false,"lazyGate":false,"secondAuthored":false,"consoleOutput":true,"config":"object","configEnv":false,"setupGlobal":false,"native":"none","depSubpath":false,"unbuiltDep":false,"boot":false,"budget":"two","stopRuns":[],"onlyRun":false,"pragma":false,"cliPath":"plain","concurrentStart":false,"configFile":"vite.config.ts","configPlace":"discovered","packageType":"module","configAwait":false,"configRequire":false,"contexts":4,"steps":["test","helper","revert"],"dataOwners":[],"exportOwners":[],"importer":"none","midrunBy":"a","midrunTarget":"data","prePlugin":false,"fsRead":"promise","laterSteps":[],"tornRows":false,"pauseEngages":true,"warmReplaced":false,"rejectBatch":0}

    Context for replaying it: it is the fixed seed 20261105 unit of an edit sequence over the witness module, on PR #61's branch at ff5c021 (#61 merged as fac63df).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions