Skip to content

Self-oracle counterexample: timeout presentation not exercised (seed 569029462) #51

Open

Description

@KyleAMathews

A random-seed run of the framework self-oracle on main found a counterexample that fails law timeout. This happened after #31 (e2af13a).

  • Run: PR ci(oracles): run the framework self-oracle as eight jobs #47's first CI run, "Framework self-oracle (8/8)". ci(oracles): run the framework self-oracle as eight jobs #47 changes only how CI splits the tests into jobs. A rerun with a new seed passed.
  • Unit: self-oracle.framework > random seed: generated
  • Violation: checkpoint "child", law timeout.
  • Difference: "timeout presentation was not exercised"
  • Replay:
    ORACLE_SEED=569029462 ORACLE_PATH=1:0:0:0:0:0:0:0:0:0:0:0:0:0:2:1:1:1:1:1:1:1:1:1:1:1:1:4 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' vitest run -t '^self-oracle\.framework(?: | > )random seed: generated$'
    

A generated history appears to declare a timeout presentation that its run never reaches. Either the grammar generates a history whose timeout cannot fire, or the timeout path does not present as the model owes. Fix it oracle-first, or record it in the coverage map with this trace.

This issue was written by Isaac.

Activity

  1. KyleAMathews commented on Nov 13, 2026

    @KyleAMathews
    ContributorAuthor

    Seen again on PR #46's CI (run 43354806939, job "Test on Vitest 5 (oracle-framework 8)"). It has the same law timeout at checkpoint "child", "timeout presentation was not exercised", with a new seed. #46 changes nothing in packages/oracles/src.

    Replay: ORACLE_SEED=-277524832 ORACLE_PATH=1:0:1:1:1:1:1:2:1:2:2:2:2:2:2:2:2:2:2:2:2:2:2:2:5 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' vitest run -t '^self-oracle\.framework(?: | > )random seed: generated$'

    This comment was written by Isaac.

  2. KyleAMathews commented on Nov 16, 2026

    @KyleAMathews
    ContributorAuthor

    Replay by history (register M, 2026-11-16, "Replay by history"; review of #122, F1). This records the full counterexample history, because #122 changes which history a seed and path select for this property's generated unit.

    • Replayed at: ff85fbd (the head of the run that found it), on an Arclet, with --maxWorkers=2. A scratch, test-only line printed each history the unit judged; the first one is the replayed counterexample.
    • Replay: cd packages/oracles && ORACLE_SEED=569029462 ORACLE_PATH=1:0:0:0:0:0:0:0:0:0:0:0:0:0:2:1:1:1:1:1:1:1:1:1:1:1:1:4 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' pnpm exec vitest --run --config vitest.oracle-framework.config.mjs -t '^self-oracle\.framework(?: | > )random seed: generated$'
    • Result: reproduces: law timeout at child, timeout presentation was not exercised. The printed counterexample equals the history below. (The first replay line of this issue, from ci(oracles): run the framework self-oracle as eight jobs #47's first CI run.)

    History:

    {"drafts": [{"probeEntered": {"fixed": "fixed-0"}, "probeNotEntered": {"fixed": "fixed-0"}, "calibrationHistory": null, "isolatedPlans": false, "identityHistories": false, "probeName": "saw-0", "probeBinding": "empty", "duplicateHistoryNames": false, "source": {"kind": "enumerated", "histories": [[0]]}, "statesMode": "exact", "constructed": [], "fixed": [], "constructedNames": "indexed", "fixedNames": "indexed", "secondSeed": false, "arbitraryMap": "unmapper", "mapAt": "history", "enumeratedStates": ["required", "required", "required"], "throwingBuild": false, "nondeterministic": false, "reachVia": "reached", "timing": "slow", "timeoutMode": "default", "calibrations": [{"kind": "right", "difference": "empty text"}], "extraReach": false, "calibrationNames": "indexed", "selectorPrototype": "plain", "warningProbe": false, "numberDifference": false, "unregistered": false, "probe": "agree", "overReport": false, "noisyProbe": false, "predictMode": "match", "ownsMode": "none", "clock": false, "form": "plain", "labelStyle": "plain", "enumeratedShape": "ok", "zeroNoise": false, "orderNoise": false, "emptyMessage": false, "odd": "cycle", "nested": false, "strictModel": false, "sevenModel": false, "errorNoise": false, "throwOnSpin": false, "oddThrow": false, "domThrow": true, "liveNoise": false, "liveDate": false, "realmCheck": false, "snapshotCheck": false, "deepCheck": false}], "seedless": false, "timeoutPresentation": true}

    The seed-and-path line selects this history only on commits before #122, and only while the property's constructed list is the one at ff85fbd; appending a constructed history before #122 also moved it. To replay it on a later commit, add the history above as a fixed history of the property, or as a constructed history whose build returns it, and run that test.

    This comment was written by Isaac.

  3. KyleAMathews commented on Nov 16, 2026

    @KyleAMathews
    ContributorAuthor

    Replay by history (register M, 2026-11-16, "Replay by history"; review of #122, F1). This records the full counterexample history, because #122 changes which history a seed and path select for this property's generated unit.

    • Replayed at: 606e525 (the head of the run that found it), on an Arclet, with --maxWorkers=2. A scratch, test-only line printed each history the unit judged; the first one is the replayed counterexample.
    • Replay: cd packages/oracles && ORACLE_SEED=-277524832 ORACLE_PATH=1:0:1:1:1:1:1:2:1:2:2:2:2:2:2:2:2:2:2:2:2:2:2:2:5 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' pnpm exec vitest --run --config vitest.oracle-framework.config.mjs -t '^self-oracle\.framework(?: | > )random seed: generated$'
    • Result: reproduces: law timeout at child, timeout presentation was not exercised. The printed counterexample equals the history below. (The second replay line of this issue, from feat(oracle-gaps): every streamed line starts with oracle-gaps: <kind> #46's run 43354806939.)

    History:

    {"drafts": [{"probeEntered": {"fixed": "fixed-0"}, "probeNotEntered": {"constructed": "constructed-0"}, "calibrationHistory": {"enumerated": "items 0 | 1"}, "isolatedPlans": false, "identityHistories": false, "probeName": "saw-0", "probeBinding": "unbound", "duplicateHistoryNames": false, "source": {"kind": "enumerated", "histories": [[0]]}, "statesMode": "exact", "constructed": [], "fixed": [], "constructedNames": "indexed", "fixedNames": "indexed", "secondSeed": false, "arbitraryMap": "none", "mapAt": "history", "enumeratedStates": null, "throwingBuild": false, "nondeterministic": false, "reachVia": "reached", "timing": "slow", "timeoutMode": "default", "calibrations": [{"kind": "right", "difference": "text match"}], "extraReach": false, "calibrationNames": "indexed", "selectorPrototype": "plain", "warningProbe": true, "numberDifference": false, "unregistered": false, "probe": "none", "overReport": false, "noisyProbe": false, "predictMode": "match", "ownsMode": "list", "clock": false, "form": "plain", "labelStyle": "plain", "enumeratedShape": "ok", "zeroNoise": false, "orderNoise": true, "emptyMessage": false, "odd": "cycle", "nested": true, "strictModel": true, "sevenModel": false, "errorNoise": false, "throwOnSpin": false, "oddThrow": false, "domThrow": true, "liveNoise": false, "liveDate": true, "realmCheck": false, "snapshotCheck": false, "deepCheck": false}], "seedless": false, "timeoutPresentation": true}

    The seed-and-path line selects this history only on commits before #122, and only while the property's constructed list is the one at 606e525; appending a constructed history before #122 also moved it. To replay it on a later commit, add the history above as a fixed history of the property, or as a constructed history whose build returns it, and run that test.

    This comment was written by Isaac.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions