Skip to content

Self-oracle counterexample: calibration selector finds no history when history names repeat (seed 1187311827) #50

Open

Description

@KyleAMathews

A random-seed run of the framework self-oracle on main found a counterexample in the calibration-history selector from #31 (e2af13a). The selector found no history for an enumerated source when the property declares duplicate history names.

  • Run: PR feat(oracle-gaps): every streamed line starts with oracle-gaps: <kind> #46, run 43288589486, job 131428487356, "Framework self-oracle (3/3)". feat(oracle-gaps): every streamed line starts with oracle-gaps: <kind> #46 changes no file under packages/oracles/src or tests/.
  • Unit: self-oracle.framework > random seed: generated
  • Violation: checkpoint "child", law registration.
  • Difference: unexpected child load/suite errors: {"id":"self-oracle.synthetic-0","errors":["self-oracle.synthetic-0: calibration history selector for \"cal-0\": enumerated \"items 0 | 1\" must identify exactly one history (found 0)."]}
  • History shape: duplicateHistoryNames: true, an enumerated source [[0]], and an unbound probe.
  • Replay:
    ORACLE_SEED=1187311827 ORACLE_PATH=1:0:0:0:1:1:1:1:2:5:2:2:2:2:2:2:2:2:2:2:2:2 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' vitest run -t '^self-oracle\.framework(?: | > )random seed: generated$'
    

Either the selector mishandles duplicate history names, or the grammar generates a property that #31's model does not cover. Fix it oracle-first: shrink it, decide whether the defect is in production or in the model, show RED, then fix. Otherwise, record it in the coverage map with this trace.

This issue was written by Isaac.

Activity

  1. KyleAMathews commented on Nov 16, 2026

    @KyleAMathews
    ContributorAuthor

    Replay by history (register M, 2026-11-16, "Replay by history"; review of #122, F1). This records the full counterexample history, because #122 changes which history a seed and path select for this property's generated unit.

    • Replayed at: 8260bda (the head of the run that found it), on an Arclet, with --maxWorkers=2. A scratch, test-only line printed each history the unit judged; the first one is the replayed counterexample.
    • Replay: cd packages/oracles && ORACLE_SEED=1187311827 ORACLE_PATH=1:0:0:0:1:1:1:1:2:5:2:2:2:2:2:2:2:2:2:2:2:2 ORACLE_PROPERTY='self-oracle.framework' ORACLE_UNIT='random seed: generated' pnpm exec vitest --run --config vitest.oracle-framework.config.mjs -t '^self-oracle\.framework(?: | > )random seed: generated$'
    • Result: reproduces: law registration at child, calibration history selector for "cal-0": enumerated "items 0 | 1" must identify exactly one history (found 0). The printed counterexample equals the history below.

    History:

    {"drafts": [{"probeEntered": {"fixed": "fixed-0"}, "probeNotEntered": {"fixed": "fixed-0"}, "calibrationHistory": null, "isolatedPlans": false, "identityHistories": false, "probeName": "saw-0", "probeBinding": "unbound", "duplicateHistoryNames": true, "source": {"kind": "enumerated", "histories": [[0]]}, "statesMode": "exact", "constructed": [], "fixed": [], "constructedNames": "indexed", "fixedNames": "indexed", "secondSeed": false, "arbitraryMap": "unmapper", "mapAt": "history", "enumeratedStates": null, "throwingBuild": false, "nondeterministic": true, "reachVia": "reached", "timing": "normal", "timeoutMode": "default", "calibrations": [{"kind": "right", "history": {"enumerated": "items 0 | 1"}, "difference": "none"}], "extraReach": false, "calibrationNames": "indexed", "selectorPrototype": "plain", "warningProbe": false, "numberDifference": false, "unregistered": false, "probe": "none", "overReport": false, "noisyProbe": false, "predictMode": "match", "ownsMode": "list", "clock": false, "form": "plain", "labelStyle": "reserved-reach", "enumeratedShape": "ok", "zeroNoise": false, "orderNoise": false, "emptyMessage": false, "odd": "none", "nested": true, "strictModel": true, "sevenModel": false, "errorNoise": false, "throwOnSpin": false, "oddThrow": false, "domThrow": false, "liveNoise": true, "liveDate": false, "realmCheck": false, "snapshotCheck": false, "deepCheck": true}], "seedless": false}

    The seed-and-path line selects this history only on commits before #122, and only while the property's constructed list is the one at 8260bda; appending a constructed history before #122 also moved it. To replay it on a later commit, add the history above as a fixed history of the property, or as a constructed history whose build returns it, and run that test.

    This comment was written by Isaac.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions