Skip to content

random seed: generated (seed 4): a timeout classification counterexample, on main too #120

Open

Description

@KyleAMathews

Summary

The random unit random seed: generated, with ORACLE_SEED=4, finds a timeout-classification counterexample. Main fails it too, so under the M phase-1 rule it is filed, not blocking (P4).

  • Campaign (fix/model-work-bound at 39f776c, on an Arclet, 1 Vitest worker): property failed after 2 tests and shrank twice. Law classification at cli: src/sut.ts:f0:33:delete-call#0: timeout, owed indeterminate.
  • Replay of its printed path (ORACLE_PATH=1:2:1), twice each on main (eec7f77) and on fix/model-work-bound (6c813287): it fails every time under authored mutants, with the authored mutant a1 is unregistered-kill, owed timeout.

The replayed path fails under another law than the campaign's first failure. It is the same on main and on the branch, so the branch's model change (work counted in probe code) does not cause it. Both failures concern a timeout: the model owes a timeout where the CLI reports another outcome, or the reverse.

Replay

cd packages/oracles && ORACLE_SEED=4 ORACLE_PATH=1:2:1 ORACLE_PROPERTY=oracle-gaps.report ORACLE_UNIT='random seed: generated' pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "^oracle-gaps\.report(?: | > )random seed: generated$"

(The printed Replay line omits the config, #108.)

Not known

I did not establish the cause. A hang cut or the idle sampler under load are candidates; see the Windows idle gap in the Limits and #99.

Activity

  1. KyleAMathews commented on Nov 16, 2026

    @KyleAMathews
    ContributorAuthor

    Re-pin (review of #121, F2): the replay line in this issue stops working once #121 merges. #121 appends a constructed history, and the framework draws the campaign's constructed and generated halves from one fc.oneof whose weights depend on how many constructed histories there are. So a seed and path pick another history after the merge. Here are the full counterexample histories, so the failure can be replayed as a fixed or constructed history on any commit.

    1. The campaign's counterexample. Seed 4 at 39f776c (fix/model-work-bound before the merge; same generator as main eec7f77). Law classification: src/sut.ts:f0:33:delete-call#0: timeout, owed indeterminate.

    History (JSON)
    {"functions":[[{"kind":"until","quote":"'"},{"kind":"mixed","op":"<=","k":3,"quote":"'"}]],"containers":["free"],"overloads":[true,true,true],"owners":[],"library":[[{"kind":"if","op":">","k":9,"not":true,"quote":"'"},{"kind":"push","quote":"'"},{"kind":"dollar"}],[{"kind":"dollar"},{"kind":"if","op":"<","k":4,"not":false,"quote":"'"}]],"libraryOwners":["b"],"runes":[],"derived":{"params":"plain","field":false,"before":[{"kind":"if","op":"<","k":5},{"kind":"if","op":">=","k":0},{"kind":"early","op":"!==","k":3}],"form":"arrow","after":[{"kind":"if","op":">","k":2}]},"sourceSpelling":"linked","dotGen":false,"ownsSpelling":"plain","wholeFile":"b","template":true,"tests":[{"file":"a","route":"probe calibration","registered":true,"name":"route","fn":7,"input":4,"judges":true,"law":"L","channel":"observe","flaky":false,"judgeBy":"soft","hangs":true,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":false,"cleanupThrows":false,"asProp":false,"then":{"route":"fixed history","input":3},"work":null,"timed":false},{"file":"c","route":"calibration","registered":true,"name":"route","fn":4,"input":2,"judges":true,"law":"reach agreement","channel":"observe","flaky":false,"judgeBy":"verdict","hangs":false,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":false,"cleanupThrows":false,"asProp":false,"then":{"route":"calibration","input":2},"work":null,"timed":false},{"file":"a","route":"probe calibration","registered":true,"name":"route","fn":7,"input":3,"judges":true,"law":"reach agreement","channel":"expect","flaky":false,"judgeBy":"soft","hangs":false,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":true,"cleanupThrows":false,"asProp":false,"then":null,"work":null,"timed":false},{"file":"b","route":"fixed seed","registered":true,"name":"route","fn":5,"input":5,"judges":false,"law":"reach agreement","channel":"expect","flaky":true,"judgeBy":"soft","hangs":false,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":false,"cleanupThrows":true,"asProp":false,"then":{"route":"calibration","input":8},"work":null,"timed":false},{"file":"a","route":"determinism","registered":true,"name":"route","fn":6,"input":3,"judges":false,"law":"reach agreement","channel":"observe","flaky":false,"judgeBy":"verdict","hangs":false,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":false,"cleanupThrows":false,"asProp":false,"then":{"route":"fixed history","input":0},"work":null,"timed":false}],"witness":false,"authored":{"stmt":5,"kind":"ambiguous"},"declaration":"both","ownerless":true,"fault":"apiLessTypescript","stability":"insertGuard","ordinary":true,"lazyGate":false,"secondAuthored":true,"consoleOutput":true,"config":"promise","configEnv":true,"setupGlobal":false,"native":"none","depSubpath":true,"unbuiltDep":false,"boot":false,"budget":"two","stopRuns":[],"onlyRun":false,"pragma":false,"cliPath":"plain","concurrentStart":true,"configFile":"vitest.config.ts","configPlace":"none","packageType":"commonjs","configAwait":true,"configRequire":false,"contexts":4,"steps":null,"dataOwners":[],"exportOwners":["b","b"],"importer":"none","midrunBy":"b","midrunTarget":"data","prePlugin":false,"fsRead":"statDir","laterSteps":[],"tornRows":false,"pauseEngages":true,"warmReplaced":false}

    2. The history that ORACLE_PATH=1:2:1 replays. Main eec7f77 plus scratch histories (a4b5b9c9), run twice. Law authored mutants: the authored mutant a1 is unregistered-kill, owed timeout. On main 9aed105, the #121 review saw this path fail as in (1).

    History (JSON)
    {"functions":[[{"kind":"until","quote":"'"},{"kind":"mixed","op":"<=","k":3,"quote":"'"}]],"containers":["free"],"overloads":[true,true,true],"owners":["a"],"library":[[{"kind":"if","op":">","k":9,"not":true,"quote":"'"},{"kind":"push","quote":"'"},{"kind":"dollar"}],[{"kind":"dollar"},{"kind":"if","op":"<","k":4,"not":false,"quote":"'"}]],"libraryOwners":["b"],"runes":[],"derived":{"params":"plain","field":false,"before":[{"kind":"if","op":"<","k":5},{"kind":"if","op":">=","k":0},{"kind":"early","op":"!==","k":3}],"form":"arrow","after":[{"kind":"if","op":">","k":2}]},"sourceSpelling":"dot","dotGen":false,"ownsSpelling":"plain","wholeFile":"none","template":true,"tests":[{"file":"a","route":"probe calibration","registered":true,"name":"route","fn":7,"input":4,"judges":true,"law":"L","channel":"observe","flaky":false,"judgeBy":"soft","hangs":true,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":false,"cleanupThrows":false,"asProp":false,"then":{"route":"fixed history","input":3},"work":null,"timed":false},{"file":"c","route":"calibration","registered":true,"name":"route","fn":4,"input":2,"judges":true,"law":"reach agreement","channel":"observe","flaky":false,"judgeBy":"verdict","hangs":false,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":false,"cleanupThrows":false,"asProp":false,"then":{"route":"calibration","input":2},"work":null,"timed":false},{"file":"a","route":"probe calibration","registered":true,"name":"route","fn":7,"input":3,"judges":true,"law":"reach agreement","channel":"expect","flaky":false,"judgeBy":"soft","hangs":false,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":true,"cleanupThrows":false,"asProp":false,"then":null,"work":null,"timed":false},{"file":"b","route":"fixed seed","registered":true,"name":"route","fn":5,"input":5,"judges":false,"law":"reach agreement","channel":"expect","flaky":true,"judgeBy":"soft","hangs":false,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":false,"cleanupThrows":true,"asProp":false,"then":{"route":"calibration","input":8},"work":null,"timed":false},{"file":"a","route":"determinism","registered":true,"name":"route","fn":6,"input":3,"judges":false,"law":"reach agreement","channel":"observe","flaky":false,"judgeBy":"verdict","hangs":false,"slow":false,"shrinks":false,"draws":false,"ioWait":false,"stops":false,"rawImport":false,"cleanupThrows":false,"asProp":false,"then":{"route":"fixed history","input":0},"work":null,"timed":false}],"witness":"capture","authored":{"stmt":0,"kind":"apply"},"declaration":"wrapper","ownerless":false,"fault":"none","stability":"insertGuard","ordinary":true,"lazyGate":false,"secondAuthored":true,"consoleOutput":true,"config":"async","configEnv":true,"setupGlobal":true,"native":"gyp","depSubpath":true,"unbuiltDep":true,"boot":false,"budget":"two","stopRuns":[{"signal":"SIGTERM","pool":"forks","when":"run"},{"signal":"SIGINT","pool":"threads","when":"run"},{"signal":"SIGINT","pool":"threads","when":"after"}],"onlyRun":false,"pragma":true,"cliPath":"spaced","concurrentStart":true,"configFile":"vitest.config.ts","configPlace":"discovered","packageType":"module","configAwait":true,"configRequire":false,"contexts":4,"steps":null,"dataOwners":[],"exportOwners":["b","b"],"importer":"none","midrunBy":"b","midrunTarget":"data","prePlugin":false,"fsRead":"statDir","laterSteps":[],"tornRows":false,"pauseEngages":true,"warmReplaced":false}

    The seed and path replay holds on main up to 9aed105 only:

    cd packages/oracles && ORACLE_SEED=4 ORACLE_PATH=1:2:1 ORACLE_PROPERTY=oracle-gaps.report ORACLE_UNIT='random seed: generated' pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "^oracle-gaps\.report(?: | > )random seed: generated$"
    

    On a later commit, append one of the histories above as a constructed history (build: (generated) => ({ ...<json>, stopRuns: generated.stopRuns }); stopRuns: [] must be the drawn constant's own reference) and run its fixed seed 20261105: <name> unit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions