Skip to content

gap CLI: a run under load can end non-zero (an internal error) where it should complete #83

Open

Description

@KyleAMathews

Summary

Under load, the gap CLI sometimes ends a run with a non-zero exit where the run should complete. Twice it was an internal error ("This is a bug in oracle-gaps"). The e2e oracle's fixed-seed shard 3 failed in 2 of 8 Linux runs on #55's head. Each failure had a different symptom, and neither one reproduces when the unit runs alone.

Traces

  1. CI, job 132489470739 (run 43608937018), commit 16f510c, "Gap report end-to-end oracle (fixed seed, 3/3)":
    • Unit: fixed seed 20261105: generated, history ORACLE_PATH=1 (fault brokenOwner).
    • Error: a history entered "refused brokenOwner", which the property does not declare.
    • Meaning: the CLI exited non-zero on a brokenOwner history. The history owes exit 0 with blocked mutants (blocked by brokenOwner). The log does not show which non-zero exit or its error.log.
    • Replay: ORACLE_SEED=20261105 ORACLE_PATH=1 ORACLE_PROPERTY='oracle-gaps.report' ORACLE_UNIT='random seed: generated' vitest run -t '^oracle-gaps\.report(?: | > )random seed: generated$'
  2. Classic Arca VM, tag r55-gen-shard3, commit edb24ba (16f510c plus a test-only diagnostic), the same shard-3 command:
    • Unit: fixed seed 20261105: an edit sequence through the daemon, step dieMidCheck.
    • Error: through the daemon at dieMidCheck: the run wrote a line on stderr that is not a line of the stream: This is a bug in oracle-gaps, not in your oracle. Its stack is in <package>/out-daemon/error.log.
    • The VM was shared with another session's long loops when this ran.

The shard command, from packages/oracles:

pnpm exec vitest --run --maxWorkers=2 --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "$(node scripts/pr-shards.mjs pattern oracle-gaps-fixed-seed 3)"

Tally

Commit Where Shard-3 runs Failed
16f510c (#55 head) CI, Classic Arca, Arclets 8 2
35422e3 (main) Classic Arca, Arclets 6 0
ed091f3 (#55 before the #58 merge) Arclets 4 0

The unit generated alone passed on 16f510c locally (macOS) and on the VM, and on main locally. All 12 runs on the dedicated Arclets passed, with no internal error.

Next

This issue was written by Isaac.

Activity

  1. KyleAMathews commented on Nov 14, 2026

    @KyleAMathews
    ContributorAuthor

    Cause found

    Under load, a shard-3 run on #55's head (Arclet tag r55-l-head-3, with 8 yes loops alongside) ended in this internal error:

    SyntaxError: Unterminated string in JSON at position 523 (line 1 column 524)
        at JSON.parse (<anonymous>)
        at finished (oracle-gaps/check.mjs:1139:61)
        at Timeout.poll [as _onTimeout] (oracle-gaps/check.mjs:1156:20)
    

    runBatch's watchdog polls the capture file every 50 ms (finished). It parsed every line that holds "closed":true, including a closing row that a worker was still appending. Under load the poll can read the row half written. The JSON error is then thrown in a timer, and the run ends as an internal error. The run also leaves its session directory, which is why the e2e oracle first reported the namespace law.

    The defect predates #55: main has the same finished. #55 adds readAt (each read file's hash) to the closing row, which lengthens it and widens the window. On the load runs, head failed 1 of 4 and main 0 of 4. Over all runs, head failed 3 of 12 and main 0 of 10.

    Fix (in #55)

    • The witness is deterministic, not timed. It uses a test-only hook, ORACLE_GAPS_TEST_TORN_ROWS. A batch copy writes its closing row in two appends. The second waits until the watchdog has polled twice, and the watchdog counts its polls after it reads the capture. So a poll reads the half-written row by construction. A constructed history, "a capture row read while the worker wrote it", runs with the hook.
    • The fix: finished parses the capture only up to its last newline. Every row ends with one.
    • RED: CI on 0b5c39f (job 132603967855).
    • GREEN: Arclet run r55-torn-green-2 on f7dac92.
    • Mutant: with the fix reverted, the history fails with "the run ended in an internal error: … Unterminated string in JSON".
    • Diagnostics: the e2e check now reports an internal error first, with its error.log, so a future crash shows its stack.

    This comment was written by Isaac.

  2. KyleAMathews commented on Nov 16, 2026

    @KyleAMathews
    ContributorAuthor

    Replay by history (register M, 2026-11-16, "Replay by history"; review of #122, F1). This records the full counterexample history, because #122 changes which history a seed and path select for this property's generated unit.

    • Replayed at: 16f510c (the head of the run that found it), on an Arclet, with --maxWorkers=2. A scratch, test-only line printed each history the unit judged; the first one is the replayed counterexample.
    • Replay: cd packages/oracles && ORACLE_SEED=20261105 ORACLE_PATH=1 ORACLE_PROPERTY='oracle-gaps.report' ORACLE_UNIT='random seed: generated' pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t '^oracle-gaps\.report(?: | > )random seed: generated$'
    • Result: passes when run alone, as this issue reports: the failure needs load. The history below is the one that path 1 selects at 16f510c, the brokenOwner history of the first CI failure.

    History:

    {"functions": [[{"kind": "if", "op": ">=", "k": 1, "not": true, "quote": "'"}, {"kind": "fieldCall", "target": 2}, {"kind": "mixed", "op": "===", "k": 8, "quote": "'"}, {"kind": "if", "op": "!==", "k": 4, "not": false, "quote": "`"}], [{"kind": "newCall", "target": 3}, {"kind": "valueCall", "target": 1}, {"kind": "flag", "value": false}, {"kind": "push", "quote": "`"}, {"kind": "mixedHelperCall"}]], "containers": [], "overloads": [], "owners": ["a", "b", "b"], "library": [[{"kind": "if", "op": ">=", "k": 3, "not": true, "quote": "`"}], [{"kind": "push", "quote": "`"}, {"kind": "dollar"}]], "libraryOwners": ["none"], "runes": [], "derived": null, "sourceSpelling": "plain", "dotGen": false, "ownsSpelling": "plain", "wholeFile": "none", "template": false, "tests": [{"file": "a", "route": "determinism", "registered": true, "name": "plain", "fn": 2, "input": 1, "judges": false, "law": "L", "channel": "observe", "flaky": false, "judgeBy": "verdict", "hangs": false, "slow": false, "shrinks": false, "draws": false, "ioWait": false, "stops": false, "rawImport": true, "cleanupThrows": true, "asProp": true, "then": {"route": "fixed seed", "input": 9}, "work": null, "timed": false}], "witness": true, "authored": null, "declaration": "wrapper", "ownerless": false, "fault": "brokenOwner", "stability": "none", "ordinary": false, "lazyGate": true, "secondAuthored": true, "consoleOutput": false, "config": "async", "configEnv": false, "setupGlobal": false, "native": "gyp", "depSubpath": false, "unbuiltDep": false, "boot": false, "budget": "two", "stopRuns": [], "onlyRun": false, "pragma": true, "cliPath": "spaced", "concurrentStart": false, "configFile": "vitest.config.ts", "configPlace": "discovered", "packageType": "module", "configAwait": false, "configRequire": true, "contexts": 4, "steps": null, "dataOwners": [], "exportOwners": ["none", "a", "b", "b", "b", "a", "a"], "importer": "b", "midrunBy": "b", "midrunTarget": "helper", "prePlugin": true, "fsRead": "promise", "laterSteps": ["midrun"]}

    The seed-and-path line selects this history only on commits before #122, and only while the property's constructed list is the one at 16f510c; appending a constructed history before #122 also moved it. To replay it on a later commit, add the history above as a fixed history of the property, or as a constructed history whose build returns it, and run that test.

    This comment was written by Isaac.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions