Repository navigation
Open
Description
Summary
Under load, the gap CLI sometimes ends a run with a non-zero exit where the run should complete. Twice it was an internal error ("This is a bug in oracle-gaps"). The e2e oracle's fixed-seed shard 3 failed in 2 of 8 Linux runs on #55's head. Each failure had a different symptom, and neither one reproduces when the unit runs alone.
Traces
- CI, job 132489470739 (run 43608937018), commit 16f510c, "Gap report end-to-end oracle (fixed seed, 3/3)":
- Unit:
fixed seed 20261105: generated, historyORACLE_PATH=1(faultbrokenOwner). - Error:
a history entered "refused brokenOwner", which the property does not declare. - Meaning: the CLI exited non-zero on a
brokenOwnerhistory. The history owes exit 0 with blocked mutants (blocked by brokenOwner). The log does not show which non-zero exit or its error.log. - Replay:
ORACLE_SEED=20261105 ORACLE_PATH=1 ORACLE_PROPERTY='oracle-gaps.report' ORACLE_UNIT='random seed: generated' vitest run -t '^oracle-gaps\.report(?: | > )random seed: generated$'
- Unit:
- Classic Arca VM, tag r55-gen-shard3, commit edb24ba (16f510c plus a test-only diagnostic), the same shard-3 command:
- Unit:
fixed seed 20261105: an edit sequence through the daemon, stepdieMidCheck. - Error:
through the daemon at dieMidCheck: the run wrote a line on stderr that is not a line of the stream: This is a bug in oracle-gaps, not in your oracle. Its stack is in <package>/out-daemon/error.log. - The VM was shared with another session's long loops when this ran.
- Unit:
The shard command, from packages/oracles:
pnpm exec vitest --run --maxWorkers=2 --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "$(node scripts/pr-shards.mjs pattern oracle-gaps-fixed-seed 3)"Tally
| Commit | Where | Shard-3 runs | Failed |
|---|---|---|---|
| 16f510c (#55 head) | CI, Classic Arca, Arclets | 8 | 2 |
35422e3 (main) |
Classic Arca, Arclets | 6 | 0 |
| ed091f3 (#55 before the #58 merge) | Arclets | 4 | 0 |
The unit generated alone passed on 16f510c locally (macOS) and on the VM, and on main locally. All 12 runs on the dedicated Arclets passed, with no internal error.
Next
- Reproduce under load on purpose: shard 3 with a CPU hog on the Arclets, 4 runs on fix(oracle-gaps): a file a test reads through fs is an input of its daemon run #55's head (with the diagnostics, which print each exit-3 run's error.log) and 4 on
main. The result decides whether this is fix(oracle-gaps): a file a test reads through fs is an input of its daemon run #55's or older. - fix(oracle-gaps): a file a test reads through fs is an input of its daemon run #55 (b959d4c) now makes the incremental violation print a cold run's error log when that run exits 3, so the next failure in CI shows its stack.
- Related: the deterministic-cut follow-up (cut runs by work count, not wall time; "determinism under load"), which targets timing-dependent outcomes.
This issue was written by Isaac.
Metadata
Metadata
Assignees
Labels
No labels
Activity
test(oracle-gaps): a batch's watchdog reads a closing row half written
test(oracle-gaps): torn rows are constructed-only; an internal error …
fix(oracle-gaps): a batch's watchdog parses only whole capture rows
KyleAMathews commentedon Nov 14, 2026
Cause found
Under load, a shard-3 run on #55's head (Arclet tag r55-l-head-3, with 8
yesloops alongside) ended in this internal error:runBatch's watchdog polls the capture file every 50 ms (finished). It parsed every line that holds"closed":true, including a closing row that a worker was still appending. Under load the poll can read the row half written. The JSON error is then thrown in a timer, and the run ends as an internal error. The run also leaves its session directory, which is why the e2e oracle first reported the namespace law.The defect predates #55: main has the same
finished. #55 addsreadAt(each read file's hash) to the closing row, which lengthens it and widens the window. On the load runs, head failed 1 of 4 and main 0 of 4. Over all runs, head failed 3 of 12 and main 0 of 10.Fix (in #55)
ORACLE_GAPS_TEST_TORN_ROWS. A batch copy writes its closing row in two appends. The second waits until the watchdog has polled twice, and the watchdog counts its polls after it reads the capture. So a poll reads the half-written row by construction. A constructed history, "a capture row read while the worker wrote it", runs with the hook.finishedparses the capture only up to its last newline. Every row ends with one.This comment was written by Isaac.
fix(oracle-gaps): both readers of a batch's capture take whole rows (#83
fix(oracle-gaps): a batch's capture is read strictly once the run has…
fix(oracle-gaps): a file a test reads through fs is an input of its d…
test(oracle-gaps): the rejected batch skips main's internal-error and…
fix(oracle-gaps): retire a batch's copies from Vitest's state (#61)
KyleAMathews commentedon Nov 16, 2026
Replay by history (register M, 2026-11-16, "Replay by history"; review of #122, F1). This records the full counterexample history, because #122 changes which history a seed and path select for this property's
generatedunit.--maxWorkers=2. A scratch, test-only line printed each history the unit judged; the first one is the replayed counterexample.cd packages/oracles && ORACLE_SEED=20261105 ORACLE_PATH=1 ORACLE_PROPERTY='oracle-gaps.report' ORACLE_UNIT='random seed: generated' pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t '^oracle-gaps\.report(?: | > )random seed: generated$'brokenOwnerhistory of the first CI failure.History:
{"functions": [[{"kind": "if", "op": ">=", "k": 1, "not": true, "quote": "'"}, {"kind": "fieldCall", "target": 2}, {"kind": "mixed", "op": "===", "k": 8, "quote": "'"}, {"kind": "if", "op": "!==", "k": 4, "not": false, "quote": "`"}], [{"kind": "newCall", "target": 3}, {"kind": "valueCall", "target": 1}, {"kind": "flag", "value": false}, {"kind": "push", "quote": "`"}, {"kind": "mixedHelperCall"}]], "containers": [], "overloads": [], "owners": ["a", "b", "b"], "library": [[{"kind": "if", "op": ">=", "k": 3, "not": true, "quote": "`"}], [{"kind": "push", "quote": "`"}, {"kind": "dollar"}]], "libraryOwners": ["none"], "runes": [], "derived": null, "sourceSpelling": "plain", "dotGen": false, "ownsSpelling": "plain", "wholeFile": "none", "template": false, "tests": [{"file": "a", "route": "determinism", "registered": true, "name": "plain", "fn": 2, "input": 1, "judges": false, "law": "L", "channel": "observe", "flaky": false, "judgeBy": "verdict", "hangs": false, "slow": false, "shrinks": false, "draws": false, "ioWait": false, "stops": false, "rawImport": true, "cleanupThrows": true, "asProp": true, "then": {"route": "fixed seed", "input": 9}, "work": null, "timed": false}], "witness": true, "authored": null, "declaration": "wrapper", "ownerless": false, "fault": "brokenOwner", "stability": "none", "ordinary": false, "lazyGate": true, "secondAuthored": true, "consoleOutput": false, "config": "async", "configEnv": false, "setupGlobal": false, "native": "gyp", "depSubpath": false, "unbuiltDep": false, "boot": false, "budget": "two", "stopRuns": [], "onlyRun": false, "pragma": true, "cliPath": "spaced", "concurrentStart": false, "configFile": "vitest.config.ts", "configPlace": "discovered", "packageType": "module", "configAwait": false, "configRequire": true, "contexts": 4, "steps": null, "dataOwners": [], "exportOwners": ["none", "a", "b", "b", "b", "a", "a"], "importer": "b", "midrunBy": "b", "midrunTarget": "helper", "prePlugin": true, "fsRead": "promise", "laterSteps": ["midrun"]}The seed-and-path line selects this history only on commits before #122, and only while the property's constructed list is the one at 16f510c; appending a constructed history before #122 also moved it. To replay it on a later commit, add the history above as a
fixedhistory of the property, or as a constructed history whosebuildreturns it, and run that test.This comment was written by Isaac.