Skip to content
5 changes: 5 additions & 0 deletions .changeset/gap-cli-evidence-ran.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@tanstack/oracles': patch
---

The `oracle-gaps` report's `evidence` in `results.json` has a new count, `ran`: the units a check computed that ended without a cut. `cut` counts cut events (a cut copy that is confirmed alone is cut in its batch and again if its confirmation is cut), so `computed - cut` is not a count of runs.
2 changes: 2 additions & 0 deletions packages/oracles/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,6 @@ Under the hook, each check also writes `check.pid` (the id of the process that r
- For a settled or rejected batch: `after` (the sizes after its copies are retired), `copyNodes` (Vite module nodes that still name a copy, must be 0), and `nonCopyDup` (non-copy files with more than one module node in an environment, must be 0 beyond the measured plain-plus-`?raw` pair).
- For a cut batch: `cut: true` and no `after`. Its context is replaced, not retired.

`ORACLE_GAPS_TEST_HOLD_OVERLAP` (a file path, unset in normal use) makes a daemon hold the first check it runs, before the check starts, until the daemon has accepted a second check request. Then it writes the number of check requests it has accepted to the path, and the check runs. It holds once per daemon, and it is released by that count, never by time. The two checks then overlap in the daemon, as two checks started at once do when the second request arrives in time. The hook changes no outcome: the second request waits in the queue, as it always does while a check runs. The e2e oracle sets it for its two checks at once in the edit sequence through the daemon, so that the overlap is held, not left to the timing of two processes (issue #82).

`ORACLE_GAPS_TEST_REJECT_BATCH` (a batch number from 1, unset in normal use) makes the gap CLI's n-th batch run reject, as a crash in Vitest would. It is a test-only fault: the batch's run completes, so its copies are in Vitest's and Vite's state, and then the run's promise rejects, which takes the same path a real crash takes. The run then ends as an internal error: exit 3, the done error line, and `error.log` in its output directory. The batch's copies are still retired and its retention line is still written.
14 changes: 10 additions & 4 deletions packages/oracles/oracle-gaps/check.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -1619,13 +1619,16 @@ export default async (env) => {
options.mutantLimit,
]),
)
// How the evidence was produced: units reused and computed, runs cut off
// at their time limit or for idleness (of them, `idle`), items rerun
// after a cut-off batch, and mutants that changed no value and so needed
// no run.
// How the evidence was produced: units reused and computed, of the
// computed ones those that ended without a cut (`ran`), runs cut off at
// their time limit or for idleness (of them, `idle`), items rerun after
// a cut-off batch, and mutants that changed no value and so needed no
// run. `cut` counts cut events: a cut copy confirmed alone is cut once
// in its batch and again if its confirmation is cut too.
const evidence = {
reused: 0,
computed: 0,
ran: 0,
cut: 0,
idle: 0,
spin: 0,
Expand Down Expand Up @@ -1752,6 +1755,7 @@ export default async (env) => {
})
evidence.computed += 1
if (result.timedOut) evidence.cut += 1
else evidence.ran += 1
// A run cut off keeps its file's known inputs and any it named itself
// (third review of #55, F-D).
const paths = result.timedOut
Expand Down Expand Up @@ -1818,6 +1822,7 @@ export default async (env) => {
if (u.spin) evidence.spin += 1
remember(rest[k].id, u, batch.inputs[k])
evidence.computed += 1
evidence.ran += 1
})
if (batch.timedOutAt === undefined) break
// A copy past its first budget may only be slow: a loaded machine
Expand Down Expand Up @@ -1907,6 +1912,7 @@ export default async (env) => {
const u = confirmed(alone.units[0])
remember(hung.id, u, inputs)
evidence.computed += 1
evidence.ran += 1
return u
}

Expand Down
20 changes: 20 additions & 0 deletions packages/oracles/oracle-gaps/daemon.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,19 @@ async function serve(pkg) {
let parseArgs
let idle
let watch
// Test-only (ORACLE_GAPS_TEST_HOLD_OVERLAP, a path): the first check this
// daemon runs waits, before it starts, until the daemon has accepted a
// second check request, so two checks started at once overlap by a count,
// never by time. The release writes that count to the path. Once: a check
// after the first never waits. The hold changes no outcome; the second
// request waits in the queue, as it does whenever two checks overlap.
const overlapMark = process.env.ORACLE_GAPS_TEST_HOLD_OVERLAP
let accepted = 0
let overlapHeld = false
let overlapped
const secondAccepted = new Promise((resolvePromise) => {
overlapped = resolvePromise
})
// Requests wait until the session is open.
let opened
let queue = new Promise((resolvePromise) => {
Expand Down Expand Up @@ -382,6 +395,8 @@ async function serve(pkg) {
queue = queue.then(() => shutdown(connection))
return
}
accepted += 1
if (accepted === 2) overlapped()
queue = queue.then(async () => {
clearTimeout(idle)
// A request queued before the session failed hears why.
Expand Down Expand Up @@ -410,6 +425,11 @@ async function serve(pkg) {
const options = parseArgs(message.argv, message.cwd)
if (options.pkg !== pkg)
throw new Error(`this daemon serves ${pkg}, not ${options.pkg}`)
if (overlapMark !== undefined && !overlapHeld) {
overlapHeld = true
await secondAccepted
writeMark(overlapMark, `${accepted}`)
}
stdout = await session.check(options, store, (line) =>
stderr.push(`${line}\n`),
)
Expand Down
11 changes: 11 additions & 0 deletions packages/oracles/oracle-gaps/gap-report-model-oracle.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -860,6 +860,13 @@ const drawn = fc.record({
// The batch whose run rejects: a constructed-only field (0 is no fault). Only the constructed history at
// the end of the campaign sets it; the take-back below returns it to 0 (the generator rule, extended).
rejectBatch: fc.constant(0),
// Whether two checks at once are held to overlap in the daemon (the
// test-only hold ORACLE_GAPS_TEST_HOLD_OVERLAP, issue #82): the first check
// a daemon serves waits until it has accepted a second request. A hook is
// never switched on by a drawn field, so the constant draws nothing and
// only the constructed edit sequence through the daemon sets it; the
// take-back below returns it to false.
holdOverlap: fc.constant(false),
})

// The edit sequence over the witness module, the one list of steps that a
Expand Down Expand Up @@ -932,13 +939,15 @@ const histories = drawn.map(
pauseEngages: true,
warmReplaced: false,
rejectBatch: 0,
holdOverlap: false,
}
const earlier = history.steps.filter((step) => !laterEdits.includes(step))
return {
...taken,
pauseEngages: true,
warmReplaced: false,
rejectBatch: 0,
holdOverlap: false,
steps: earlier.length > 0 ? earlier : null,
laterSteps: later,
}
Expand Down Expand Up @@ -2969,6 +2978,8 @@ function resolveHistory(history) {
ordinary: history.ordinary === true,
// The batch whose run rejects (a test-only fault; 0 is none).
rejectBatch: history.rejectBatch ?? 0,
// Whether two checks at once are held to overlap in the daemon.
holdOverlap: history.holdOverlap === true,
// An `owns` key in an assertion's arguments, in the ordinary file: its
// assertion form and whether its list is a literal, or none.
matcherOwns: history.matcherOwns ?? `none`,
Expand Down
Loading
Loading