Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
d1b6f67
test(oracle-gaps): draw warmReplaced, testRoot and loadPlugin (random…
KyleAMathews Nov 16, 2026
1dfe3ae
test(oracle-gaps): a check whose daemon stops answering owes a cold f…
KyleAMathews Nov 16, 2026
0f10264
test(oracle-gaps): warm the held-reply daemon before its pause is armed
KyleAMathews Nov 16, 2026
37c3e0c
fix(oracle-gaps): a CLI whose daemon stops answering runs its check cold
KyleAMathews Nov 16, 2026
d0e32ec
test(oracle-gaps): a spun copy that reads data only its mutant reads …
KyleAMathews Nov 16, 2026
85e4dd2
test(oracle-gaps): restore the warm reset state; the mid-run edit und…
KyleAMathews Nov 16, 2026
971f22e
fix(oracle-gaps): a copy that spins in a batch is stored under what i…
KyleAMathews Nov 16, 2026
ae8ccc7
test(oracle-gaps): limits and shards for the cold fallback and F-D hi…
KyleAMathews Nov 16, 2026
8951893
refactor(oracle-gaps): one constructor for a spun run's unit, alone o…
KyleAMathews Nov 16, 2026
907d05c
test(oracle-gaps): the alone path's F-D limit names the shared constr…
KyleAMathews Nov 16, 2026
59a7f9e
test(oracle-gaps): no drawn stop switches a test-only hook; restore t…
KyleAMathews Nov 16, 2026
9173d35
test(oracles): the history canary's stop line is POSIX's
KyleAMathews Nov 16, 2026
cc6af3d
Merge remote-tracking branch 'origin/main' into fix/merged-conformance
KyleAMathews Nov 16, 2026
c58783d
test(oracle-gaps): a refused run beside the CLI's directory, under a …
KyleAMathews Nov 16, 2026
c0f034b
test(oracles): the hook canary judges what the driver would switch on
KyleAMathews Nov 16, 2026
122fec1
test(oracle-gaps): no drawn step arms a daemon pause; one history on …
KyleAMathews Nov 16, 2026
3d7097d
test(oracle-gaps): a test-only pause that holds a daemon's check befo…
KyleAMathews Nov 16, 2026
4fbe231
test(oracle-gaps): the cold-fallback witness holds the check before i…
KyleAMathews Nov 16, 2026
d1e233b
fix(oracle-gaps): a CLI that falls back ends the daemon that held its…
KyleAMathews Nov 16, 2026
08e967a
test(oracle-gaps): F-D on the alone path, a spun copy that throws out…
KyleAMathews Nov 16, 2026
f22ef48
fix(oracle-gaps): each run reads only its own errors outside any test
KyleAMathews Nov 16, 2026
daa8d02
test(oracle-gaps): the held witness poisons an --out its CLI never wrote
KyleAMathews Nov 16, 2026
52f9cdf
test(oracle-gaps): a held check's CLI that never ends is reported bef…
KyleAMathews Nov 16, 2026
21d37bc
Merge remote-tracking branch 'origin/main' into fix/merged-conformance
KyleAMathews Nov 16, 2026
addd378
test(oracle-gaps): the lazy-hang history keeps file b's verdict reach…
KyleAMathews Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ coverage
# The gap CLI writes scratch packages and state beside the package it checks,
# and its oracle writes scratch packages and CLI copies beside itself.
/packages/oracles/oracle-gaps/.scratch-*
/packages/oracles/oracle-gaps-sibling-*/
/packages/oracles/oracle-gaps-mutant*/
.oracle-gaps/
.oracle-gaps-*
Expand Down
28 changes: 18 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,18 +146,21 @@ agent.
`ORACLE_GAPS_TEST_PAUSE_MARK` naming a file the blocked process writes its
pid to: `daemon-reply` holds a daemon's reply after the check, and
`daemon-read` holds its first connection unread, once `ORACLE_GAPS_TEST_PAUSE_ARM`
names an existing file. The test kills the process by that pid, so no signal
names an existing file (`daemon-reply` too, when `ORACLE_GAPS_TEST_PAUSE_ARM`
is set). `daemon-check` holds a daemon's first armed check before it runs,
until the file `ORACLE_GAPS_TEST_PAUSE_RELEASE` names exists. The test kills the process by that pid, so no signal
is needed to pause on win32. Both marks are written whole (a temp file, then
a rename), and the test parses the pid as a positive integer before it sends
any signal.
- `ORACLE_GAPS_TEST_WAIT_MARK` names a file a request writes when its reply has
not come `ORACLE_GAPS_TEST_WAIT_MS` after it was sent. A request has no
timeout of its own: this timer starts with the request, and is cleared when
the request settles, whether it resolves or rejects. The test acts on the
CLI's wait, not on the clock.
- `ORACLE_GAPS_TEST_WAIT_MARK` names a file a check's request writes when
its daemon has sent nothing for `ORACLE_GAPS_TEST_WAIT_MS`, the point at
which the CLI would run the check cold. With the mark set, the request
writes it there and keeps waiting, so the test acts on the CLI's wait, not
on the clock.
- `ORACLE_GAPS_TEST_WAIT_MS` (milliseconds, default 60000) is the CLI's wait
for a daemon to start or to close; a test shortens it to outlast a paused
daemon. It is a test-only knob, not a setting for users.
for a daemon to start or to close, and for a daemon that holds its check
to send anything; a test shortens it to outlast a paused daemon. It is a
test-only knob, not a setting for users.
- `ORACLE_GAPS_TEST_SERVED_MARK` names a file the daemon writes its pid to
after it answers a check, so a test can tell which daemon served an
earlier check (the warm daemon that a later check's pause must match).
Expand Down Expand Up @@ -191,8 +194,13 @@ exited with <signal or code>` and exits 3; after an out-of-memory crash
of its own (as some browser drivers are). On SIGINT or SIGTERM the CLI
signals the whole group, kills what is left 2 s later, and exits 130 or 143.
A check through the daemon (`--daemon`) runs in the daemon's process; stop
it with `oracle-gaps --stop-daemon --package <dir>`. On Windows, which has
no process groups, the run is not supervised.
it with `oracle-gaps --stop-daemon --package <dir>`. While a check waits or
runs there, the daemon tells the CLI each time a check makes progress. A
daemon that sends nothing for 60 s has stopped answering: the CLI ends
it, prints `oracle-gaps: note the package's daemon sent nothing for 60 s;
this check runs cold`, and runs the check in its own process, which
reports what the daemon would have. On Windows, which has no process groups, the run is not
supervised.

A check through the daemon reuses a run while its inputs are unchanged:
the modules the run loaded, and the files in the package that its tests
Expand Down
73 changes: 49 additions & 24 deletions packages/oracles/oracle-gaps/check.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -862,12 +862,15 @@ export default async (env) => {
// and for how long, so that a long run never looks frozen. Its label is
// its mutant's id, an authored mutant's `authored:<id>`, or `baseline`
// for a run with no mutant. The interval is 30 s; the CLI's oracle
// shortens it (ORACLE_GAPS_HEARTBEAT_MS). Each check sets where it logs.
let pulse = { log: () => {}, every: 30_000 }
// shortens it (ORACLE_GAPS_HEARTBEAT_MS). Each check sets where it logs,
// and whom each poll of a running copy tells that the check progresses
// (`alive`: the daemon's beat to the CLIs that wait on it).
let pulse = { log: () => {}, alive: () => {}, every: 30_000 }
function heartbeat() {
let since
let sent = 0
return (label, file, started) => {
pulse.alive()
if (started !== since) {
since = started
sent = 0
Expand Down Expand Up @@ -930,6 +933,7 @@ export default async (env) => {
context.getRootProject?.().provide(`oracleGaps`, state)
writeFileSync(join(out, `state.json`), JSON.stringify(state))
const started = Date.now()
clearErrors(context)
const running = context.runTestSpecifications(
contextSpecs.get(context).get(file) ?? [],
false,
Expand Down Expand Up @@ -1013,16 +1017,7 @@ export default async (env) => {
}
rmSync(capture)
}
// A run that spun ended, so its closing row names what it read (F-D).
if (spun)
return {
file,
timedOut: true,
spin: true,
seconds,
inputs: importsOf(file, modules, closing),
readAt: closing.readAt,
}
if (spun) return spunRun(file, seconds, modules, closing)
const tests = modules.flatMap((module) =>
[...module.children.allTests()].map((testCase) => {
const testResult = testCase.result()
Expand Down Expand Up @@ -1069,6 +1064,22 @@ export default async (env) => {
}
}

/**
* The unit of a run that spun past its call budget, alone or in a batch:
* a timeout. It ended, so its closing row names what it read, which it is
* stored under (F-D).
*/
function spunRun(file, seconds, modules, closing = {}) {
return {
file,
timedOut: true,
spin: true,
seconds,
inputs: importsOf(file, modules, closing),
readAt: closing.readAt,
}
}

/**
* The modules a run of `file` loaded: those Vitest reports it imported,
* and those its copy evaluated (`loaded`, from the capture's closing row),
Expand Down Expand Up @@ -1173,6 +1184,7 @@ export default async (env) => {
writeFileSync(join(out, `state.json`), JSON.stringify(state))
const project = context.getRootProject()
const started = Date.now()
clearErrors(context)
const run = context.runTestSpecifications(
copies.map((copy) => project.createSpecification(copy)),
false,
Expand Down Expand Up @@ -1277,15 +1289,6 @@ export default async (env) => {
return rows
}
const skippedUnit = (item) => ({ file: item.file, skipped: true })
// A copy that ran past its call budget, even one whose throw ended the
// loop and let it finish, is a timeout as it stands (a count, not a
// clock: no confirming run).
const spunUnit = (item) => ({
file: item.file,
timedOut: true,
spin: true,
seconds,
})
if (result === undefined) {
// Cancelled without waiting; the caller replaces the context (see
// runUnit).
Expand All @@ -1306,7 +1309,11 @@ export default async (env) => {
units: items.slice(0, current).map((item, i) => {
if (!closed.has(copies[i])) return undefined
if (skipped.has(copies[i])) return skippedUnit(item)
if (spun.has(copies[i])) return spunUnit(item)
// A copy that ran past its call budget, even one whose throw
// ended the loop and let it finish, is a timeout as it stands (a
// count, not a clock: no confirming run).
if (spun.has(copies[i]))
return spunRun(item.file, seconds, [], closings.get(copies[i]))
const map = keyFor(copies[i], item.file)
const own = rows
.get(copies[i])
Expand Down Expand Up @@ -1350,7 +1357,8 @@ export default async (env) => {
units: items.map((item, i) => {
const copy = copies[i]
if (skipped.has(copy)) return skippedUnit(item)
if (spun.has(copy)) return spunUnit(item)
if (spun.has(copy))
return spunRun(item.file, seconds, [], closings.get(copy))
const map = keyFor(copy, item.file)
const modules = result.testModules.filter((m) => m.moduleId === copy)
return {
Expand Down Expand Up @@ -1427,6 +1435,8 @@ export default async (env) => {
throw new Error(
`oracle-gaps: this Vitest keeps no filesMap, idMap, or pathsSet in its state, so a check cannot retire its copies`,
)
// The batch's errors outside any test are plain data in its units now.
clearErrors(context)
const gone = new Set(copies)
for (const [id, task] of idMap)
if (gone.has(task.file?.filepath ?? task.filepath)) idMap.delete(id)
Expand All @@ -1437,6 +1447,20 @@ export default async (env) => {
retireModules(context.getRootProject().vite, gone)
}

/**
* Drops the errors outside any test that Vitest's state holds. Vitest keeps
* them across runs (its own CLI clears them per run), so a run that does
* not start without them reads an earlier run's errors as its own (its
* batch then runs alone), and a batch that keeps them retains them.
*/
function clearErrors(context) {
if (typeof context.state?.clearErrors !== `function`)
throw new Error(
`oracle-gaps: this Vitest's state has no clearErrors, so a run cannot tell its own errors outside any test`,
)
context.state.clearErrors()
}

/**
* Removes the copies' module nodes from every environment's Vite graph.
* Each node is invalidated through Vite's API (which drops its transform),
Expand Down Expand Up @@ -1492,7 +1516,7 @@ export default async (env) => {
return environments
}

async function check(options, store, log = console.error) {
async function check(options, store, log = console.error, alive = () => {}) {
const authored = options.mutants ? loadAuthored(options.mutants) : []
seedVariable = `${options.envPrefix}_SEED`
if (options.hangLimit > options.mutantLimit)
Expand All @@ -1501,6 +1525,7 @@ export default async (env) => {
)
pulse = {
log,
alive,
every: Number(process.env.ORACLE_GAPS_HEARTBEAT_MS ?? 30_000),
}
mkdirSync(options.out, { recursive: true })
Expand Down
Loading
Loading