Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
2bbfecc
test(oracle-gaps): law 9 over an edit that lands during a check (WIP)
KyleAMathews Nov 12, 2026
f17ccd1
Merge origin/main (#47) into test/gap-daemon-midrun-edit
KyleAMathews Nov 12, 2026
15cc6dd
test(oracle-gaps): each mid-run edit changes file a's text as well
KyleAMathews Nov 12, 2026
5772215
test(oracle-gaps): law 9 over a data file that a test reads through fs
KyleAMathews Nov 12, 2026
d98af2e
fix(oracle-gaps): a file a test reads through fs is an input of its run
KyleAMathews Nov 13, 2026
4be4a1c
docs(oracle-gaps): the daemon's fs inputs, their limits, and a changeset
KyleAMathews Nov 13, 2026
2dcb803
Merge origin/main (#46) into test/gap-daemon-midrun-edit
KyleAMathews Nov 13, 2026
6a789a2
test(oracle-gaps): a test that stats its directory, fs functions keep…
KyleAMathews Nov 13, 2026
0d57c35
fix(oracle-gaps): hash a directory a test stats as a marker; keep the…
KyleAMathews Nov 13, 2026
ef40cbb
Merge origin/main (#48, #54) into test/gap-daemon-midrun-edit
KyleAMathews Nov 13, 2026
40527b4
test(oracle-gaps): a run that edits the input it read must not be reu…
KyleAMathews Nov 13, 2026
b5578ee
test(oracle-gaps): the mid-run marker sits in the daemon's session di…
KyleAMathews Nov 13, 2026
82b5c17
fix(oracle-gaps): store a run under the text it read
KyleAMathews Nov 13, 2026
4b765a4
test(oracle-gaps): unchanged reuse under a pre plugin, and the mid-ru…
KyleAMathews Nov 13, 2026
1cfab2b
fix(oracle-gaps): hash a transformed module as every input is hashed;…
KyleAMathews Nov 13, 2026
5dd0ab4
test(oracle-gaps): an unchanged check reuses runs whose tests read a …
KyleAMathews Nov 13, 2026
b687d5b
fix(oracle-gaps): a test's fs read of a file and the module served fr…
KyleAMathews Nov 13, 2026
846deb6
Merge origin/main (#63) into test/gap-daemon-midrun-edit
KyleAMathews Nov 13, 2026
af4a008
ci(oracle-gaps): put the F-C1 calibration in an e2e shard
KyleAMathews Nov 13, 2026
90d7520
test(oracle-gaps): the incremental violation names a cold run's error…
KyleAMathews Nov 13, 2026
b959d4c
test(oracle-gaps): the incremental observation carries a cold run's e…
KyleAMathews Nov 13, 2026
8a42ae8
test(oracle-gaps): draw new history fields last, guarded by a canary;…
KyleAMathews Nov 13, 2026
c575207
docs(oracle-gaps): the README's fs catch-all, and one spread less in …
KyleAMathews Nov 13, 2026
3df4d1b
test(oracle-gaps): type the history canary's samples as records
KyleAMathews Nov 13, 2026
bf3b13e
test(oracle-gaps): the editing file's mutants go unjudged in an overl…
KyleAMathews Nov 14, 2026
ed091f3
test(oracle-gaps): no construction takes a random later edit; histori…
KyleAMathews Nov 14, 2026
16f510c
Merge origin/main (#58, Windows) into test/gap-daemon-midrun-edit
KyleAMathews Nov 14, 2026
0b5c39f
test(oracle-gaps): a batch's watchdog reads a closing row half written
KyleAMathews Nov 14, 2026
d9ca325
test(oracle-gaps): torn rows are constructed-only; an internal error …
KyleAMathews Nov 14, 2026
e8f4b83
fix(oracle-gaps): a batch's watchdog parses only whole capture rows
KyleAMathews Nov 14, 2026
f7dac92
test(oracle-gaps): record the torn-row state after the driver's state…
KyleAMathews Nov 14, 2026
c3b757b
fix(oracle-gaps): both readers of a batch's capture take whole rows (…
KyleAMathews Nov 15, 2026
4b24539
fix(oracle-gaps): a batch's capture is read strictly once the run has…
KyleAMathews Nov 15, 2026
0be2638
test(oracle-gaps): the two-batch witness is in the e2e shard partition
KyleAMathews Nov 15, 2026
8d15f0f
test(oracle-gaps): every batch's hold is witnessed; the open cut-off …
KyleAMathews Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/gap-cli-daemon-fs-inputs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@tanstack/oracles': patch
---

A check through `--daemon` now reruns a test after an edit to a file in the package that the test reads through `fs` (`readFileSync`, `readFile`, `existsSync`, `statSync`, or `fs/promises` `readFile` or `stat`), such as a JSON fixture or a snapshot. Before, the daemon reused the run that read the old file, while a cold run saw the edit. A directory that a test stats is an input as a directory. Reads outside the package, under `node_modules`, through other APIs, or from a test's own worker threads or child processes are no inputs.

The daemon also stores each run under the text that the run read. Before, it stored a run under each input's text at the end of the run, so a module or file that changed during the run that read it (a test that rewrites a fixture it read, or a helper edited mid-check) kept the old verdict for the new text, and the next check through the daemon reported that stale verdict. A run whose input changed after it read it is now run again.

A test that reads a source under test through `fs` (or as a `?raw` import) no longer makes each check through the daemon run its tests again. The test's read of the file and the instrumented module that the runner serves from it are now two inputs, each hashed by its own text.
45 changes: 43 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,11 +125,17 @@ agent.
A run through the daemon prints the same lines, at the end of its check.
`--help` is not a check, so its usage text has no prefix.

For contributors: two environment variables in the `ORACLE_GAPS_TEST_`
For contributors: four environment variables in the `ORACLE_GAPS_TEST_`
namespace are test-only hooks for the CLI's own oracle, not settings for
users. `ORACLE_GAPS_TEST_HOLD_EXIT_MS` holds the run's exit after its done
line, and `ORACLE_GAPS_TEST_WARM_PROBE` makes the first config warm-up
wait where a stop signal can reach it.
wait where a stop signal can reach it. `ORACLE_GAPS_TEST_STOP` names a
file: when it appears, an unsupervised run takes the same stop as a
SIGINT. `ORACLE_GAPS_TEST_TORN_ROWS` names a directory where each batch's
watchdog counts its polls, and a batch's closing row is held half written
until the watchdog has polled twice. The directory holds a `.begun` marker
per batch and a `.reached` or `.unreached` marker per hold (`.unreached`
after 60 s with no two polls).

A few endings print no done line, or one more line: a SIGKILL of the CLI,
or a signal that comes while Node is still starting the CLI, before its
Expand Down Expand Up @@ -158,6 +164,41 @@ exited with <signal or code>` and exits 3; after an out-of-memory crash
it with `oracle-gaps --stop-daemon --package <dir>`. On Windows, which has
no process groups, the run is not supervised.

A check through the daemon reuses a run while its inputs are unchanged:
the modules the run loaded, and the files in the package that its tests
read through `fs` (`readFileSync`, `readFile`, `existsSync`, `statSync`,
and `fs/promises` `readFile` and `stat`), each by the content that the
run read. A run that an edit overlapped, because a file changed after the
run read it, is not reused. A directory that a test stats or tests for is
an input as a directory, not by its content.

Two kinds of input are not tracked yet. A Vite plugin whose output depends
on anything but a module's text, such as a sidecar file that it reads, can
make the daemon serve a run that read the plugin's old output. And the
daemon's environment is fixed when it starts, so an environment variable
that a later check sets does not reach the daemon's runs. After such a
change, run the check cold (without `--daemon`), or stop the daemon with
`oracle-gaps --stop-daemon --package <dir>`. The maintainer decided on
2026-11-13 that the daemon tracks both and restarts when they change; a
later release does that.

These reads are no input, so an edit to the file alone does not rerun the
tests that read it: a read outside the package or under `node_modules`; a
read from a worker thread or child process that the test starts; and a
read through any `fs` or `fs/promises` function other than the six above,
or through one of those six given a file descriptor. That catch-all
includes, for example, `readdir`, `opendir`, `glob`, `globSync`,
`createReadStream`, `realpath`, `lstat`, `access`, `open`, `read`,
`readSync`, `readv`, `fstat`, `fstatSync`, `copyFile`, `copyFileSync`, `cp`,
`cpSync`, `openAsBlob`, `statfs`, `readlink`, the callback `stat`,
`exists`, `watch`, and `watchFile`, and the `fs/promises` `readdir`,
`opendir`, `glob`, `lstat`, `realpath`, `readlink`, `access`, `open`,
`copyFile`, `cp`, `statfs`, and `watch`. Run the check cold (without
`--daemon`) after an edit to a file read that way. A later release records
all of them except `watch` and `watchFile`, which wait for future changes
and stay no input (maintainer decision, 2026-11-13, with `watchFile` in the
family of `watch`).

- `packages/eslint-plugin-oracles` (`@tanstack/eslint-plugin-oracles`): the
`oracle/*` lint rules. The rules recognize the framework by its package
specifier and by a relative module with its file name. A project names its
Expand Down
120 changes: 115 additions & 5 deletions packages/oracles/oracle-gaps/capture-setup.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,12 @@
// In a mutant run, fast-check stops a property at its first failure instead
// of shrinking it: a kill needs one failing history, not the smallest one.
import { createHash } from 'node:crypto'
import { appendFileSync, readFileSync } from 'node:fs'
import fs, { appendFileSync, readFileSync } from 'node:fs'
import { createRequire, syncBuiltinESMExports } from 'node:module'
import { dirname, isAbsolute, join } from 'node:path'
import { dirname, isAbsolute, join, resolve as resolvePath } from 'node:path'
import { fileURLToPath } from 'node:url'
import { afterAll, beforeEach, chai, expect, inject, vi } from 'vitest'
import { posix, writeWhole } from './paths.mjs'

const run =
inject(`oracleGaps`) ??
Expand Down Expand Up @@ -237,6 +239,105 @@ if (run.units && !skippedCopy) {
vi.resetModules()
}
const state = unit ? { ...run, ...unit } : run
// A test-only hook for the CLI's own oracle, not for users:
// ORACLE_GAPS_TEST_TORN_ROWS names a directory where each batch's watchdog
// counts its polls (`state.hold`, check.mjs). A batch's closing row is then
// written in two appends, and the second waits until the watchdog has polled
// twice after the first, so a poll reads the row half written, by construction
// and not by timing (issue #83). The hold writes `<hold>.reached` once that
// happens, or `<hold>.unreached` if the watchdog does not poll within 60 s,
// which the oracle reports as an unreached witness. Every row ends with a
// newline in either case.
function appendClosingRow(row) {
const hold = state.hold
// Only a batch's copy (`unit`) has a watchdog that polls.
if (hold === undefined || unit === undefined)
return appendFileSync(state.capture, row)
const half = row.length >> 1
appendFileSync(state.capture, row.slice(0, half))
const counted = () => {
try {
return Number(readFileSync(`${hold}.polls`, `utf8`))
} catch {
return 0
}
}
const seen = counted()
const pause = new Int32Array(new SharedArrayBuffer(4))
// The bound only keeps a broken watchdog from holding the worker forever.
let waited = 0
while (counted() < seen + 2 && waited < 60_000) {
Atomics.wait(pause, 0, 0, 10)
waited += 10
}
const marker = counted() >= seen + 2 ? `reached` : `unreached`
writeWhole(`${hold}.${marker}`, ``)
appendFileSync(state.capture, row.slice(half))
}
// The files this copy reads through fs inside the package: by a sync, a
// promise, or a callback read, or a check of whether a file exists or what
// it is. The daemon stores the run under them as it does the modules the
// copy evaluated (the closing row's `loaded`), so an edit of only such a
// file runs it again, as a cold run sees the edit (law 9). Reads under
// node_modules and of the runner's own files are left out. Each path keeps
// its hash from the copy's first read of it, taken as the runner hashes an
// input (check.mjs, fileHashes), so the run is stored under the text it
// read and not under an edit that lands later in the run (review of #55,
// F1).
globalThis.__oracleGapsReads = new Map()
if (!globalThis.__oracleGapsReading) {
globalThis.__oracleGapsReading = true
// A read's key follows the gap CLI's path rule (paths.mjs, copied beside
// this file with it). The file system gets the native path.
const root = `${posix(process.cwd()).replace(/\/+$/, ``)}/`
const { readFileSync: readNow, statSync: statNow } = fs
const hashNow = (path) => {
try {
return statNow(path).isFile()
? createHash(`sha1`).update(readNow(path)).digest(`hex`)
: `not a file`
} catch {
return `missing`
}
}
const record = (target) => {
let path
try {
if (target instanceof URL) path = fileURLToPath(target)
else if (typeof target === `string`) path = resolvePath(target)
else if (Buffer.isBuffer(target)) path = resolvePath(target.toString())
} catch {
return
}
if (path === undefined) return
const key = posix(path)
const reads = globalThis.__oracleGapsReads
if (
key.startsWith(root) &&
!key.includes(`/node_modules/`) &&
!key.includes(`/.oracle-gaps`) &&
!reads.has(key)
)
reads.set(key, hashNow(path))
}
// A wrapper keeps the function's own properties (its name and length,
// which code that dispatches on arity reads), so a test that passes alone
// passes under the capture (law 3; review of #55).
for (const [owner, names] of [
[fs, [`readFileSync`, `readFile`, `existsSync`, `statSync`]],
[fs.promises, [`readFile`, `stat`]],
])
for (const name of names) {
const original = owner[name]
owner[name] = keepProperties(original, function (target, ...rest) {
record(target)
return original.call(this, target, ...rest)
})
}
// An ES module's `import { readFileSync } from 'node:fs'` (and from
// 'node:fs/promises') reads these from now on.
syncBuiltinESMExports()
}
// A test's own writes to the process's stdout and stderr (a debugging
// `process.stdout.write`) go to the run's test-output.log, beside its
// capture, not to the CLI's streams, whose every line is a line of the
Expand Down Expand Up @@ -1027,8 +1128,17 @@ afterAll(() => {
.filter(([, node]) => node.evaluated)
.map(([id]) => id.split(`?`)[0])
.filter((path) => isAbsolute(path))
appendFileSync(
state.capture,
`${JSON.stringify({ file: thisFile, closed: true, loaded, ...(skippedCopy ? { skipped: true } : {}) })}\n`,
// The hash of each file the copy read, from its first read, under its own
// key (`fs:<path>`): a test's read of a file is another input than the
// module that its context serves from that file, which can differ from
// the file's text (fourth review of #55, F-A2).
const readAt = Object.fromEntries(
[...globalThis.__oracleGapsReads].map(([path, hash]) => [
`fs:${path}`,
hash,
]),
)
appendClosingRow(
`${JSON.stringify({ file: thisFile, closed: true, loaded, readAt, ...(skippedCopy ? { skipped: true } : {}) })}\n`,
)
})
Loading
Loading