Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/gap-cli-daemon-every-read.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@tanstack/oracles': patch
---

A check through `--daemon` now treats every read as an input of the runs that made it, so after any of these changes it reports what a cold run reports. Before, the daemon reused a run that read the old input. The new inputs are a file that a test reads through `fs` outside the package or under `node_modules`; an environment variable that a test reads (or the set of variable names, when a test lists them); and a file or variable that Vitest, the Vite config, or one of the config's own plugins reads while it loads or in one of its hooks, such as a sidecar file that a plugin writes into the code it transforms.

The daemon now runs each check in the environment of the CLI that asked for it. A check from another environment, or after a file that the config or a plugin read has changed, starts a new daemon, which still reuses every stored run whose inputs are unchanged.

Each check now hashes every file input from its content, so an edit that keeps a file's size and modification time, as `cp -p`, `rsync -t`, or `touch -r` leave it, is no longer missed.
6 changes: 3 additions & 3 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -106,20 +106,20 @@ jobs:
env:
SHARD: ${{ matrix.shard }}
test-oracle-gaps-end-to-end:
name: Gap report end-to-end oracle (fixed seed, ${{ matrix.shard }}/3)
name: Gap report end-to-end oracle (fixed seed, ${{ matrix.shard }}/4)
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3]
shard: [1, 2, 3, 4]
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Setup Tools
uses: tanstack/config/.github/setup@e4b48f16568324f76f467aa4c2aac2f05db632c3
# The end-to-end oracle's fixed-seed units, in three shards. Its random
# The end-to-end oracle's fixed-seed units, in four shards. Its random
# units, calibrations, and determinism units run locally.
- run: pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "$(node scripts/pr-shards.mjs pattern oracle-gaps-fixed-seed "$SHARD")"
working-directory: packages/oracles
Expand Down
70 changes: 36 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,40 +194,42 @@ exited with <signal or code>` and exits 3; after an out-of-memory crash
it with `oracle-gaps --stop-daemon --package <dir>`. On Windows, which has
no process groups, the run is not supervised.

A check through the daemon reuses a run while its inputs are unchanged:
the modules the run loaded, and the files in the package that its tests
read through `fs` (`readFileSync`, `readFile`, `existsSync`, `statSync`,
and `fs/promises` `readFile` and `stat`), each by the content that the
run read. A run that an edit overlapped, because a file changed after the
run read it, is not reused. A directory that a test stats or tests for is
an input as a directory, not by its content.

Two kinds of input are not tracked yet. A Vite plugin whose output depends
on anything but a module's text, such as a sidecar file that it reads, can
make the daemon serve a run that read the plugin's old output. And the
daemon's environment is fixed when it starts, so an environment variable
that a later check sets does not reach the daemon's runs. After such a
change, run the check cold (without `--daemon`), or stop the daemon with
`oracle-gaps --stop-daemon --package <dir>`. The maintainer decided on
2026-11-13 that the daemon tracks both and restarts when they change; a
later release does that.

These reads are no input, so an edit to the file alone does not rerun the
tests that read it: a read outside the package or under `node_modules`; a
read from a worker thread or child process that the test starts; and a
read through any `fs` or `fs/promises` function other than the six above,
or through one of those six given a file descriptor. That catch-all
includes, for example, `readdir`, `opendir`, `glob`, `globSync`,
`createReadStream`, `realpath`, `lstat`, `access`, `open`, `read`,
`readSync`, `readv`, `fstat`, `fstatSync`, `copyFile`, `copyFileSync`, `cp`,
`cpSync`, `openAsBlob`, `statfs`, `readlink`, the callback `stat`,
`exists`, `watch`, and `watchFile`, and the `fs/promises` `readdir`,
`opendir`, `glob`, `lstat`, `realpath`, `readlink`, `access`, `open`,
`copyFile`, `cp`, `statfs`, and `watch`. Run the check cold (without
`--daemon`) after an edit to a file read that way. A later release records
all of them except `watch` and `watchFile`, which wait for future changes
and stay no input (maintainer decision, 2026-11-13, with `watchFile` in the
family of `watch`).
A check through the daemon reuses a run while its inputs are unchanged.
Every read is an input (maintainer decision, 2026-11-16): the modules the
run loaded; the files its tests read through `fs` on any path, inside the
package or outside it, `node_modules` included (`readFileSync`,
`readFile`, `existsSync`, `statSync`, and `fs/promises` `readFile` and
`stat`), each by the content that the run read; the environment variables
its tests read, by value, or all of them when a test lists them; and what
Vitest, the Vite config, and the config's own plugins read in the CLI's
process while it loads them and in each plugin hook. Each check hashes
every file input from its content, so an edit that keeps a file's size and
modification time (`cp -p`, `rsync -t`, `touch -r`) is seen. A run that an
edit overlapped, because a file changed after the run read it, is not
reused. A directory that a test stats or tests for is an input as a
directory, not by its content.

The daemon runs every check in the environment that the CLI asking for it
has. A check from another environment, or after a file that Vitest, the
config, or a plugin read has changed, stops the daemon and starts a new
one, which reads them again; the runs it stored are still reused where
their inputs are unchanged.

These reads are no input yet, so an edit to the file alone does not rerun
the tests that read it: a read from a worker thread or child process that
the test starts; and a read through any `fs` or `fs/promises` function
other than the six above, or through one of those six given a file
descriptor. That catch-all includes, for example, `readdir`, `opendir`,
`glob`, `globSync`, `createReadStream`, `realpath`, `lstat`, `access`,
`open`, `read`, `readSync`, `readv`, `fstat`, `fstatSync`, `copyFile`,
`copyFileSync`, `cp`, `cpSync`, `openAsBlob`, `statfs`, `readlink`, the
callback `stat`, `exists`, `watch`, and `watchFile`, and the `fs/promises`
`readdir`, `opendir`, `glob`, `lstat`, `realpath`, `readlink`, `access`,
`open`, `copyFile`, `cp`, `statfs`, and `watch`. Run the check cold
(without `--daemon`) after an edit to a file read that way. A later
release records all of them except `watch` and `watchFile`, which wait for
future changes and stay no input (maintainer decision, 2026-11-13, with
`watchFile` in the family of `watch`).

- `packages/eslint-plugin-oracles` (`@tanstack/eslint-plugin-oracles`): the
`oracle/*` lint rules. The rules recognize the framework by its package
Expand Down
109 changes: 27 additions & 82 deletions packages/oracles/oracle-gaps/capture-setup.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@
// In a mutant run, fast-check stops a property at its first failure instead
// of shrinking it: a kill needs one failing history, not the smallest one.
import { createHash } from 'node:crypto'
import fs, { appendFileSync, readFileSync } from 'node:fs'
import { appendFileSync, readFileSync } from 'node:fs'
import { createRequire, syncBuiltinESMExports } from 'node:module'
import { dirname, isAbsolute, join, resolve as resolvePath } from 'node:path'
import { fileURLToPath } from 'node:url'
import { dirname, isAbsolute, join } from 'node:path'
import { afterAll, beforeEach, chai, expect, inject, vi } from 'vitest'
import { posix, writeWhole } from './paths.mjs'
import { fileHash, keepProperties, watchReads } from './inputs.mjs'
import { writeWhole } from './paths.mjs'

const run =
inject(`oracleGaps`) ??
Expand Down Expand Up @@ -68,14 +68,7 @@ const realUse = real.use

// A wrapper keeps every own property of the function it wraps, such as
// `util.promisify.custom`, so `util.promisify(setTimeout)` still resolves
// with its value under the capture (law 3: a test that passes alone passes
// under the CLI).
function keepProperties(from, to) {
for (const key of Reflect.ownKeys(from))
if (key !== `prototype`)
Object.defineProperty(to, key, Object.getOwnPropertyDescriptor(from, key))
return to
}
// with its value under the capture (inputs.mjs, keepProperties).

// The timers that code in this worker has pending, counted where they are
// made, each with when it is next due: Node's resource list does not
Expand Down Expand Up @@ -274,70 +267,26 @@ function appendClosingRow(row) {
writeWhole(`${hold}.${marker}`, ``)
appendFileSync(state.capture, row.slice(half))
}
// The files this copy reads through fs inside the package: by a sync, a
// promise, or a callback read, or a check of whether a file exists or what
// it is. The daemon stores the run under them as it does the modules the
// copy evaluated (the closing row's `loaded`), so an edit of only such a
// file runs it again, as a cold run sees the edit (law 9). Reads under
// node_modules and of the runner's own files are left out. Each path keeps
// its hash from the copy's first read of it, taken as the runner hashes an
// input (check.mjs, fileHashes), so the run is stored under the text it
// read and not under an edit that lands later in the run (review of #55,
// F1).
// What this copy reads: each file through fs, on any path, node_modules
// included, by a sync, a promise, or a callback read, or a check of whether
// a file exists or what it is; and each environment variable, or the set of
// names when the copy lists them (inputs.mjs). Every read is a daemon input
// (maintainer decision, 2026-11-16): the daemon stores the run under them as
// it does the modules the copy evaluated (the closing row's `loaded`), so an
// edit of only such an input runs it again, as a cold run sees the edit
// (law 9). A file keeps its hash from the copy's first read of it, so the
// run is stored under the text it read and not under an edit that lands
// later in the run (review of #55, F1). A variable's hash is the runner's,
// from the environment it gives the check (check.mjs, servedHash). A file
// the copy wrote before it read it is its own output, no input.
globalThis.__oracleGapsReads = new Map()
if (!globalThis.__oracleGapsReading) {
globalThis.__oracleGapsReading = true
// A read's key follows the gap CLI's path rule (paths.mjs, copied beside
// this file with it). The file system gets the native path.
const root = `${posix(process.cwd()).replace(/\/+$/, ``)}/`
const { readFileSync: readNow, statSync: statNow } = fs
const hashNow = (path) => {
try {
return statNow(path).isFile()
? createHash(`sha1`).update(readNow(path)).digest(`hex`)
: `not a file`
} catch {
return `missing`
}
}
const record = (target) => {
let path
try {
if (target instanceof URL) path = fileURLToPath(target)
else if (typeof target === `string`) path = resolvePath(target)
else if (Buffer.isBuffer(target)) path = resolvePath(target.toString())
} catch {
return
}
if (path === undefined) return
const key = posix(path)
const reads = globalThis.__oracleGapsReads
if (
key.startsWith(root) &&
!key.includes(`/node_modules/`) &&
!key.includes(`/.oracle-gaps`) &&
!reads.has(key)
)
reads.set(key, hashNow(path))
}
// A wrapper keeps the function's own properties (its name and length,
// which code that dispatches on arity reads), so a test that passes alone
// passes under the capture (law 3; review of #55).
for (const [owner, names] of [
[fs, [`readFileSync`, `readFile`, `existsSync`, `statSync`]],
[fs.promises, [`readFile`, `stat`]],
])
for (const name of names) {
const original = owner[name]
owner[name] = keepProperties(original, function (target, ...rest) {
record(target)
return original.call(this, target, ...rest)
})
}
// An ES module's `import { readFileSync } from 'node:fs'` (and from
// 'node:fs/promises') reads these from now on.
syncBuiltinESMExports()
}
const wrote = new Set()
watchReads((key, write) => {
const reads = globalThis.__oracleGapsReads
if (write) wrote.add(key)
else if (!reads.has(key) && !wrote.has(key))
reads.set(key, key.startsWith(`fs:`) ? fileHash(key.slice(3)) : null)
})
// A test's own writes to the process's stdout and stderr (a debugging
// `process.stdout.write`) go to the run's test-output.log, beside its
// capture, not to the CLI's streams, whose every line is a line of the
Expand Down Expand Up @@ -1122,13 +1071,9 @@ afterAll(() => {
// The hash of each file the copy read, from its first read, under its own
// key (`fs:<path>`): a test's read of a file is another input than the
// module that its context serves from that file, which can differ from
// the file's text (fourth review of #55, F-A2).
const readAt = Object.fromEntries(
[...globalThis.__oracleGapsReads].map(([path, hash]) => [
`fs:${path}`,
hash,
]),
)
// the file's text (fourth review of #55, F-A2). A variable it read has no
// hash here (`env:<name>`, null).
const readAt = Object.fromEntries(globalThis.__oracleGapsReads)
appendClosingRow(
`${JSON.stringify({ file: thisFile, closed: true, loaded, readAt, ...(skippedCopy ? { skipped: true } : {}) })}\n`,
)
Expand Down
Loading
Loading