Skip to content
5 changes: 5 additions & 0 deletions .changeset/gap-cli-probe-not-work.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@tanstack/oracles': patch
---

The `oracle-gaps` work count no longer includes probe code. A site in probe-only code (the probe guard, a probe statement, or a probe helper) counts nothing toward a test's work, so a work bound no longer kills a mutant whose only extra work is a probe, such as an `if` forced into a branch that only calls `report()`.
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,9 @@ agent.
no unmodified run to compare with, so there the bound judges nothing. The
count does not see work without a site, such as a plain assignment, a
`for...of` pass, or a builtin's own work, so a call to a builtin counts
once, whatever it costs. Where a shortcut's cost is in such a builtin
once, whatever it costs. Probe code is never work: a site in probe-only
code (the probe guard, a probe statement, or a probe helper) counts
nothing, so a mutant that only enters probe code does no extra work. Where a shortcut's cost is in such a builtin
(`Object.defineProperty` and the like), make that cost an explicit
observation of the driver, such as a count of those calls, and let a law
judge it, in place of a very tight work bound. Work is counted per test, not per history: a
Expand Down
6 changes: 4 additions & 2 deletions packages/oracles/oracle-gaps/capture-setup.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -465,13 +465,15 @@ const active = state.mutant
// comparison, a logical operator, an `if`, and each pass of a loop's
// condition count, and code with no site (a plain assignment, a `for...of`
// pass, a builtin's own work, code outside the instrumented sources) counts
// nothing.
// nothing. Probe code is never work (maintainer decision, 2026-11-16): a
// probe-only site (check.mjs `probeSites`) counts nothing either.
let calls = 0
let spun = false
const probeOnly = new Set(state.probeSites)
const hit = (id) => {
;(inTest ? hits : background).add(id)
historyHits?.add(id)
if (inTest) calls += 1
if (inTest && !probeOnly.has(id)) calls += 1
if (spun || (state.callBudget && calls > state.callBudget)) spin()
}
function spin() {
Expand Down
7 changes: 7 additions & 0 deletions packages/oracles/oracle-gaps/check.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -610,6 +610,10 @@ export default async (env) => {
// The name of the seed variable that the framework reads (`${envPrefix}_SEED`),
// set by each check and carried in each run's state (capture-setup.mjs).
let seedVariable = `ORACLE_SEED`
// The probe-only sites (each one's number), set by each check and carried
// in each run's state: probe code is never work (maintainer decision,
// 2026-11-16), so the capture counts no call there (capture-setup.mjs).
let probeSites = []
const contextSpecs = new Map()
let runCounter = 0
// The retention test hook (retention-hook.mjs, README "Test hooks"): loaded
Expand Down Expand Up @@ -915,6 +919,7 @@ export default async (env) => {
capture,
seed,
seedVariable,
probeSites,
stopOnKill,
ignore,
callBudget,
Expand Down Expand Up @@ -1159,6 +1164,7 @@ export default async (env) => {
capture,
seed,
seedVariable,
probeSites,
units,
hold,
}
Expand Down Expand Up @@ -1578,6 +1584,7 @@ export default async (env) => {
options.sources,
[...contextSpecs.get(contexts[0]).keys()],
)
probeSites = mutants.filter((m) => m.probeGuard).map((m) => m.number)
const previous = [...globalThis.__oracleGapsSchemata.keys()]
globalThis.__oracleGapsSchemata = instrumented
invalidateChanged(
Expand Down
6 changes: 4 additions & 2 deletions packages/oracles/oracle-gaps/gap-report-model-oracle.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3636,10 +3636,12 @@ function interpret(plan, fnId, x, sites, active) {
const trace = []
const reached = new Set()
// The run's work: one for each site evaluation, as the capture counts a
// test's `calls` (capture-setup.mjs), which a work bound judges.
// test's `calls` (capture-setup.mjs), which a work bound judges. Probe code
// is never work (maintainer decision, 2026-11-16): a probe-only site's
// evaluation counts nothing.
let work = 0
const reach = (id) => {
work += 1
if (!sites[id].probe) work += 1
reached.add(id)
}
const outcomes = []
Expand Down
85 changes: 64 additions & 21 deletions packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions packages/oracles/scripts/e2e-shards.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,11 @@ export const shards = {
],
// 522 s in the measured run, about 221 s idle.
'e2e-5': [
// A probe branch under a generated test's work bound (about 3 s each on
// an Arclet for the fixed-seed and random units).
`fixed seed 20261105: a probe branch under a generated test's work bound`,
`random seed: a probe branch under a generated test's work bound`,
`determinism: constructed history: a probe branch under a generated test's work bound`,
`fixed seed 20261105: every outcome class, the warm and abort stops`,
`random seed: every outcome class, the warm and abort stops`,
`determinism: constructed history: every outcome class, the warm and abort stops`,
Expand Down
1 change: 1 addition & 0 deletions packages/oracles/scripts/pr-shards.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -592,6 +592,7 @@ export const partitions = {
'a package with no config',
'a declaration fault: configThrows',
'a reset check after its warm daemon was replaced',
"a probe branch under a generated test's work bound",
]),
},
},
Expand Down
Loading