Repository navigation
fix(oracle-gaps): probe code is never work (#102) - #121
Merged
Merged
Conversation
… e2e model The model's interpreter counted no site evaluation in probe code: a guard declaration, a probe statement, and a `report()` call with its body. The capture counts every site evaluation as work, probe code included (the work law in the Limits), so a mutant that forces a probe branch (an elseIf's if-true, say) did more work under a generated test's bound of 1 than the model predicted, and the CLI's `work bound` kill was owed as a random-only-kill (#102). The interpreter now counts a guard declaration's `===` and `!==`, each probe statement's three sites, and each `report()` call, `reportTwice`, and `reportAndPush` with their bodies. The disarmed-pause history keeps its drawn tests (its inputs are no longer pinned to avoid the bound), and the Limits drop the recorded model gap. Co-authored-by: Isaac <no-reply@databricks.com>
…k bound kills a probe branch No fixed-seed generated history reaches "killed by a work bound in a generated function" (0 of the 52 fixed-seed units), so it was only counted. A constructed history, appended at the end of the list, now enters it: a generated elseIf whose if-true calls report() where the unmutated run pushes, under a passing test of input 3 (bound 1). The state moves from measured to required. On main's model this history fails classification (f0:32:if-true: killed, owed not-judged). Co-authored-by: Isaac <no-reply@databricks.com>
Main's constructed histories (#103) come first; this branch's history stays appended after them. The generated work-bound state is reached only by its own constructed history, so `every outcome class` no longer names it (it is listed beside the disarmed pause's states). Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: 0453d59 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
commit: |
… histories
Maintainer decision, 2026-11-16 ("Probe code is never work"): site
evaluations inside probe-only code count zero toward a test's work and
its bound. This replaces the branch's earlier rule, which counted probe
code as work.
- Model: main's interpreter, plus one rule: a probe-only site's evaluation
counts no work.
- The constructed history now owes that a mutant whose only extra work is
probe code (a generated elseIf's if-true, under a generated test's bound
of 1) is not killed by the bound. It is built on the one-owner base,
and it enters its own state ("a probe branch's work under a generated
test's bound judged nothing"), which is required. "killed by a work bound
in a generated function" goes back to measured.
- `disarmedOnly` is now `constructedOnly`, with the reason each state is
left out of `every outcome class`.
- The Limits record the rule and the states' reach.
Co-authored-by: Isaac <no-reply@databricks.com>
…l at a probe-only site
Maintainer decision, 2026-11-16 ("Probe code is never work"): site
evaluations inside probe-only code count zero toward a test's work and
its work bound. The check passes the probe-only sites (the mutants that
schemata.mjs marks probeGuard) in each run's state, and the capture's
`hit` counts a call only at another site. A mutant whose only extra work
is probe code, such as a generated elseIf's if-true that calls report(),
is no longer killed by a work bound (#102).
Co-authored-by: Isaac <no-reply@databricks.com>
…geset Co-authored-by: Isaac <no-reply@databricks.com>
Main's constructed history from #61 ("a batch whose run rejects") comes first; this branch's history stays appended after it. The Limits take main's text with this branch's work-bound sentence. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
I merged main fac63df (#61) at 2008c99. #61's constructed history "a batch whose run rejects" comes first, and this branch's history stays appended after it. Reruns on the merged head:
|
…mment, and M6 in the Limits - N1: the disarmed-pause history's comment no longer says probe code is work (v21.2 R53, M-23). - N2: the probe-branch state counts only a production mutant (`!m.probe`) under a bound test whose route judges work (R19a), so a probe-only mutant cannot enter the reach count. - The Limits name M6 (a counter that skips every site on a probe's line), which survives because work is observed only as a ratio and a rise (O1), beside M7, which is killed. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Nov 16, 2026
This was referenced Nov 16, 2026
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
…stic-cuts #121's and #128's constructed histories come before #99's. The rejected batch's `outBreak` and #128's `foreign` ride on the history's shared run options beside #99's hook env. capture-setup's `hit` keeps #121's probe-site skip beside #99's cut rows. Co-authored-by: Isaac <no-reply@databricks.com>
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This settles #102 under the maintainer's decision of 2026-11-16, "Probe code is never work" (M). Site evaluations inside probe-only code count zero toward a test's work and its work bound.
In #102 the CLI killed a generated elseIf's
if-truebywork bound, and the model owedrandom-only-kill. The mutant's only extra work was thereport()call that the forced branch runs. Under the M entry the model was right and the CLI was wrong. So this PR changes the CLI's counter and keeps main's model, plus one explicit rule.The branch first counted probe code as work in the model. The M entry replaces that, and those model changes are reverted.
Changes
gap-report-model-oracle.mjs): main's interpreter, plus one rule: a probe-only site's evaluation counts no work. It is 4 lines; the rest of the diff is comments. Main's interpreter already counted no probe body, but it did count a probe-onlyif's condition (probeIf,probeBlock). The new rule makes the law explicit instead of a coincidence.check.mjspasses the probe-only sites (the mutants thatschemata.mjsmarksprobeGuard) in each run's state, asprobeSites, the same way it passesseedVariable.capture-setup.mjshitcounts a call only at another site.gap-report-oracle.property.test.mjs):a probe branch under a generated test's work bound, is appended last. It is built ononeOwnerHistory, so a random base cannot remove the state (F4). It runs a generated elseIf under a passing test of input 3, which declares bound 1. It owesnot-judged, not a work-bound kill.a probe branch's work under a generated test's bound judged nothing, is required. It is recorded only for a production mutant (!m.probe) whose function has a test that declares a bound on a route that judges work (R19a). So a probe-only mutant cannot enter the reach count (review N2).killed by a work bound in a generated functionstays measured.disarmedOnlyis renamedconstructedOnly, and each member now has a non-circular reason (F3). The disarmed pause and the replaced warm daemon need a field that only their own histories set. The probe-branch state needs a generated test whose input declares a bound, andevery outcome classgives its generated tests input 1.e2e-5, Linux fixed-seed shard 3 (the rest), and Windows shard 10.Evidence
Law: Probe code is never work (M, 2026-11-16; v21.2 M-23, R53 and I19), under the work law (P3, R19 and R19a). One counter feeds every consumer (O1, G7), with the probe exclusion only (R31).
Reach of
a probe branch's work under a generated test's bound judged nothing:RED (oracle change at 3260a50, unchanged CLI; local, one worker). Each fails under
classification:src/sut.ts:f0:31:if-true#0: killed, owed not-judged;f0:35:if-true#0: killed, owed random-only-kill;f0:34:if-true#0: killed, owed random-only-kill.GREEN (CLI change at f9d3df6):
random seed: generatedand the new history's random unit pass on seeds 1 to 8.Mutant. M5 counts probe-only sites as work again (
if (inTest) calls += 1). It fails the new history withclassification: src/sut.ts:f0:31:if-true#0: killed, owed not-judged.Fixed-seed shift (review F2 and N3; v21.2 R54, K18, R42 and K19). Appending a constructed history changes the fixed seed's generated draws. The campaign draws from one
fc.oneofwhose weights depend on the number of constructed histories. Constructed histories keep their bases, because each is drawn at its index. The history-fingerprint canary does not cover this mixing. A separate framework PR is to make generated draws independent of the constructed list.The
fixed seed 20261105: generatedunit, measured against the current base, main fac63df:Main reaches 17 states; this branch reaches 21.
Lost (6):
a capture row read while the worker wrote it;a daemon pause that did not engage;authored mutant;edit dieMidCheck changed the cold result;incremental check reused evidence;two checks started the daemon at once.Each is still reached by its own constructed unit: all three Linux fixed-seed shards pass.
Gained (10):
a config exported as an async function ran,a config in vitest.config.ts ran;a hang cut off for idleness,a package with a native addon ran in child processes;a setup error,a setup error beside a timeout under one budget;stability rerun,stale authored mutant;the CLI ran from a path with a space and #,timeout.A base replay of a head counterexample now runs another history (R42, K19). random seed: generated (seed 4): a timeout classification counterexample, on main too #120's seed and path stop replaying after this merges, so I recorded both of its counterexample histories in random seed: generated (seed 4): a timeout classification counterexample, on main too #120 itself.
Windows (#114):
w_fastturnhas no probe code, so the rule does not touch it. The model oweskilled(work bound) on every platform, so #114 stays a CLI/platform issue. On the earlier head ba370f9, the Windows shard 10 job, which holds the new units, passed (run 43812825710). Windows 2/10 and 4/10 fail the same way they do on main.Limits:
This pull request and its description were written by Isaac.