Repository navigation
test(oracle-gaps): judge stdout gap lines against results.json, by id - #115
Merged
Merged
Conversation
The e2e oracle's greppable-stream law compared the number of stdout gap lines with the model's number of gaps, and ran before the classification law. A mutant whose outcome differed from the model's was then reported as "the stdout has N gap lines for N±1 gaps" under the stream law, not at its site. #102, #87 and the Windows "every outcome class" failures were all of this kind: in each, stdout, stderr and results.json agree. The stream law now compares the stdout with the run's own results.json: each reported gap has exactly one gap line under its id, with its kind, line and change, and no other gap line is printed. It runs after the report completeness law, so model agreement stays with classification. Two calibrations calibrate the new comparison: one gap printed twice and another dropped (the count holds), and one line with another line number. Co-authored-by: Isaac <no-reply@databricks.com>
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
commit: |
This was referenced Nov 16, 2026
…e daemon checks too Review of #115: - F1: one calibration per rejection branch of the stdout gap-line check: a dropped line (the #102 symptom), another change, another kind, an unreadable line, and a line for a mutant that is not a gap, beside the duplicate and the wrong line number. - F4: each completed check through the daemon (every step's check and the check a mid-run edit overlaps) lists its gaps on stdout against its own results.json, at checkpoint incremental, with a calibration there. - F2, F3: the oracle's limits record which runs are judged, the loose reading of a cut (the classify oracle's law 8 owns the exact cut), and #99's cause suffix, which fails loudly until readGapLine reads it. A cut now keeps a non-empty part. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Nov 16, 2026
This was referenced Nov 16, 2026
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
…t results.json #115 (merged as eec7f77) reads each stdout gap line against results.json and fails loudly on any text after the change, so #99's ` (cause: <cause>)` suffix on timeout lines did not read: on the merge, "a copy beside the runner's pending write" fails with `does not read as a kind, a line, an id, and a change`. #99 lands second, so it extends the reader: a timeout's line names its cause after its change, equal to results.json's `detail`, and no other line names one. The calibration "a stdout gap line names another cause" swaps one timeout line's cause (backstop and no progress); the stream law rejects it. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The "stdout has N gap lines for N±1 gaps" failures (#102, #87, and the Windows "every outcome class" shard) are not stream defects. In every run I could check, the stdout gap lines, the streamed stderr gap lines, and results.json list the same gaps: the same ids, once each. Each failure is one or two mutants that the CLI classifies differently from the model.
They were reported under the greppable-stream law because of how the e2e oracle checked that law. It compared the count of stdout gap lines with the model's gap count, and it ran before the classification law. So any classification difference showed up as a missing or extra stream line, with no site named.
This PR changes only the oracle, and the CLI is unchanged:
incremental.readGapLine) reads the line independently. It shares the code-span reading withreadSummary, throughreadCodeSpanin the model module. It does not import the printer.e2e-5ofscripts/e2e-shards.mjs. Calibrations do not run in CI, so they add no CI cost.the stdout prints one gap twice and drops another(duplicate id);the stdout drops a gap line(a gap with no line; the random seed: generated finds a greppable-stream counterexample on main too: stdout has 189 gap lines for 190 gaps (seed 333199054) #102 symptom);a stdout gap line gives another line than results.json,a stdout gap line shows another change,a stdout gap line names another kind(fields);a stdout gap line does not read as a gap(unreadable);the stdout has a gap line for a mutant that is not a gap(an id that is not a reported gap);a check through the daemon drops a gap line(checkpointincremental).Each one feeds a corrupted observation to the shared
check, with no expected answer of its own.limits: which runs are judged; the loose reading of a cut, whose exact form the classify oracle's law 8 (shortened) owns; and fix(oracle-gaps): no wall-clock cut decides an outcome #99's(cause: …)suffix. That suffix does not read yet and fails loudly, so whichever of fix(oracle-gaps): no wall-clock cut decides an outcome #99 and test(oracle-gaps): judge stdout gap lines against results.json, by id #115 lands second extendsreadGapLine. I commented on fix(oracle-gaps): no wall-clock cut decides an outcome #99. A cut must now keep a non-empty part.Evidence
Law: P10, one diagnostic per failure, and the maintainer's greppable-stream request (2026-11-12): every gap the run reports has exactly one gap line. The authority for the report being right is a different law: classification and report completeness (the e2e oracle's laws 4 and 7). The coordinator approved separating the two on this branch.
Reach: the comparison applies to every completed
clirun and every completed check through the daemon. The fixed-seed units listed under GREEN reach it, including:does-not-applymutant (theL?form);All eight calibrations found a history and were rejected under
greppable stream.RED. I wrote two hostile CLI mutants of
report.mjsrenderStdout, on scratch branches:Unit
fixed seed 20261105: generated, run withpnpm exec vitest --run --maxWorkers=1 --config oracle-gaps/vitest.config.mjs gap-report-oracle -t '^oracle-gaps\.report(?: | > )fixed seed 20261105: generated$':greppable stream: "the stdout has two gap lines for sut.ts:M1"greppable stream: "the stdout's gap line for sut.ts:M1 is "… L5 …", and results.json reports it as unreached at line 4"GREEN (at d0f6438, with the unchanged CLI; local, one Vitest worker):
generated,every outcome class,every outcome class, the warm and abort stops,a stale authored mutant and a reformat,an ambiguous authored mutant,two authored mutants,an edit during a check through the daemon,a test edits the helper it read, during a check through the daemon,an edit sequence over the witness module,a second daemon check of the budget witness,an edit to data a test reads through fs (sync);a crash is reported as a kill,the summary omits a gapandan owned mutant is reported as unownedalso passed;constructed history: an edit during a check through the daemon,constructed history: a test edits the helper it read, during a check through the daemon,generated 0(3 passed). The daemon observations now carry their sorted stdout gap lines;node scripts/e2e-shards.mjs check: 5 shards partition the e2e tests.Mutant: the RED table above. Each mutant fails the new comparison at checkpoint
cli. The calibrations kill one corrupted observation for each rejection branch, including the dropped line that M1 does not reach (M1 hits the duplicate branch first).The original failures, now attributed:
greppable stream: the stdout has 188 gap lines for 189 gaps.classification: src/sut.ts:f0:35:if-true#0: killed, owed random-only-kill. The CLI killed the mutant by lawwork bound.C0.run:373:if-true(laws: ["work bound"]). There stdout, stderr and results.json each hold 189 gaps.sut.ts:M112,w_fastturn:if-false. On macOS at 4d3c8a9 it is killed by work bound; on win32 it is not-observed. Filed as Windows e2e: w_fastturn's work-bound kill is not-observed on win32 (sut.ts:M91, every outcome class) #114, with the trace.sut.ts:M91,w_fastturn:if-false. It is killed by work bound off Windows and not-observed on win32. Same class, Windows e2e: w_fastturn's work-bound kill is not-observed on win32 (sut.ts:M91, every outcome class) #114.sut.ts:M163andsut.ts:M165,w_hangkill. They are killed by law L off Windows; on win32 they aretimeout (cause: backstop). This is fix(oracle-gaps): no wall-clock cut decides an outcome #99's Windows idleness problem, and fix(oracle-gaps): no wall-clock cut decides an outcome #99 owns it.With this PR, a run that reproduces these differences fails under
classificationand names the site. I added no exclusion, so these failures stay red until their owners fix them.Windows on this branch (run 43783774902):
every outcome class"did not enter a hang cut off for idleness". That is main's Windows idle gap, owned by fix(oracle-gaps): no wall-clock cut decides an outcome #99.stopped(SIGABRT), which is fix(oracle-gaps): supervise the check on win32; log each CLI run in the e2e driver #79.Limits:
limits: the loose cut reading (exact cut owned by classify law 8), fix(oracle-gaps): no wall-clock cut decides an outcome #99's suffix, and the runs judged (cold and daemon checks, not stopped runs or refusals).readGapLinereads the id as\S+. A source path with a space would fail loudly; the grammar generates none.Register entries touched: P10, P4 (counterexamples filed: #114), P7 (no law removed: the count check becomes an id check, and model agreement stays with classification), and P2 (the reader does not import the printer).
This pull request and its description were written by Isaac.