Skip to content

test(oracle-gaps): judge stdout gap lines against results.json, by id - #115

Merged
KyleAMathews merged 2 commits into
mainfrom
fix/gap-line-count
Nov 16, 2026
Merged

KyleAMathews merged 2 commits into
mainfrom
fix/gap-line-count

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The "stdout has N gap lines for N±1 gaps" failures (#102, #87, and the Windows "every outcome class" shard) are not stream defects. In every run I could check, the stdout gap lines, the streamed stderr gap lines, and results.json list the same gaps: the same ids, once each. Each failure is one or two mutants that the CLI classifies differently from the model.

They were reported under the greppable-stream law because of how the e2e oracle checked that law. It compared the count of stdout gap lines with the model's gap count, and it ran before the classification law. So any classification difference showed up as a missing or extra stream line, with no site named.

This PR changes only the oracle, and the CLI is unchanged:

  • The stream law is compared against the run's own report. Each gap in results.json has exactly one stdout gap line, under its id, with its kind, line and change. stdout has no gap line that results.json does not report. The comparison is by id, not by count. For the cold run it now runs after the report-completeness law. It also applies to each completed check through the daemon (every step's check, and the check that a mid-run edit overlaps), at checkpoint incremental.
  • Model agreement stays with the classification law, unchanged. A mutant the model owes another outcome is now named at its site.
  • A small reader for the gap line (readGapLine) reads the line independently. It shares the code-span reading with readSummary, through readCodeSpan in the model module. It does not import the printer.
  • Eight new calibrations, one or more for each rejection branch, in shard e2e-5 of scripts/e2e-shards.mjs. Calibrations do not run in CI, so they add no CI cost.
    • the stdout prints one gap twice and drops another (duplicate id);
    • the stdout drops a gap line (a gap with no line; the random seed: generated finds a greppable-stream counterexample on main too: stdout has 189 gap lines for 190 gaps (seed 333199054) #102 symptom);
    • a stdout gap line gives another line than results.json, a stdout gap line shows another change, a stdout gap line names another kind (fields);
    • a stdout gap line does not read as a gap (unreadable);
    • the stdout has a gap line for a mutant that is not a gap (an id that is not a reported gap);
    • a check through the daemon drops a gap line (checkpoint incremental).
      Each one feeds a corrupted observation to the shared check, with no expected answer of its own.
  • The limits are in the oracle's own limits: which runs are judged; the loose reading of a cut, whose exact form the classify oracle's law 8 (shortened) owns; and fix(oracle-gaps): no wall-clock cut decides an outcome #99's (cause: …) suffix. That suffix does not read yet and fails loudly, so whichever of fix(oracle-gaps): no wall-clock cut decides an outcome #99 and test(oracle-gaps): judge stdout gap lines against results.json, by id #115 lands second extends readGapLine. I commented on fix(oracle-gaps): no wall-clock cut decides an outcome #99. A cut must now keep a non-empty part.

Evidence

Law: P10, one diagnostic per failure, and the maintainer's greppable-stream request (2026-11-12): every gap the run reports has exactly one gap line. The authority for the report being right is a different law: classification and report completeness (the e2e oracle's laws 4 and 7). The coordinator approved separating the two on this branch.

Reach: the comparison applies to every completed cli run and every completed check through the daemon. The fixed-seed units listed under GREEN reach it, including:

  • authored mutants (the authored line form) and a does-not-apply mutant (the L? form);
  • daemon steps and the check that a mid-run edit overlaps.

All eight calibrations found a history and were rejected under greppable stream.

RED. I wrote two hostile CLI mutants of report.mjs renderStdout, on scratch branches:

  • M1 prints the first gap twice and drops the last one.
  • M2 prints the first gap's line number plus one.

Unit fixed seed 20261105: generated, run with pnpm exec vitest --run --maxWorkers=1 --config oracle-gaps/vitest.config.mjs gap-report-oracle -t '^oracle-gaps\.report(?: | > )fixed seed 20261105: generated$':

main's oracle (4d3c8a9) this PR's oracle (0fe686a)
M1 passes (the count holds) fails: greppable stream: "the stdout has two gap lines for sut.ts:M1"
M2 passes fails: greppable stream: "the stdout's gap line for sut.ts:M1 is "… L5 …", and results.json reports it as unreached at line 4"

GREEN (at d0f6438, with the unchanged CLI; local, one Vitest worker):

  • fixed-seed units: generated, every outcome class, every outcome class, the warm and abort stops, a stale authored mutant and a reformat, an ambiguous authored mutant, two authored mutants, an edit during a check through the daemon, a test edits the helper it read, during a check through the daemon, an edit sequence over the witness module, a second daemon check of the budget witness, an edit to data a test reads through fs (sync);
  • all 8 new calibrations (8 passed). At 0fe686a, the neighbors a crash is reported as a kill, the summary omits a gap and an owned mutant is reported as unowned also passed;
  • determinism: constructed history: an edit during a check through the daemon, constructed history: a test edits the helper it read, during a check through the daemon, generated 0 (3 passed). The daemon observations now carry their sorted stdout gap lines;
  • node scripts/e2e-shards.mjs check: 5 shards partition the e2e tests.

Mutant: the RED table above. Each mutant fails the new comparison at checkpoint cli. The calibrations kill one corrupted observation for each rejection branch, including the dropped line that M1 does not reach (M1 hits the duplicate branch first).

The original failures, now attributed:

With this PR, a run that reproduces these differences fails under classification and names the site. I added no exclusion, so these failures stay red until their owners fix them.

Windows on this branch (run 43783774902):

Limits:

  • These limits are recorded in the oracle's limits: the loose cut reading (exact cut owned by classify law 8), fix(oracle-gaps): no wall-clock cut decides an outcome #99's suffix, and the runs judged (cold and daemon checks, not stopped runs or refusals).
  • readGapLine reads the id as \S+. A source path with a space would fail loudly; the grammar generates none.
  • The classify oracle's in-process stdout check (gap-report-classify-oracle, a count against its model) is unchanged. There the model is the printer's input, so it does not conflate the two laws.
  • I did not rerun the new calibrations on the parent commit, because they are new. The CLI mutants are the parent-commit RED.
  • determinism: an edit sequence through the daemon records a test-output note in one run and not the other #92, the intermittent test-output note, has a different cause. That note is about whether a test wrote output during a daemon check, not about gap lines, and none of the failing runs above differs in its notes.

Register entries touched: P10, P4 (counterexamples filed: #114), P7 (no law removed: the count check becomes an id check, and model agreement stays with classification), and P2 (the reader does not import the printer).

This pull request and its description were written by Isaac.

The e2e oracle's greppable-stream law compared the number of stdout gap
lines with the model's number of gaps, and ran before the classification
law. A mutant whose outcome differed from the model's was then reported
as "the stdout has N gap lines for N±1 gaps" under the stream law, not at
its site. #102, #87 and the Windows "every outcome class" failures were
all of this kind: in each, stdout, stderr and results.json agree.

The stream law now compares the stdout with the run's own results.json:
each reported gap has exactly one gap line under its id, with its kind,
line and change, and no other gap line is printed. It runs after the
report completeness law, so model agreement stays with classification.
Two calibrations calibrate the new comparison: one gap printed twice and
another dropped (the count holds), and one line with another line number.

Co-authored-by: Isaac <no-reply@databricks.com>
@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d0f6438

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1e07971a-36c3-426b-a2b8-806c1790b8cb
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pkg-pr-new

pkg-pr-new Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@115
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@115

commit: d0f6438

…e daemon checks too

Review of #115:
- F1: one calibration per rejection branch of the stdout gap-line check:
  a dropped line (the #102 symptom), another change, another kind, an
  unreadable line, and a line for a mutant that is not a gap, beside the
  duplicate and the wrong line number.
- F4: each completed check through the daemon (every step's check and the
  check a mid-run edit overlaps) lists its gaps on stdout against its own
  results.json, at checkpoint incremental, with a calibration there.
- F2, F3: the oracle's limits record which runs are judged, the loose
  reading of a cut (the classify oracle's law 8 owns the exact cut), and
  #99's cause suffix, which fails loudly until readGapLine reads it. A cut
  now keeps a non-empty part.

Co-authored-by: Isaac <no-reply@databricks.com>
@KyleAMathews
KyleAMathews merged commit eec7f77 into main Nov 16, 2026
48 of 50 checks passed
@KyleAMathews
KyleAMathews deleted the fix/gap-line-count branch November 16, 2026 03:54
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
…t results.json

#115 (merged as eec7f77) reads each stdout gap line against results.json and
fails loudly on any text after the change, so #99's ` (cause: <cause>)` suffix
on timeout lines did not read: on the merge, "a copy beside the runner's
pending write" fails with `does not read as a kind, a line, an id, and a
change`. #99 lands second, so it extends the reader: a timeout's line names its
cause after its change, equal to results.json's `detail`, and no other line
names one. The calibration "a stdout gap line names another cause" swaps one
timeout line's cause (backstop and no progress); the stream law rejects it.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
Takes #115's stdout gap-line law (judged by id against results.json, with
stdoutGaps observed per run) alongside #57's stream-log observations. The
logNoteOf helper and stdoutGapBreach keep their own doc blocks.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant