Skip to content

Shared fixtures: keep a tail-only test report's failing tests in the artifact summary - #833

Merged
rmcdaniel merged 2 commits into
mainfrom
rust-325-tap-failure-excerpts
Oct 28, 2026
Merged

rmcdaniel merged 2 commits into
mainfrom
rust-325-tap-failure-excerpts

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Part of #325 (CI diagnosability).

The artifact summary keeps the last 64 KiB of each test report. The comparator report is about 450 KiB, so #826's one failing test was missing from the retained summary. It was reported early, before the tail. Finding it took a local reproduction of the exact test list.

Change. When a test summary (*-tests.txt) is cut to its tail, artifact-summary.mjs also writes <report>.failures.txt.

  • It contains every TAP not ok line with its indented diagnostic block, up to the block's ....
  • It stays within the same per-file and total limits, and is recorded with derived_from and the failure count.
  • A whole report or a passing report gets no excerpt.

Checked: ServerParityArtifactSummaryTest passes 8/8. The new case puts a failure ahead of a 4,000-test passing tail, and also checks a passing report and a small failing report.

🤖 Generated with Claude Code

The artifact summary keeps a test report's last 64 KiB. The comparator
report is about 450 KiB, so #826's one failing test (reported early) was
missing from its retained summary, and the failure needed a local
reproduction to find.

When a test summary is cut to its tail, the summary also writes
<report>.failures.txt: every TAP `not ok` line with its diagnostic block,
within the same per-file and total limits, recorded with derived_from and
the failure count. A whole report or a passing one gets no excerpt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

rmcdaniel commented Oct 28, 2026 •

Copy link
Copy Markdown
Member Author

Resolved at b2b6e75: independent follow-up verification. The original 53a61f5 findings below are preserved.

Review at 53a61f5ade27cbd57cddae3acb225d53a81bcb1f found two regressions to address before merge. The existing artifact-summary suite independently passes 8/8.

  1. The new failure scan loses the bounded-input behavior. Line 97 rereads the entire report with readFileSync(..., 'utf8'), then splits the whole string and accumulates all failure blocks before truncating the output. The 64 KiB/file and 2 MiB/content limits therefore do not bound the scanner's memory use. I compared a zero-allocation sparse 3 GiB report: the baseline retains only 65,536 bytes; the new path attempts the whole-file read. A controlled read guard intercepted that request, so this check did not allocate/read 3 GiB or establish an actual OOM. Please scan with bounded chunks/line/block buffers and bounded excerpt accumulation, with a regression proving large reports preserve those bounds.

  2. Derived outputs make omitted-file accounting negative. With a single comparator-tests.txt containing one early TAP failure followed by more than 64 KiB of passing lines, the result is scanned_files=1, two retained entries (tail plus derived excerpt), and omitted_files=-1 at line 114. No source file was omitted. Count retained source files separately from derived entries and cover this one-source case.

Both checks used isolated offline Node 22.23.3. No owner-branch changes or Rust execution.

Cloud's review of #833 (comment 6461500906) found two problems:
- the failure scan read the whole report with readFileSync and split it,
  so a large report defeated the summary's memory bounds;
- a derived excerpt counted as a retained source file, so one failing
  report gave scanned_files 1, retained 2, omitted_files -1.

scanTapFailures now reads the report in 64 KiB chunks through a UTF-8
StringDecoder, one line of at most 8 KiB at a time. It keeps at most the
excerpt limit and 200 lines per block, and still counts failures past the
limit. omitted_files counts source files only. New test: a 256 MiB sparse
report with an early failure gives the exact excerpt and scanned 1 /
retained 2 / omitted 0. A 1 GiB sparse report scans at 77 MiB peak RSS.
Artifact summary tests pass 9/9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

rmcdaniel commented Oct 28, 2026 •

Copy link
Copy Markdown
Member Author

Independent follow-up at b2b6e7531919c6512163d6a06ba00ff8318c5e63 resolves both findings in 6461500906.

  • Artifact-summary suite: 9/9 passed, including the 256 MiB sparse-report regression.
  • Repeated the whole-file-read guard against a 32 MiB sparse input: collection succeeds, retains 64 KiB, and never requests a whole-file read. Reviewed reads in 64 KiB chunks, bounded pending lines/excerpt accumulation, UTF-8 decoding and O_NOFOLLOW.
  • Original one-source accounting case now reports scanned_files=1, two retained entries and omitted_files=0.
  • Additional controls preserve a UTF-8 character split across the 64 KiB chunk boundary and count all 7,000 failures after the excerpt fills, retaining 65,533 bytes. That separate 32 MiB control process peaked at 55,964 KiB RSS; this is a scoped measurement, not an arbitrary-input memory benchmark.

All checks ran with cached Node 22.23.3 in isolated offline containers. No Rust execution or owner-branch changes. No further finding in this correction; normal hosted checks still apply.

Hosted PR qualification independently verified at 2026-10-28 06:10 UTC, same b2b6e753: all 57 head checks are terminal, 56 successful / 1 skipped. The skipped check is Candidate cancellation and prepared-local source qualification. Shared Server fixtures40639629907, PHP40639629800, Public Boundary40639629810 and Sonar40639629835 all passed that exact head. The PR is OPEN/ready with a clean mergeable state. Normal merge-group qualification remains required.

Combined qualification and source delivery are complete: queue/merge commit dc46dc842988b54fab2c8fe7b513d8e39a24a244 passed Shared40643095197, PHP40643095237 and Boundary40643095158. GitHub confirms #833 merged at 2026-10-28 06:40:27 UTC. Both original review findings are corrected in the merged source.

@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit dc46dc8 Oct 28, 2026
57 checks passed
@rmcdaniel
rmcdaniel deleted the rust-325-tap-failure-excerpts branch October 28, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant