feat(ci): cap gzipped junit upload at 10 MiB and split oversized reports#1722
Merged
Conversation
Contributor
Merge Protections🟢 All 6 merge protections satisfied — ready to merge. Show 6 satisfied protections🟢 🤖 Continuous Integration
🟢 👀 Review Requirements
🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 🔎 Reviews
🟢 📕 PR description
🟢 🚦 Auto-queueWhen all merge protections are satisfied, this pull request will be queued automatically. |
There was a problem hiding this comment.
Pull request overview
This PR updates mergify ci junit-process to avoid losing the entire OTLP trace upload when a large JUnit run produces a gzipped payload that exceeds the ingest endpoint’s size limit, by introducing a 10 MiB gzipped cap and splitting oversized traces into multiple under-cap uploads.
Changes:
- Introduces
MAX_GZIPPED_UPLOAD_BYTES(10 MiB) and a newsplitmodule that partitions oversizedExportTraceServiceRequests into multiple gzipped chunks. - Adds
upload_compressedso split chunks can be POSTed without re-gzipping. - Updates the
junit-processcommand to upload multiple chunks, report dropped oversized single cases, and refine thetest_results_uploadGitHub output status.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/mergify-ci/src/testing.rs | Adds an “incompressible” test helper for reliably forcing gzip-size-based split behavior in tests. |
| crates/mergify-ci/src/junit_process/upload.rs | Adds the gzipped upload size cap constant, exposes gzip internally, and adds upload_compressed for pre-gzipped payloads. |
| crates/mergify-ci/src/junit_process/split.rs | New module implementing recursive split-by-encoded-size with exact gzip verification, plus unit tests. |
| crates/mergify-ci/src/junit_process/mod.rs | Exposes the new split API and re-exports the cap and new upload helper. |
| crates/mergify-ci/src/junit_process/command.rs | Integrates splitting + multi-upload, adds reporting for oversized cases and “nothing uploaded” status handling, and adds end-to-end tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
`mergify ci junit-process` builds every parsed JUnit report into one OTLP `ExportTraceServiceRequest`, gzips it, and POSTs it in a single request. A large report (tens of thousands of cases, or a few with huge stack traces) can gzip past what the ingest endpoint accepts, and the whole upload is lost. Cap the compressed body at a named constant `MAX_GZIPPED_UPLOAD_BYTES` (10 MiB) and split an oversized payload into several uploads that each gzip under the cap. The cap is on the *gzipped* bytes actually sent, not the raw XML or the uncompressed protobuf: gzip ratios swing wildly between a run of short case names and one of megabyte stack traces, so only the compressed size is meaningful. A normal report stays a single upload, byte-identical to before. How the split works (new `junit_process::split`): - Decompose the built request into its session / suite / case spans, then recursively halve the case list — by encoded size, so one giant case is isolated fast — verifying each candidate chunk with an actual gzip. - Each chunk is a self-contained trace: the session span, the suite spans for the cases it carries, and those cases. Every chunk keeps the original trace_id and span_ids, so the backend reassembles one trace from N uploads. - Each chunk carries the gzipped bytes it was sized against, so the upload step posts them directly instead of compressing a second time. Oversize single case (one test whose trace alone exceeds the cap gzipped): skip-and-warn. There is no upload it can fit into, so it is dropped and listed in the report under "Some test results were too large to upload". The CI verdict is unaffected — it is computed from the parsed cases, not the upload — so no test's blocking status is lost, only its telemetry. Upload semantics: a transient failure on one chunk does not abandon the rest (each chunk is a self-contained, idempotent trace, so delivering as many as possible is best), but a permanent rejection stops the loop. When nothing reaches ingest (every case oversized, or gzip fails) the run reports test_results_upload=failed rather than a false success, and dropped oversize cases emit a GitHub Actions warning naming them. Backend support: the split relies on the ingest endpoint accepting several same-test_run_id uploads and merging them additively/idempotently by (trace_id, span_id). That landed in monorepo #36926 (MRGFY-8051); without it the backend would drop every chunk after the first. This CLI change is inert until that backend is deployed. Fixes MRGFY-8050 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Change-Id: Iacd21260a73b5e249e32a88f931cd95a7a8ff57a
sileht
force-pushed
the
devs/sileht/mrgfy-8050-junit-upload-split-10mib/cap-gzipped-junit-upload-10-mib-split-oversized--acd21260
branch
from
July 17, 2026 14:23
befe5c7 to
cbd9711
Compare
Member
Author
Revision history
|
JulianMaurin
approved these changes
Jul 18, 2026
kozlek
approved these changes
Jul 20, 2026
Contributor
Merge Queue Status
This pull request spent 7 minutes 14 seconds in the queue, including 6 minutes 57 seconds running CI. Required conditions to merge
|
40 tasks
mergify
Bot
deleted the
devs/sileht/mrgfy-8050-junit-upload-split-10mib/cap-gzipped-junit-upload-10-mib-split-oversized--acd21260
branch
July 20, 2026 07:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mergify ci junit-processbuilds every parsed JUnit report into one OTLPExportTraceServiceRequest, gzips it, and POSTs it in a single request. Alarge report (tens of thousands of cases, or a few with huge stack traces)
can gzip past what the ingest endpoint accepts, and the whole upload is lost.
Cap the compressed body at a named constant
MAX_GZIPPED_UPLOAD_BYTES(10 MiB) and split an oversized payload into several uploads that each gzip
under the cap. The cap is on the gzipped bytes actually sent, not the raw
XML or the uncompressed protobuf: gzip ratios swing wildly between a run of
short case names and one of megabyte stack traces, so only the compressed
size is meaningful. A normal report stays a single upload, byte-identical to
before.
How the split works (new
junit_process::split):recursively halve the case list — by encoded size, so one giant case is
isolated fast — verifying each candidate chunk with an actual gzip.
the cases it carries, and those cases. Every chunk keeps the original
trace_id and span_ids, so the backend reassembles one trace from N uploads.
step posts them directly instead of compressing a second time.
Oversize single case (one test whose trace alone exceeds the cap gzipped):
skip-and-warn. There is no upload it can fit into, so it is dropped and
listed in the report under "Some test results were too large to upload". The
CI verdict is unaffected — it is computed from the parsed cases, not the
upload — so no test's blocking status is lost, only its telemetry.
Upload semantics: a transient failure on one chunk does not abandon the
rest (each chunk is a self-contained, idempotent trace, so delivering as
many as possible is best), but a permanent rejection stops the loop. When
nothing reaches ingest (every case oversized, or gzip fails) the run reports
test_results_upload=failed rather than a false success, and dropped oversize
cases emit a GitHub Actions warning naming them.
Backend support: the split relies on the ingest endpoint accepting several
same-test_run_id uploads and merging them additively/idempotently by
(trace_id, span_id). That landed in monorepo #36926 (MRGFY-8051); without it
the backend would drop every chunk after the first. This CLI change is inert
until that backend is deployed.
Fixes MRGFY-8050
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com