feat(snapshots): Compress preprod snapshot manifest with zstd#3336
Open
NicoHinderling wants to merge 2 commits into
Open
feat(snapshots): Compress preprod snapshot manifest with zstd#3336NicoHinderling wants to merge 2 commits into
NicoHinderling wants to merge 2 commits into
Conversation
Contributor
|
rbro112
approved these changes
Jun 16, 2026
The preprod snapshot manifest can be large; send it as a zstd-compressed JSON body so create_preprod_snapshot transfers fewer bytes over the wire. Adds a with_zstd_json_body request builder that serializes the payload to JSON, compresses it with zstd, and sets the Content-Encoding: zstd header.
dab1ec0 to
597d303
Compare
Comment on lines
+1365
to
+1380
| pub fn with_zstd_json_body<S: Serialize>(mut self, body: &S) -> ApiResult<Self> { | ||
| let mut body_bytes: Vec<u8> = vec![]; | ||
| serde_json::to_writer(&mut body_bytes, &body) | ||
| .map_err(|err| ApiError::with_source(ApiErrorKind::CannotSerializeAsJson, err))?; | ||
| let compressed = zstd::encode_all(body_bytes.as_slice(), 0) | ||
| .map_err(|err| ApiError::with_source(ApiErrorKind::CompressionFailed, err))?; | ||
| debug!( | ||
| "zstd json body: {} bytes compressed to {} bytes", | ||
| body_bytes.len(), | ||
| compressed.len() | ||
| ); | ||
| self.body = Some(compressed); | ||
| self.headers.append("Content-Type: application/json")?; | ||
| self.headers.append("Content-Encoding: zstd")?; | ||
| Ok(self) | ||
| } |
Member
There was a problem hiding this comment.
m: The current code unnecessarily serializes the entire body into memory uncompressed prior to compressing it.
With the following patch, we avoid the extra allocation by writing directly into a zstd encoder.
diff --git a/src/api/mod.rs b/src/api/mod.rs
index e7ca1c00..24a37418 100644
--- a/src/api/mod.rs
+++ b/src/api/mod.rs
@@ -42,6 +42,7 @@ use symbolic::common::DebugId;
use symbolic::debuginfo::ObjectKind;
use url::Url;
use uuid::Uuid;
+use zstd::Encoder as ZstdEncoder;
use crate::api::errors::{ProjectRenamedError, RetryError};
use crate::config::{Auth, Config};
@@ -1363,16 +1364,23 @@ impl ApiRequest {
}
pub fn with_zstd_json_body<S: Serialize>(mut self, body: &S) -> ApiResult<Self> {
- let mut body_bytes: Vec<u8> = vec![];
- serde_json::to_writer(&mut body_bytes, &body)
- .map_err(|err| ApiError::with_source(ApiErrorKind::CannotSerializeAsJson, err))?;
- let compressed = zstd::encode_all(body_bytes.as_slice(), 0)
+ let mut encoder = ZstdEncoder::new(Vec::new(), 0)
.map_err(|err| ApiError::with_source(ApiErrorKind::CompressionFailed, err))?;
- debug!(
- "zstd json body: {} bytes compressed to {} bytes",
- body_bytes.len(),
- compressed.len()
- );
+
+ serde_json::to_writer(&mut encoder, &body).map_err(|err| {
+ let kind = if err.is_io() {
+ ApiErrorKind::CompressionFailed
+ } else {
+ ApiErrorKind::CannotSerializeAsJson
+ };
+ ApiError::with_source(kind, err)
+ })?;
+
+ let compressed = encoder
+ .finish()
+ .map_err(|err| ApiError::with_source(ApiErrorKind::CompressionFailed, err))?;
+
+ debug!("zstd json body: {} bytes compressed", compressed.len());
self.body = Some(compressed);
self.headers.append("Content-Type: application/json")?;
self.headers.append("Content-Encoding: zstd")?;
lcian
approved these changes
Jun 17, 2026
lcian
left a comment
Member
There was a problem hiding this comment.
Stamping to unblock (I disabled auto-merge), please address or dismiss Daniel's comment.
szokeasaurusrex
approved these changes
Jun 18, 2026
szokeasaurusrex
left a comment
Member
There was a problem hiding this comment.
also stamping, the rest should be fine once my previous comment is addressed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The preprod snapshot manifest sent by
create_preprod_snapshotcan be large. This change compresses the manifest JSON body with zstd before uploading, reducing the number of bytes transferred over the wire.A new
with_zstd_json_bodyrequest builder serializes the payload to JSON, compresses it withzstd::encode_all, and setsContent-Type: application/jsonalongsideContent-Encoding: zstd(the content type describes the decoded payload; the encoding describes the wire transform).Changes
zstd = "0.13.3"dependency.ApiRequest::with_zstd_json_bodyhelper.create_preprod_snapshotthrough the zstd-compressed body path.Notes
Content-Encoding: zstdon request bodies — backend support should be confirmed deployed before this ships.