Skip to content

feat(snapshots): Compress preprod snapshot manifest with zstd#3336

Open
NicoHinderling wants to merge 2 commits into
masterfrom
nico/feat/zstd-snapshot-manifest
Open

feat(snapshots): Compress preprod snapshot manifest with zstd#3336
NicoHinderling wants to merge 2 commits into
masterfrom
nico/feat/zstd-snapshot-manifest

Conversation

@NicoHinderling

Copy link
Copy Markdown
Contributor

Summary

The preprod snapshot manifest sent by create_preprod_snapshot can be large. This change compresses the manifest JSON body with zstd before uploading, reducing the number of bytes transferred over the wire.

A new with_zstd_json_body request builder serializes the payload to JSON, compresses it with zstd::encode_all, and sets Content-Type: application/json alongside Content-Encoding: zstd (the content type describes the decoded payload; the encoding describes the wire transform).

Changes

  • Add zstd = "0.13.3" dependency.
  • Add ApiRequest::with_zstd_json_body helper.
  • Route create_preprod_snapshot through the zstd-compressed body path.

Notes

  • Requires the receiving endpoint to decode Content-Encoding: zstd on request bodies — backend support should be confirmed deployed before this ships.

@NicoHinderling NicoHinderling requested review from a team and szokeasaurusrex as code owners June 16, 2026 22:26
@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor
Messages
📖 Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against 597d303

The preprod snapshot manifest can be large; send it as a zstd-compressed
JSON body so create_preprod_snapshot transfers fewer bytes over the wire.

Adds a with_zstd_json_body request builder that serializes the payload to
JSON, compresses it with zstd, and sets the Content-Encoding: zstd header.
@NicoHinderling NicoHinderling force-pushed the nico/feat/zstd-snapshot-manifest branch from dab1ec0 to 597d303 Compare June 16, 2026 22:53
@NicoHinderling NicoHinderling enabled auto-merge (squash) June 16, 2026 23:03
@NicoHinderling NicoHinderling disabled auto-merge June 16, 2026 23:03
@NicoHinderling NicoHinderling enabled auto-merge (squash) June 16, 2026 23:04
Comment thread src/api/mod.rs
Comment on lines +1365 to +1380
pub fn with_zstd_json_body<S: Serialize>(mut self, body: &S) -> ApiResult<Self> {
let mut body_bytes: Vec<u8> = vec![];
serde_json::to_writer(&mut body_bytes, &body)
.map_err(|err| ApiError::with_source(ApiErrorKind::CannotSerializeAsJson, err))?;
let compressed = zstd::encode_all(body_bytes.as_slice(), 0)
.map_err(|err| ApiError::with_source(ApiErrorKind::CompressionFailed, err))?;
debug!(
"zstd json body: {} bytes compressed to {} bytes",
body_bytes.len(),
compressed.len()
);
self.body = Some(compressed);
self.headers.append("Content-Type: application/json")?;
self.headers.append("Content-Encoding: zstd")?;
Ok(self)
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

m: The current code unnecessarily serializes the entire body into memory uncompressed prior to compressing it.

With the following patch, we avoid the extra allocation by writing directly into a zstd encoder.

diff --git a/src/api/mod.rs b/src/api/mod.rs
index e7ca1c00..24a37418 100644
--- a/src/api/mod.rs
+++ b/src/api/mod.rs
@@ -42,6 +42,7 @@ use symbolic::common::DebugId;
 use symbolic::debuginfo::ObjectKind;
 use url::Url;
 use uuid::Uuid;
+use zstd::Encoder as ZstdEncoder;
 
 use crate::api::errors::{ProjectRenamedError, RetryError};
 use crate::config::{Auth, Config};
@@ -1363,16 +1364,23 @@ impl ApiRequest {
     }
 
     pub fn with_zstd_json_body<S: Serialize>(mut self, body: &S) -> ApiResult<Self> {
-        let mut body_bytes: Vec<u8> = vec![];
-        serde_json::to_writer(&mut body_bytes, &body)
-            .map_err(|err| ApiError::with_source(ApiErrorKind::CannotSerializeAsJson, err))?;
-        let compressed = zstd::encode_all(body_bytes.as_slice(), 0)
+        let mut encoder = ZstdEncoder::new(Vec::new(), 0)
             .map_err(|err| ApiError::with_source(ApiErrorKind::CompressionFailed, err))?;
-        debug!(
-            "zstd json body: {} bytes compressed to {} bytes",
-            body_bytes.len(),
-            compressed.len()
-        );
+
+        serde_json::to_writer(&mut encoder, &body).map_err(|err| {
+            let kind = if err.is_io() {
+                ApiErrorKind::CompressionFailed
+            } else {
+                ApiErrorKind::CannotSerializeAsJson
+            };
+            ApiError::with_source(kind, err)
+        })?;
+
+        let compressed = encoder
+            .finish()
+            .map_err(|err| ApiError::with_source(ApiErrorKind::CompressionFailed, err))?;
+
+        debug!("zstd json body: {} bytes compressed", compressed.len());
         self.body = Some(compressed);
         self.headers.append("Content-Type: application/json")?;
         self.headers.append("Content-Encoding: zstd")?;

@lcian lcian disabled auto-merge June 17, 2026 13:08

@lcian lcian left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stamping to unblock (I disabled auto-merge), please address or dismiss Daniel's comment.

@szokeasaurusrex szokeasaurusrex left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also stamping, the rest should be fine once my previous comment is addressed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants