feat(hfile): support ZSTD compression for metadata files - #20149
Open
linliu-code wants to merge 5 commits into
Open
linliu-code wants to merge 5 commits into
linliu-code wants to merge 5 commits into
Conversation
Exclude trailing checksum bytes from the native GZIP input stream and add a regression test for Hudi issue apache#19929.
Adapt the ZSTD codec from apache#14378 on top of the bounded decompression fix in apache#19930. Preserve the existing file-cache default and cover buffer, checksum, multi-block scan, and metadata configuration boundaries. Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #20149 +/- ##
============================================
- Coverage 80.42% 79.90% -0.53%
+ Complexity 34900 34649 -251
============================================
Files 2546 2547 +1
Lines 142929 142944 +15
Branches 17381 17381
============================================
- Hits 114948 114215 -733
- Misses 20064 20726 +662
- Partials 7917 8003 +86
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Collaborator
linliu-code
marked this pull request as ready for review
September 30, 2026 13:31
hudi-agent
reviewed
Sep 30, 2026
hudi-agent
left a comment
hudi-agent
left a comment
Contributor
There was a problem hiding this comment.
Thanks for working on this! This slice of the stack only updates the documentation for hoodie.hfile.compression.algorithm. The new text says the setting also applies to metadata table HFiles and that the native HFile reader supports NONE, GZ and ZSTD. That matches what this PR introduces: ZSTD in the native compressor factory, and the setting now carried into the metadata write config. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.
cc @yihua
wombatu-kun
approved these changes
Oct 1, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Closes #14369; part of #14367. The native HFile compressor factory does not support ZSTD, and metadata writes do not inherit the main table's HFile compression setting.
This is a tested successor to Ethan's #14378. It depends on #19930: the branch currently includes that PR's three original commits, preserving their authorship. Keep this PR as a draft until #19930 lands, then rebase to remove the prerequisite commits from this diff. #14378 remains open for its author to retire when this successor is accepted.
Summary and Changelog
readFullyfor decompression.hoodie.hfile.compression.algorithminto the metadata write configuration.Impact
Users can opt into ZSTD for MDT HFile base files through
hoodie.hfile.compression.algorithm=ZSTD. GZ remains the default. Existing NONE/GZ files remain readable. Deploy readers with ZSTD support before enabling ZSTD writes; older native readers do not support this codec.Risk Level
medium. Adds an opt-in storage codec and propagates its setting into metadata writes.
Validation on JDK 11:
hudi-iosuite: 138 tests passed, zero failures/errors/skips; Checkstyle and RAT passed.TestHoodieMetadataWriteUtilswith its reactor dependencies: 31 tests passed, zero failures/errors/skips; Checkstyle and RAT passed.Documentation Update
The existing option and its GZ default are unchanged. The configuration description documents propagation to MDT and the native reader's supported codecs. A website configuration reference update should accompany release publication.
Contributor's checklist