Skip to content

fix(fm_index): store blocks in Java's codec envelope, not the SST one - #826

Open
jackylee-ch wants to merge 2 commits into
apache:mainfrom
jackylee-ch:fix/fm-index-codec-envelope
Open

jackylee-ch wants to merge 2 commits into
apache:mainfrom
jackylee-ch:fix/fm-index-codec-envelope

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

write_block sent FM blocks through compress_block, which prepends an
uncompressed-size varint. The varint belongs to the SST and bitmap block
formats, not the codec: Java's FMIndexFile#writeBlock stores
BlockCompressor output verbatim with storedLength = compressedLength, and
decodeStoredBlock decompresses it directly. So paimon-rust cannot read a
Java-written FM global index, failing with Compressed block size does not match its metadata, and Java cannot read ours. Both default to lz4, and ZSTD is
affected too since the varint precedes the codec branch.

Fix: add a codec-only pair, compress_codec_block and
decompress_codec_block, and point FM at it. compress_block keeps the varint
for SST and bitmap, where Java does prepend it.

The V1 goldens store blocks uncompressed, so no test covered this; the new one
asserts the stored bytes against Java's envelope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant