Skip to content

check pledged src size at end of multithreaded frame - #4773

Open
avionicharshit-byte wants to merge 1 commit into
facebook:devfrom
avionicharshit-byte:mt-check-pledged-srcsize
Open

check pledged src size at end of multithreaded frame#4773
avionicharshit-byte wants to merge 1 commit into
facebook:devfrom
avionicharshit-byte:mt-check-pledged-srcsize

Conversation

@avionicharshit-byte

@avionicharshit-byte avionicharshit-byte commented Sep 8, 2026

Copy link
Copy Markdown

with nbWorkers >= 1 and an input spanning more than one job , a wrong pledged source size is never checked. the frame header carries the pledged size , compression returns success , and decompression fails later with Data corruption detected. both directions , too large and too small. the single-thread path has always caught this in ZSTD_compressEnd_public.

repro on current dev:

head -c 10000000 /dev/urandom > t.bin
cat t.bin | zstd -3 -T4 --stream-size=10000001 > out.zst ; echo $?   # 0
zstd -d out.zst -o /dev/null                                         # Data corruption detected
cat t.bin | zstd -3 --single-thread --stream-size=10000001 > /dev/null ; echo $?   # 11

#1095 set the rule that the pledge is validated whenever it is written into the frame header , and #1106 enforced it "whenever known" , but only in ZSTD_compressContinue_internal and ZSTD_compressEnd. the MT branch of ZSTD_compressStream2 was not part of it. this applies the same check there: at end of frame , compare pledgedSrcSizePlusOne with consumedSrcSize + 1 and return srcSize_wrong. both counters already exist , no new state.

tests: tests/cli-tests/compression/mt-stream-size.sh (over , under , exact , none with -T2 -B512K) and three cases in tests/zstreamtest.c with nbWorkers=2 and a 1 MB job size , streamed with ZSTD_e_continue so the first-call ZSTD_e_end override does not mask it. both fail on unfixed dev. make -C tests test passes.

one caveat: the error fires after the frame bytes have been handed to the caller , same as the single-thread path today.

@meta-cla meta-cla Bot added the CLA Signed label Sep 8, 2026
@avionicharshit-byte
avionicharshit-byte marked this pull request as ready for review September 8, 2026 18:32
@avionicharshit-byte

Copy link
Copy Markdown
Author

@Cyan4973 small one: the MT path never validates pledgedSrcSize at end of frame , so -T4 --stream-size=<wrong> writes a bad frame and exits 0 while --single-thread returns 11. this adds the same check ZSTD_compressEnd has. is ZSTD_compressStream2 the right place for it , or would you rather have it in zstdmt_compress.c when the last job is created ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant