Skip to content

gh-160842: Read every frame of a multi-frame Zstandard zipfile member - #160843

Closed
HipHoff wants to merge 1 commit into
python:mainfrom
HipHoff:gh-160842-zipfile-zstd-frames
Closed

HipHoff wants to merge 1 commit into
python:mainfrom
HipHoff:gh-160842-zipfile-zstd-frames

Conversation

@HipHoff

@HipHoff HipHoff commented Oct 28, 2026

Copy link
Copy Markdown

A zstd stream may consist of several concatenated frames (RFC 8878, section 3.1), which multi-threaded and streaming compressors produce. ZipExtFile._read1() used the decompressor's eof as the end of the member, so it returned only the first frame and then failed the CRC check.

When the zstd decompressor reaches the end of a frame and compressed input remains (unused_data, or _compress_left > 0), _read1() now continues with a new ZstdDecompressor. The member ends when the compressed input is exhausted on a frame boundary, or when the declared file_size has been produced, as before.

Failure behaviour:

  • bytes between frames that are not a valid frame raise ZstdError;
  • a truncated last frame still produces short output and fails the CRC check with BadZipFile.

The other methods are unchanged. The repack scan in _scan_data_descriptor_no_sig_by_decompression() still stops at the first frame. For a multi-frame member written with a data descriptor and no signature, the size check there rejects the candidate, so the result is the same as before this PR.

Tests: ZstdMultipleFramesTests in test_zipfile/test_core.py covers two frames read in several chunk sizes, 49 small frames, garbage between frames, and a truncated last frame. Without the fix, three of the four fail. ./python.exe -m test test_zipfile passes on a macOS debug build of main. That build had no zlib, so the zlib-dependent tests were skipped.

The issue asks one question: should trailing bytes after the last frame, inside compress_size, be an error? This PR leaves that as it is for every method.

Disclosure: this PR was authored by Claude (Anthropic's AI assistant, via Claude Code) on behalf of @HipHoff, who takes responsibility for the change. It is a draft until he has reviewed it and signed the CLA.

https://claude.ai/code/session_016uX6RoHqnajMDyVi3LrJjs

…member

ZipExtFile._read1() treated the end of the first zstd frame as the end
of the member.  When the decompressor reaches the end of a frame and
compressed input remains, continue with a new ZstdDecompressor.  Bytes
that are not a valid frame raise ZstdError, and a truncated last frame
still fails the CRC check.

Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016uX6RoHqnajMDyVi3LrJjs
@python-cla-bot

Copy link
Copy Markdown

The following commit authors need to sign the Contributor License Agreement:

CLA not signed

@emmatyping emmatyping added the AI-guidelines-not-met The contribution does not meet our guidelines for using AI tools and is closed automatically. label Oct 28, 2026
@bedevere-app

bedevere-app Bot commented Oct 28, 2026

Copy link
Copy Markdown

Thank you for your interest in contributing to Python. This is a community-driven project, and we want it to be a welcoming place to learn and collaborate.

Contributors are responsible for reviewing and understanding what they submit, checking its claims, and being able to explain changes in their own words. This helps reviewers focus on the contribution and leaves more time to support other contributors.

We believe this contribution does not meet our guidelines for using AI tools, so we're closing it. If you think we've misunderstood your contribution, please let us know. You're welcome to explain your approach or ask questions about the guidelines.

Please read our guidelines for using AI tools.

@bedevere-app bedevere-app Bot closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-guidelines-not-met The contribution does not meet our guidelines for using AI tools and is closed automatically.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants