Conversation
_deserialize_bitmap read an 8-byte count and used it directly as a loop bound, so a small blob could declare a large number of bitmaps. Reading past the payload surfaced as an IndexError from the native bitmap decoder. Validate the count against the remaining bytes, since every bitmap contributes at least a 4-byte key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3979
Rationale for this change
_deserialize_bitmapread an 8-byte count from the start of the payload and used it directly as a loop bound, without relating it to how much payload actually follows. A small blob could therefore declare a very large number of bitmaps.The count cannot be trusted on its own, but it can be checked against the payload: every bitmap contributes at least a 4-byte key, so a count larger than
len(payload) // 4cannot be satisfied by the bytes present. This rejects such a payload up front with aValueErrorinstead of letting the loop run off the end of the buffer, which surfaced asIndexError: Out of bounds on buffer access (axis 0)from the native bitmap decoder.Per
SECURITY-THREAT-MODEL.mdthis is parser hardening (§7) rather than a vulnerability, so it is filed as an ordinary bug fix.Are these changes tested?
Yes.
tests/table/test_deletion_vector.pygains a case for a payload that declares more bitmaps than its bytes can hold. It fails onmainwith theIndexErrorabove and passes with this change; the existing deletion-vector tests are unaffected.Are there any user-facing changes?
A deletion vector whose declared bitmap count exceeds its payload now raises
ValueErrorinstead ofIndexError. Valid deletion vectors are unaffected.One related case is deliberately left out of this PR to keep it to a single concern: a single large key still drives the empty-bitmap fill loop up to
MAX_JAVA_SIGNEDappends, which is the same shape of problem reached through a different field. Happy to follow up on that separately if you would like it addressed.