Repository navigation
Conversation
…collision flood When a hash chain contains > MAX_CHAIN_LEN identical colliding blocks (such as 0x00 sparse regions colliding with 0xFF blocks at 64K/128K block sizes), walking every duplicate entry exhausts MAX_CHAIN_LEN before reaching valid matching blocks further down the chain (RsyncProject#1115). Track the strong checksum of the last failed chain entry (last_false_sum2) and unlink subsequent duplicate entries sharing the same strong checksum (unless protected by SUMFLG_SAME_OFFSET in --inplace mode) so duplicates are neither re-walked nor counted repeatedly against MAX_CHAIN_LEN. Additionally, if 4 consecutive 1-byte rolling offsets experience a weak-checksum collision where the entire chain fails to match, advance offset by a full block rather than computing a full strong checksum on every single byte.
steadytao
requested changes
Oct 17, 2026
Comment on lines
+363
to
+367
| if (!matched_block && (done_csum2 || chain_len > MAX_CHAIN_LEN)) { | ||
| if (++consecutive_false_offsets >= 4 && k > 1) { | ||
| OFF_T skip = MIN((OFF_T)(k - 1), end - 1 - offset); | ||
| if (skip > 0) { | ||
| offset += skip; |
Member
There was a problem hiding this comment.
Four misses do not prove that the next offsets cannot match. This skips valid basis matches and can substantially increase literal data. Split out the #1115 chain correction; any collision-flood policy needs explicit CPU and bandwidth bounds plus regressions with valid matches at each skip boundary. Also cover --inplace same-offset preservation when mutating the chain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR resolves the sparse-file performance regression reported in #1115 (introduced by the
MAX_CHAIN_LENcap for #217) and bounds the CPU work spent inhash_search()when rolling across regions with repeated weak-checksum collisions.Root Cause
Duplicate entries in a hash chain exhausting
MAX_CHAIN_LEN(rsync 3.5.0 is very slow and uses a lot of CPU compared to 3.4.x with large sparse files #1115):When transferring large sparse or disk image files (where the block size reaches 64 KiB or 128 KiB), long runs of identical blocks (such as
0x00sparse holes) land on the same hash chain as other constant-byte blocks (such as0xFF) due toget_checksum1()arithmetic modulo 65536. Becausebuild_hash_table()inserts blocks in sequential order, thousands of identical0x00blocks sit at the front of the chain. Whenhash_search()looks for a block sitting behind them, it walks andmemcmps every identical0x00entry until hittingMAX_CHAIN_LEN(1024), aborting before it can reach valid matching blocks further down the chain.Per-byte strong-checksum evaluation on unmatched collision runs:
When a chain fails to produce a strong-checksum match (either after exhausting
MAX_CHAIN_LENor after computingget_checksum2()),hash_search()falls back to rolling forward 1 byte at a time. Across a constant-byte or colliding region that does not exist in the basis file, this forces a fullget_checksum2()calculation at every single byte offset.Changes
last_false_sum2):When a chain entry fails the strong-checksum comparison, record its
sum2pointer inlast_false_sum2. Subsequent entries in the same chain that share the exact samesum1,len, andsum2(and do not carrySUMFLG_SAME_OFFSETin--inplacemode) are unlinked from the chain (*prev = s->sums[i].chain) without incrementingchain_len. One representative of that block remains in the chain for future matches (withwant_i/aligned_icontinuing to resolve sequential indices directly froms->sums[]), while eliminating redundantmemcmpcalls and allowinghash_search()to reach valid blocks behind long duplicate runs.consecutive_false_offsets):If 4 consecutive 1-byte rolling offsets hit a weak-checksum collision (
done_csum2orchain_len > MAX_CHAIN_LEN) without matching any block in the chain, advanceoffsetby a full block window (k - 1prior to the loop's++offset) rather than computing a strong checksum on every byte.testsuite/collision_test.py):Verifies both duplicate-chain unlinking (rsync 3.5.0 is very slow and uses a lot of CPU compared to 3.4.x with large sparse files #1115) and bounded
hash_hitson consecutive unmatched weak-checksum collisions.Note on Even Block Sizes and Repeated Sequences
Because$65536$ , large power-of-2 / even block sizes (such as $65536$ and gives all 256 constant-byte fills distinct weak checksums.
get_checksum1()accumulatess1ands2modulo65536and131072) can cause distinct constant-byte sequences to share the same 32-bit weak checksum. When rolling across unmatched repeated-byte sequences under an even block size, repeated weak collisions will trigger the consecutive-collision skip and transfer the colliding window as literal data. Users transferring files dominated by distinct constant-byte runs who want to avoid weak-checksum collisions altogether can specify an odd--block-size(e.g.,--block-size=131071), which is coprime toMany thanks to @samueloph and @dfskoll for reporting the
MAX_CHAIN_LENregression.