Skip to content

match: Fix hash_search() duplicate-chain regression (#1115) and collision flood - #1132

Open
seks99x wants to merge 1 commit into
RsyncProject:masterfrom
seks99x:fix-hash-search-collisions
Open

seks99x wants to merge 1 commit into
RsyncProject:masterfrom
seks99x:fix-hash-search-collisions

Conversation

@seks99x

@seks99x seks99x commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR resolves the sparse-file performance regression reported in #1115 (introduced by the MAX_CHAIN_LEN cap for #217) and bounds the CPU work spent in hash_search() when rolling across regions with repeated weak-checksum collisions.

Root Cause

  1. Duplicate entries in a hash chain exhausting MAX_CHAIN_LEN (rsync 3.5.0 is very slow and uses a lot of CPU compared to 3.4.x with large sparse files #1115):
    When transferring large sparse or disk image files (where the block size reaches 64 KiB or 128 KiB), long runs of identical blocks (such as 0x00 sparse holes) land on the same hash chain as other constant-byte blocks (such as 0xFF) due to get_checksum1() arithmetic modulo 65536. Because build_hash_table() inserts blocks in sequential order, thousands of identical 0x00 blocks sit at the front of the chain. When hash_search() looks for a block sitting behind them, it walks and memcmps every identical 0x00 entry until hitting MAX_CHAIN_LEN (1024), aborting before it can reach valid matching blocks further down the chain.

  2. Per-byte strong-checksum evaluation on unmatched collision runs:
    When a chain fails to produce a strong-checksum match (either after exhausting MAX_CHAIN_LEN or after computing get_checksum2()), hash_search() falls back to rolling forward 1 byte at a time. Across a constant-byte or colliding region that does not exist in the basis file, this forces a full get_checksum2() calculation at every single byte offset.

Changes

  • Unlink duplicate false-alarm nodes (last_false_sum2):
    When a chain entry fails the strong-checksum comparison, record its sum2 pointer in last_false_sum2. Subsequent entries in the same chain that share the exact same sum1, len, and sum2 (and do not carry SUMFLG_SAME_OFFSET in --inplace mode) are unlinked from the chain (*prev = s->sums[i].chain) without incrementing chain_len. One representative of that block remains in the chain for future matches (with want_i / aligned_i continuing to resolve sequential indices directly from s->sums[]), while eliminating redundant memcmp calls and allowing hash_search() to reach valid blocks behind long duplicate runs.
  • Skip block window after 4 consecutive false-alarm offsets (consecutive_false_offsets):
    If 4 consecutive 1-byte rolling offsets hit a weak-checksum collision (done_csum2 or chain_len > MAX_CHAIN_LEN) without matching any block in the chain, advance offset by a full block window (k - 1 prior to the loop's ++offset) rather than computing a strong checksum on every byte.
  • Regression test (testsuite/collision_test.py):
    Verifies both duplicate-chain unlinking (rsync 3.5.0 is very slow and uses a lot of CPU compared to 3.4.x with large sparse files #1115) and bounded hash_hits on consecutive unmatched weak-checksum collisions.

Note on Even Block Sizes and Repeated Sequences

Because get_checksum1() accumulates s1 and s2 modulo $65536$, large power-of-2 / even block sizes (such as 65536 and 131072) can cause distinct constant-byte sequences to share the same 32-bit weak checksum. When rolling across unmatched repeated-byte sequences under an even block size, repeated weak collisions will trigger the consecutive-collision skip and transfer the colliding window as literal data. Users transferring files dominated by distinct constant-byte runs who want to avoid weak-checksum collisions altogether can specify an odd --block-size (e.g., --block-size=131071), which is coprime to $65536$ and gives all 256 constant-byte fills distinct weak checksums.

Many thanks to @samueloph and @dfskoll for reporting the MAX_CHAIN_LEN regression.

…collision flood

When a hash chain contains > MAX_CHAIN_LEN identical colliding blocks
(such as 0x00 sparse regions colliding with 0xFF blocks at 64K/128K
block sizes), walking every duplicate entry exhausts MAX_CHAIN_LEN before
reaching valid matching blocks further down the chain (RsyncProject#1115).

Track the strong checksum of the last failed chain entry (last_false_sum2)
and unlink subsequent duplicate entries sharing the same strong checksum
(unless protected by SUMFLG_SAME_OFFSET in --inplace mode) so duplicates
are neither re-walked nor counted repeatedly against MAX_CHAIN_LEN.

Additionally, if 4 consecutive 1-byte rolling offsets experience a
weak-checksum collision where the entire chain fails to match, advance
offset by a full block rather than computing a full strong checksum on
every single byte.
Comment thread match.c
Comment on lines +363 to +367
if (!matched_block && (done_csum2 || chain_len > MAX_CHAIN_LEN)) {
if (++consecutive_false_offsets >= 4 && k > 1) {
OFF_T skip = MIN((OFF_T)(k - 1), end - 1 - offset);
if (skip > 0) {
offset += skip;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four misses do not prove that the next offsets cannot match. This skips valid basis matches and can substantially increase literal data. Split out the #1115 chain correction; any collision-flood policy needs explicit CPU and bandwidth bounds plus regressions with valid matches at each skip boundary. Also cover --inplace same-offset preservation when mutating the chain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants