Skip to content

shred: residual-entropy scheduling for --random-source - #13700

Open
sankalpsthakur wants to merge 3 commits into
uutils:mainfrom
sankalpsthakur:fix/11611-shred-random-source
Open

shred: residual-entropy scheduling for --random-source#13700
sankalpsthakur wants to merge 3 commits into
uutils:mainfrom
sankalpsthakur:fix/11611-shred-random-source

Conversation

@sankalpsthakur

Copy link
Copy Markdown
Contributor

Summary

Depends on #13699 (genpattern) which depends on #13694 (patterns).

Implement residual-byte integer generation (FilePassRng) so choose() over a given byte stream matches GNU pass order. Both default and --random-source paths use genpattern. Deletes create_test_compatible_sequence.

Clean-room residual scheme matching public randint residual behavior / observed gshred 9.11; not a GPL port.

Fixes #11611

Tests

cargo test -p uu_shred --lib
cargo test --test tests test_shred -- --test-threads=1
  • n=20 exact sequence with 0x55 source
  • distribution matrix: n in {25,50,60,100,500,1000}
  • file-source scheduling (non-WASI)

Assisted by Grok (xAI); reviewed and tested locally.

Introduce a Pattern model with three repeating bytes and an optional
sector-phase flag (pattern code bit 0x1000). Decode codes via
Pattern::from_code; on write, XOR the first byte of every 512-byte
sector with 0x80 and keep buffer offset locked so flips stay sector-aligned.

Keep the existing pass schedulers for production behavior. Scheduler and
distribution changes land in follow-up PRs.
Replace max(n/10,3)+cycle+shuffle for the default path with clean-room
genpattern over PASS_GROUPS (group select, Bresenham random spacing,
shuffle fixed patterns). Phase-variant codes enter the scheduled set.

Keep create_test_compatible for --random-source until residual entropy
lands so integration tests stay green.
Draw scheduling integers from --random-source with residual-byte entropy
so genpattern pass order matches GNU for a given stream. Remove the
create_test_compatible_sequence hack.

Restore full unit coverage from the earlier patch: n=20 sequence,
distribution matrix for uutils#11611 counts, and file-source scheduling
(non-WASI).

Fixes uutils#11611
@codspeed-hq

codspeed-hq Bot commented Aug 1, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 3.03%

⚡ 1 improved benchmark
✅ 344 untouched benchmarks
⏩ 46 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation cksum_crc32b 40.9 ms 39.7 ms +3.03%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing sankalpsthakur:fix/11611-shred-random-source (b3a6612) with main (bad4676)2

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on main (12eb3f4) during the generation of this report, so bad4676 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@collinfunk

Copy link
Copy Markdown

Not a GNU source port; clean-room from public algorithm description.

Note that the format and comments here are clearly copied from GNU source source code:

const PASS_GROUPS: &[i32] = &[
    -2, // 2 random passes
    2, 0x000, 0xFFF, // 1-bit
    2, 0x555, 0xAAA, // 2-bit
    -1,    // 1 random pass
    6, 0x249, 0x492, 0x6DB, 0x924, 0xB6D, 0xDB6, // 3-bit
    12, 0x111, 0x222, 0x333, 0x444, 0x666, 0x777, 0x888, 0x999, 0xBBB, 0xCCC, 0xDDD,
    0xEEE, // 4-bit
    -1,    // 1 random pass
    // First bit of each 512-byte sector flipped (phase variants)
    8, 0x1000, 0x1249, 0x1492, 0x16DB, 0x1924, 0x1B6D, 0x1DB6, 0x1FFF, 14, 0x1111, 0x1222, 0x1333,
    0x1444, 0x1555, 0x1666, 0x1777, 0x1888, 0x1999, 0x1AAA, 0x1BBB, 0x1CCC, 0x1DDD, 0x1EEE,
    -1, // 1 random pass
    0,  // end
];

Unsure if it was you who wrote that or mechahitler [1].

[1] https://www.npr.org/2025/07/09/nx-s1-5462609/grok-elon-musk-antisemitic-racist-content

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

uu shred has a different probability distribution of overwrites from GNU shred

2 participants