Skip to content

cksum: reject oversized SHAKE --length instead of aborting#12881

Open
mistaste wants to merge 1 commit into
uutils:mainfrom
mistaste:fix-cksum-shake-length-panic
Open

cksum: reject oversized SHAKE --length instead of aborting#12881
mistaste wants to merge 1 commit into
uutils:mainfrom
mistaste:fix-cksum-shake-length-panic

Conversation

@mistaste

Copy link
Copy Markdown
Contributor

Summary

Fixes #12869.

cksum --algorithm shake128/shake256 with a very large --length aborts the
process while allocating the output buffer:

$ ./cksum -l=10011111117721172727 --algorithm shake128 a
memory allocation of 1251388889715146591 bytes failed
Aborted (core dumped)

SHAKE is an extendable-output function with no inherent maximum, but the digest
still has to be materialized and hex-encoded in memory, so an unbounded
--length turns into an unbounded allocation that aborts the process instead of
producing a clean error.

Change

  • Add MAX_SHAKE_OUTPUT_BITS and a ChecksumError::ShakeLengthTooBig variant in
    uucore::checksum.
  • Validate the SHAKE --length up front in cksum (next to the existing SHA/BLAKE
    length validation) and reject oversized requests with a clear error rather than
    letting the allocation abort.
  • Add test_shake_length_too_large covering both shake128 and shake256.
$ cksum -l 10011111117721172727 --algorithm shake128 a
cksum: maximum digest length for SHAKE is 4294967295 bits

This mirrors the approach used in the recent date width fix (a MAX_* bound that
turns an unbounded allocation request into a graceful error).

Note

The cap (u32::MAX bits ≈ 512 MiB) is intentionally generous — it only rejects
absurd values and doesn't affect any realistic length. Happy to adjust the bound
if you'd prefer a different limit.

Checklist

  • cargo fmt — clean
  • cargo clippy -p uu_cksum — clean
  • cargo test -p coreutils --features cksum test_shake — 27 passed

@github-actions

github-actions Bot commented Jun 14, 2026

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/cut/bounded-memory (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/retry (fails in this run but passes in the 'main' branch)
Note: The gnu test tests/env/env-signal-handler was skipped on 'main' but is now failing.

@mistaste mistaste force-pushed the fix-cksum-shake-length-panic branch from e45f148 to 2c84e73 Compare June 14, 2026 21:41
A huge `--length` with `--algorithm shake128`/`shake256` (e.g.
`--length 10011111117721172727`) made cksum abort while trying to
allocate the output buffer:

    memory allocation of 1251388889715146591 bytes failed

SHAKE is an extendable-output function with no inherent maximum, but the
digest still has to be materialized and hex-encoded in memory. Cap the
requested length at MAX_SHAKE_OUTPUT_BITS and return a clean error for
larger requests instead of letting the allocation abort the process.

Fixes uutils#12869
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(cksum): panics when using an algorithm shake128 and shake256 with an extremly large length

1 participant