fix(fuzz): strip real-shell error lines from stderr before banned-shape check#1623
Merged
Conversation
…pe check
When a fuzz/proptest target inlines arbitrary input bytes into a shell
script, bash and ls produce error messages that quote the input verbatim:
bash: <cmd>: command not found
bash: <path>: No such file or directory
ls: cannot access '<path>': No such file or directory
These echoes can accidentally form a banned substring even when no
internal Debug formatter ran. Two real cases:
- arithmetic_fuzz / glob_fuzz: input contained `/.rustup/toolchains/`
literally; bash echoed it back. Filtered at the input layer in #1621
and the arithmetic_fuzz follow-up.
- glob_fuzz (new): input ended with `Tok"`. Bash treated `Tok:` as a
command and rendered `bash: Tok:: command not found` — the trailing
`:` chrome from the error format glued onto the input's final `:` to
form the banned parser-token shape `Tok::`. Pre-filtering the input
for `Tok::` doesn't catch this — the substring only forms after
bash's formatter runs.
Structural fix: in `assert_fuzz_invariants`, strip lines that match a
recognized real-shell error template before running the banned-shape
check. The byte-length cap (`MAX_STDERR_BYTES`) and the host-canary
check (TM-INF-013) still run on the unfiltered stderr, so flood and
env-leak regressions are still caught. The strict `assert_no_leak`
path (used by per-builtin tests) is unchanged — non-fuzz tests must
not produce shell echoes in the first place.
Recognized templates are conservative: prefix `bash: ` or `ls: ` plus
a known suffix (`: command not found`, `: No such file or directory`,
`: Is a directory`, `: Permission denied`, `: cannot execute …`, plus
the `command not found. Did you mean: ., :, [?` variant). Lines that
look similar but don't match the exact template stay in stderr, so
real Debug leaks that happen to coexist with shell errors still trip
the assertion.
Tests: 8 unit tests in `testing::tests` cover both directions —
strips known shell echoes, keeps internal panic/Debug lines, keeps
partial matches and lines from other tools. Threat-model TM-INF-022
section updated to document the carve-out.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
bashkit | ca39280 | Commit Preview URL Branch Preview URL |
May 10 2026, 09:51 AM |
6 tasks
chaliy
added a commit
that referenced
this pull request
May 16, 2026
Yesterday's `Fuzz Testing` job on main went red again. `glob_fuzz`
inputs that contain the substring `/rustc/` make uutils `ls` (via clap)
emit its four-line error template that quotes the input verbatim:
error: unexpected argument '--i{fi/rustc/fi{{RRi' found
tip: to pass '--i{fi/rustc/fi{{RRi' as a value, use '-- ...'
Usage: ls [OPTION]... [FILE]...
For more information, try '--help'.
The `error:` line itself echoes the user input, so the banned host-path
shape `/rustc/` shows up in stderr even though no internal Debug
formatter ran. The PR #1623 strip filter only recognized `bash: ...`
and `ls: cannot access '...'` lines and missed this clap chrome.
Extend `is_real_shell_error_line` with a conservative clap-chrome
matcher: lines that begin with `error: ` and contain one of clap's
fixed argument-error fragments, lines that begin with ` tip: to pass
'...' as a value, use '...'`, well-formed `Usage: ` lines, and the
`For more information, try '...'` footer. Each branch is anchored on
strings clap emits literally — they do not occur in real Debug leaks.
The strict `assert_no_leak` path (used by per-builtin tests) is
unchanged: production builtins must never produce shell echoes.
Adds 4 unit tests in `testing::tests` covering the new strip paths
and confirming that look-alike lines without the clap shape (`error:
parser failed: Tok::Ident`, `Usage: see Span { for details`) are still
preserved so real leaks coexisting with shell chrome would trip.
Updates threat-model TM-INF-022 prose to document the carve-out.
chaliy
added a commit
that referenced
this pull request
May 16, 2026
## Summary
Yesterday's `Fuzz Testing` job on `main` went red again. `glob_fuzz`
inputs that contain `/rustc/` make uutils `ls` (via clap) emit its
four-line error template that quotes the input verbatim:
```
error: unexpected argument '--i{fi/rustc/fi{{RRi' found
tip: to pass '--i{fi/rustc/fi{{RRi' as a value, use '-- --i{fi/rustc/fi{{RRi'
Usage: ls [OPTION]... [FILE]...
For more information, try '--help'.
```
The `error:` line echoes the user input, so the banned host-path shape
`/rustc/` shows up in stderr even though no internal `Debug` formatter
ran. PR #1623's strip filter only recognized `bash: …` and `ls: cannot
access '…'` lines and missed this clap chrome — same crash class,
different formatter.
## Fix
Extend `is_real_shell_error_line` in `crates/bashkit/src/testing.rs`
with a conservative clap-chrome matcher. Each branch is anchored on a
string clap emits literally and that doesn't occur in real `Debug`
leaks:
- `error: ` lines containing one of clap's fixed fragments (`unexpected
argument '`, `invalid value '`, `the argument '`, `unrecognized
subcommand '`, `the following required arguments…`, `a value is required
for '`, `equal sign is needed…`)
- ` tip: to pass '…' as a value, use '…'`
- `Usage: <prog> [OPTION]... [FILE]...` / `--help` / `--version`
- `For more information, try '…'.` footer
Keeps the strict `assert_no_leak` path unchanged: production builtins
must never produce shell echoes. `MAX_STDERR_BYTES` flood cap and the
`FUZZ_HOST_CANARY` env-leak (TM-INF-013) check still run on the
**unfiltered** stderr.
Threat-model TM-INF-022 prose updated to document the carve-out.
## Test plan
- [x] 4 new unit tests in `testing::tests` cover both directions — strip
the real failing block (`/rustc/` glued into `error: unexpected
argument` + tip + Usage + footer), strip a stand-alone `invalid value
'…'` line, keep `error: parser failed: Tok::Ident` (real internal leak
that happens to start with `error: `), keep a `Usage: see Span { for
details` look-alike that lacks the clap shape.
- [x] `cargo test -p bashkit --lib testing::` green (12 tests)
- [x] `cargo test -p bashkit --test proptest_security --all-features`
green (18 cases)
- [x] `cargo clippy -p bashkit --lib --tests -- -D warnings` clean
- [x] `cargo fmt --check` clean
- [ ] Nightly `Fuzz Testing` on this branch green (manual dispatch after
CI)
---
_Generated by [Claude
Code](https://claude.ai/code/session_015ZVmCsU7hZZEpRcqa5kqpD)_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a fuzz/proptest target inlines arbitrary input bytes into a shell script, bash and ls produce error messages that quote the input verbatim:
These echoes can accidentally form a banned substring even when no internal Debug formatter ran. Two real cases:
/.rustup/toolchains/literally; bash echoed it back. Filtered at the input layer in fix(fuzz): drop glob_fuzz inputs that contain banned debug shapes #1621 and fix(fuzz): drop arithmetic_fuzz inputs that contain banned debug shapes #1622.Tok". Bash treatedTok:as a command and renderedbash: Tok:: command not found— the trailing:chrome from bash's error format glued onto the input's final:to form the banned parser-token shapeTok::. Pre-filtering the input forTok::doesn't catch this — the substring only forms after bash's formatter runs.Fix
Structural fix in
assert_fuzz_invariants: strip lines that match a recognized real-shell error template before running the banned-shape check.MAX_STDERR_BYTES) still runs on the unfiltered stderr — flood regressions still caught.assert_no_leakpath (used by per-builtin tests) is unchanged — non-fuzz tests must not produce shell echoes in the first place.Recognized templates are conservative: prefix
bash:orls:plus a known suffix. Lines that look similar but don't match the exact template stay in stderr, so real Debug leaks that happen to coexist with shell errors still trip the assertion.Test plan
testing::testscover both directions — strip known shell echoes, keep internal panic/Debug lines, keep partial matches and lines from other tools.cargo test -p bashkit --lib testing::greencargo test -p bashkit --test proptest_security --all-featuresgreen (18 cases)cargo clippy -p bashkit --lib --tests -- -D warningscleancargo fmt --checkcleanFuzz Testingon this branch and confirmglob_fuzzjob is green