Skip to content

expr: match valid characters before an invalid UTF-8 byte - #16154

Open
tirdesh wants to merge 1 commit into
uutils:mainfrom
tirdesh:expr-invalid-utf8-valid-prefix
Open

tirdesh wants to merge 1 commit into
uutils:mainfrom
tirdesh:expr-invalid-utf8-valid-prefix

Conversation

@tirdesh

@tirdesh tirdesh commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Closes #15760

In a UTF-8 locale, expr STRING : REGEX answered "no match" for any pattern without a \(...\) group as soon as STRING held a byte that is not valid UTF-8, even when the pattern matched the valid characters in front of it. So expr "$name" : '/' printed 0 and exited 1 for a name like /etc/x\377, and the usual idiom for refusing input, expr "$x" : 'PATTERN', stopped firing when the string carried one invalid byte anywhere.

find_match ran the regex on a Latin-1 mapping of the bytes in that case. With no capture group it returned "0" whether or not the regex had matched, which is the same value as the no-match path, so that branch could never report a match at all. With a group, the mapping let . match the invalid byte itself, so captures ran past it.

GNU matches nothing against the invalid sequence but matches the valid characters in front of it as usual. Running the regex on the longest valid prefix gives exactly that, so the Latin-1 branch is gone and both cases now share the existing UTF-8 path:

expr '/etc/x\377' : '/'          1       (was 0, exit 1)
expr '../\377' : '\.\./'         3       (was 0, exit 1)
expr 'a/../\377' : '\(.*\)'      a/../   (was empty, exit 1)
expr '\377..' : '.*\(\.\.\)'     empty, exit 1  (unchanged)

These agree with the GNU 9.7 output quoted in the issue. LC_ALL=C is untouched, and test_m6, where the invalid byte comes first and nothing may match, still passes.

I added test_m8 and test_m9 next to the existing multibyte tests. Both fail on main and pass with the change. That module is #[cfg(target_os = "linux")], so it does not run on macOS; I checked the two tests by lifting the gate locally and then put it back, so they are verified but CI on Linux is the first place they run in anger. cargo fmt --check and clippy -D warnings are clean and test_expr passes. Tested on macOS only.

@tirdesh
tirdesh force-pushed the expr-invalid-utf8-valid-prefix branch from f43b909 to cee3d98 Compare November 3, 2026 16:33
In a UTF-8 locale, find_match ran the regex on a Latin-1 mapping of the
bytes whenever the left operand was not valid UTF-8. With no capture group
that branch returned "0" whether or not the regex had matched, the same
value as the no-match path, so it could never report a match. With a group
the mapping let . match the invalid byte itself, so captures ran past it.

GNU matches nothing against the invalid sequence but matches the valid
characters in front of it as usual. Run the regex on the longest valid
prefix instead, which gives that behaviour and lets both cases share the
existing UTF-8 path.

test_invalid_utf8_with_high_bytes_before_capture pinned the old mapping:
it expected ".\(abc\)" to capture "abc" out of "\xFFabc", which needs .
to match the invalid byte. The match is anchored, so GNU reports no match
there and exits 1, the same way it does for the equivalent shape already
covered by test_m6. The test now expects that in a UTF-8 locale and keeps
the octet behaviour in a single byte one.
@tirdesh
tirdesh force-pushed the expr-invalid-utf8-valid-prefix branch from cee3d98 to 2f2ee39 Compare November 3, 2026 18:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

expr: invalid UTF-8 byte hides regex matches, bypassing script checks

1 participant