Repository navigation
Conversation
tirdesh
force-pushed
the
expr-invalid-utf8-valid-prefix
branch
from
November 3, 2026 16:33
f43b909 to
cee3d98
Compare
In a UTF-8 locale, find_match ran the regex on a Latin-1 mapping of the bytes whenever the left operand was not valid UTF-8. With no capture group that branch returned "0" whether or not the regex had matched, the same value as the no-match path, so it could never report a match. With a group the mapping let . match the invalid byte itself, so captures ran past it. GNU matches nothing against the invalid sequence but matches the valid characters in front of it as usual. Run the regex on the longest valid prefix instead, which gives that behaviour and lets both cases share the existing UTF-8 path. test_invalid_utf8_with_high_bytes_before_capture pinned the old mapping: it expected ".\(abc\)" to capture "abc" out of "\xFFabc", which needs . to match the invalid byte. The match is anchored, so GNU reports no match there and exits 1, the same way it does for the equivalent shape already covered by test_m6. The test now expects that in a UTF-8 locale and keeps the octet behaviour in a single byte one.
tirdesh
force-pushed
the
expr-invalid-utf8-valid-prefix
branch
from
November 3, 2026 18:43
cee3d98 to
2f2ee39
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15760
In a UTF-8 locale,
expr STRING : REGEXanswered "no match" for any pattern without a\(...\)group as soon as STRING held a byte that is not valid UTF-8, even when the pattern matched the valid characters in front of it. Soexpr "$name" : '/'printed 0 and exited 1 for a name like/etc/x\377, and the usual idiom for refusing input,expr "$x" : 'PATTERN', stopped firing when the string carried one invalid byte anywhere.find_matchran the regex on a Latin-1 mapping of the bytes in that case. With no capture group it returned"0"whether or not the regex had matched, which is the same value as the no-match path, so that branch could never report a match at all. With a group, the mapping let.match the invalid byte itself, so captures ran past it.GNU matches nothing against the invalid sequence but matches the valid characters in front of it as usual. Running the regex on the longest valid prefix gives exactly that, so the Latin-1 branch is gone and both cases now share the existing UTF-8 path:
These agree with the GNU 9.7 output quoted in the issue.
LC_ALL=Cis untouched, andtest_m6, where the invalid byte comes first and nothing may match, still passes.I added
test_m8andtest_m9next to the existing multibyte tests. Both fail on main and pass with the change. That module is#[cfg(target_os = "linux")], so it does not run on macOS; I checked the two tests by lifting the gate locally and then put it back, so they are verified but CI on Linux is the first place they run in anger.cargo fmt --checkandclippy -D warningsare clean andtest_exprpasses. Tested on macOS only.