Repository navigation
Fix \w not matching UTF-8 continuation bytes - #1001
Merged
Leonard Hecker (lhecker) merged 2 commits intoNov 2, 2026
Merged
Conversation
AkiPoh (AkiPoh)
force-pushed
the
fix-syntax-highlighting-bug
branch
from
November 1, 2026 04:56
bffa0e4 to
fa98ad2
Compare
\w previously only included the leading bytes of multibyte sequences
(0xC2-0xF4), so a match ended in the middle of a non-ASCII character.
Because \w+ is a single non-backtracking charset instruction, rules
like java's /(\w+)\s*\(/ (for method definition highlighting) then
failed outright, leaving non-ASCII identifiers unhighlighted.
Hence this commit adds the continuation bytes (0x80-0xBF) to the \w
matching in regex.rs, to fix this issue.
See example that previously went unhighlighted in Java:
```java
// "merkkienMäärä" was previously unhighlighted due to containing
// non-ASCII characters. After this commit this too is highlighted.
public static int merkkienMäärä (String merkkijono) {
}
// In comparison "numberOfCharacters" got previously highlighted
// because it only included solely ASCII-characters.
public static int numberOfCharacters (String text) {
}
```
AkiPoh (AkiPoh)
force-pushed
the
fix-syntax-highlighting-bug
branch
from
November 1, 2026 04:57
fa98ad2 to
d3468ac
Compare
Dustin L. Howett (DHowett)
left a comment
Dustin L. Howett (DHowett)
left a comment
Member
There was a problem hiding this comment.
Leonard Hecker (@lhecker) huh, this seems like a reasonable catch
Leonard Hecker (lhecker)
approved these changes
Nov 2, 2026
Leonard Hecker (lhecker)
left a comment
Leonard Hecker (lhecker)
left a comment
Member
There was a problem hiding this comment.
Well, someone had a brain fart there. Someone is me.
The comments are a bit LLM-sloppy. Approved with some final touches. Thanks for the fix!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
\w previously only included the leading bytes of multibyte sequences (0xC2-0xF4), so a match ended in the middle of a non-ASCII character. Because \w+ is a single non-backtracking charset instruction, rules like java's /(\w+)\s*(/ (for method definition highlighting) then failed outright, leaving non-ASCII identifiers unhighlighted.
Hence this commit adds the continuation bytes (0x80-0xBF) to the \w matching in regex.rs, to fix this issue.
See example that previously went unhighlighted in Java:
Disclaimers and potential shortcomings:
naïve—ok(with em dash in the middle) would match as single identifier with the addition of the (0x80-0xBF) range. But this should most often be fine, probably.There's probably a better way to fix this, but I hope this PR is at least of some value to someone, at least in highlighting this issue -- pun intended.