fix(parser): support unquoted non-ASCII identifiers

[iamlinjunhong]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #26715

What this PR does / why we need it:

fix(parser): support unquoted non-ASCII identifiers

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[XuPeng-SH]

[P1] Preserve raw client bytes when a digit-leading token is promoted to an identifier.

This change explicitly supports both raw high-byte client identifiers (for example t_\xe9g) and digit-leading extended identifiers, but their combination takes a different path. In pkg/sql/parsers/dialect/mysql/scanner.go:813-820, scanNumber promotes the token to ID and then returns strings.ToLower(s.buf[start:s.Pos]). Go's Unicode lowercasing replaces an invalid UTF-8 byte such as the latin1 0xe9 with U+FFFD.

I reproduced this through the public parser at head 65f88bcf1d0388e299fd2ac85d8bf6438697fd13:

stmt, _ := ParseOne(ctx, "CREATE TABLE 1\xe9 (a INT)", 1)
tree.String(stmt, dialect.MYSQL)
// actual:   "create table 1� (a int)"
// expected: "create table 1\xe9 (a int)"

This silently addresses a different catalog object and can collapse distinct byte identifiers. Please keep the identifier bytes intact (or use byte-preserving ASCII-only folding if normalization is required), and add both scanner-level and ParseOne regressions for the pairwise case.

There is a second compatibility boundary to close: isUnquotedIdentifierLetter accepts every byte >= 0x80, so a valid UTF-8 supplementary character such as U+1F600 is accepted as an unquoted ID. MySQL explicitly limits identifiers to the BMP and prohibits U+10000 and above in quoted and unquoted identifiers: https://dev.mysql.com/doc/refman/8.0/en/identifiers.html. I confirmed CREATE TABLE 😀 (a INT) parses successfully on this head. The validation needs to be charset-aware (or otherwise distinguish the supported raw-byte contract) so the fix does not broaden the grammar beyond the compatibility contract.

[aptend]

Deep-reviewed exact head 65f88bcf1d0388e299fd2ac85d8bf6438697fd13, including the existing same-head feedback. Two compatibility blockers remain:

1. [P1] Preserve raw client bytes when a digit-leading token is promoted to ID. scanNumber now accepts a following extended identifier byte, but still returns strings.ToLower(s.buf[start:s.Pos]). Go Unicode lowercasing rewrites invalid UTF-8 bytes. I reproduced CREATE TABLE 1\xe9 (a INT) through ParseOne: deparse returns create table 1� (a int) rather than preserving byte 0xe9, so the parsed catalog name changes. Use byte-preserving ASCII-only folding (or return the original identifier bytes) and cover this combined digit-leading/raw-byte path at scanner and parser levels.

2. [P1] Do not accept supplementary Unicode code points as MySQL identifiers. isUnquotedIdentifierLetter accepts every byte >= 0x80 independently, so UTF-8 for U+10000 and above is accepted as an unquoted identifier. I independently confirmed CREATE TABLE 😀 (a INT) parses successfully. MySQL restricts identifiers to the BMP; the validation boundary must distinguish supported raw client bytes from valid UTF-8 supplementary sequences instead of broadening the grammar to all high-byte sequences.

The two focused counterexamples fail on this exact head as described.

[iamlinjunhong]

当前 exact head 2f9b4f87a7a6e639956f84e36d03f1ebc7c25442 已闭环仍挂起的两位 reviewer 反馈：

• XuPeng-SH review 4916028942、aptend review 4917079351 均基于旧 head 65f88bcf1d0388e299fd2ac85d8bf6438697fd13：当前实现用 ASCII-only folding 保留 digit-leading raw client bytes，并区分 malformed single-byte data、BMP UTF-8 与 supplementary UTF-8。
• XuPeng-SH review 4925074637 基于旧 head 6bf6b244c5d23d1606b3fbab72dc2829a01bbe72：提交 2f9b4f87a7 已在公共 scanIdentifier 入口校验首 code point，并覆盖 0x😀、0b😀、@😀、@@😀、_utf8mb4😀 的错误传播。

本次在 exact head 重新运行确定性回归，exit 0：
.agents/skills/mo-dev/scripts/mo-cgo-test -v -count=1 -timeout=120s -run '^(TestUnquotedSupplementaryIdentifierRejected|TestUnquotedSupplementaryIdentifiersRejected|TestUnquotedRawByteDirectIdentifierEntries|TestUnquotedExtendedIdentifier|TestUnquotedExtendedIdentifiers|Test_CStr)$' ./pkg/sql/parsers/dialect/mysql ./pkg/sql/parsers/tree

当前 exact-head check rollup 无 failure，Coverage 为 success；PR mergeable=MERGEABLE，没有冲突。烦请基于当前 head 重新 review。

[aptend]

Deep re-review completed on exact head 2f9b4f8. The previously reported raw-byte, supplementary-code-point, and direct scanIdentifier entry issues are closed. One new correctness blocker remains:

• [P1] Preserve identifiers whose prefix is _utf8mb4. Scanner.Scan treats every leading _utf8mb4 as a charset introducer without checking that the prefix ends at an identifier boundary. With the newly supported BMP suffix, CREATE TABLE _utf8mb4数量 (a INT) parses successfully but deparses as create table (a int): the table name is silently erased. At scanner level, _utf8mb4数量 returns an ID with an empty value instead of the original identifier. This can address or attempt to create the wrong catalog object. Only enter the charset-introducer path when _utf8mb4 is followed by the syntax required for an introducer (or at least not by another identifier character); otherwise scan the complete identifier. Add scanner and public ParseOne regressions for BMP and raw-client-byte suffixes, plus a valid _utf8mb4'...' introducer control.

[iamlinjunhong]

已按 aptend 在 exact head 2f9b4f87a7a6e639956f84e36d03f1ebc7c25442 的 re-review 修复，当前 head 为 68ec2d9c884be6733e73e8061b1fbf403ad3c1a2。

根因是旧 isCollate 只匹配 _utf8mb4 前缀，未验证后续是否真的是 string literal。现在重命名并收紧为 isCharsetIntroducer：保留原有空白兼容，但只有后续为单引号时才进入 introducer 分支；否则完整输入走普通 identifier scanner。

回归覆盖：

• scanner + public ParseOne：_utf8mb4数量、_utf8mb4\xe9A 保留完整标识符和原始客户端字节；
• scanner boundaries：精确 _utf8mb4、ASCII/数字后缀均为 ID；
• valid controls：_utf8mb4'test' 与带空白形式仍返回 STRING，公开 parser 可解析；
• _utf8mb4😀 仍在 scanner 后续 token 返回 LEX_ERROR，公开 parser 仍拒绝 supplementary code point。

验证通过（均为最终语义修改后）：

• focused red/green counterexamples；
• GOWORK=off go list/build/vet -mod=readonly for parser tree/mysql；
• .agents/skills/mo-dev/scripts/mo-cgo-test -v -count=1 -timeout=120s ./pkg/sql/parsers/tree ./pkg/sql/parsers/dialect/mysql。

新 CI run 31711715051 已触发，按任务要求不等待结果。烦请基于当前 head 重新 review。

[iamlinjunhong]

当前 exact head 68ec2d9c884be6733e73e8061b1fbf403ad3c1a2 再次核验：现有 4 条 CHANGES_REQUESTED 分别基于旧 head 65f88bcf、6bf6b244 和 2f9b4f87，暂无基于当前 head 的新 review 或 inline thread。aptend 在 2f9b4f87 指出的 _utf8mb4 前缀吞掉标识符问题，已由当前 head 将该分支收紧为仅在后续确为单引号字符串时识别 charset introducer；带 BMP/raw-byte/ASCII/数字后缀时均继续扫描完整标识符。

本次在当前 head fresh 执行确定性 CGo 定向回归：

./.agents/skills/mo-dev/scripts/mo-cgo-test -v -count=1 -timeout=120s -run <focused identifier and charset cases> ./pkg/sql/parsers/dialect/mysql ./pkg/sql/parsers/tree

结果 exit 0；scanner 与 public ParseOne 覆盖并通过：扩展标识符与 _utf8mb4 各类后缀、合法 immediate/whitespace charset introducer、supplementary Unicode 拒绝、0x/0b/@/@@ raw-byte 入口，以及 tree CStr byte preservation。当前 exact-head check rollup 无 failure，仍有部分任务运行中。烦请基于当前 head re-review 并清理旧 review 状态。

[iamlinjunhong]

已将分支从精确旧 head 68ec2d9 rebase 到当前 canonical main 0fc87ec，并用精确 lease 推送；新 head 为 fb5cfe6，无冲突。

失败的 UT run 31711715051 / job 94486146456（attempt 1）完整日志显示失败命令为 make ut UT_PARALLEL=6 UT_TIMEOUT=40，唯一失败是 pkg/tests/features.TestTableFeatures；稳定错误签名是 HAKeeper 连接 ID 分配期间 connection reset by peer / broken pipe / context deadline exceeded。该 run 执行期间合入的 #27119（fix(logservice): prevent connection ID refill convoys）直接修复了同一调用链和 2 秒超时/共享 transport reset 问题，本次 rebase 已包含该修复。未 rerun 旧 head 的 run，push 已触发新 head CI。

本地验证（rebase 后）：

• parser/tree 与 mysql parser：go list、go build、go vet 通过
• mo-cgo-test -v -count=1 -timeout=120s ./pkg/sql/parsers/tree ./pkg/sql/parsers/dialect/mysql 通过
• mo-cgo-test -race -v -count=1 -timeout=120s -run ^TestTableFeatures$ ./pkg/tests/features 通过（22.305s）
• git diff --check 通过，worktree clean

现有 CHANGES_REQUESTED 均基于旧 head；raw client bytes、digit-leading/raw-byte 组合、BMP/补充平面边界及 charset introducer 反馈已包含回归覆盖。请基于 fb5cfe6 重新 review。

[iamlinjunhong]

@XuPeng-SH @aptend 重新核对当前 exact head fb5cfe62f12fb2f84f5f43dcb829fef4be820a7f：现有 4 条 CHANGES_REQUESTED 全部基于旧 head（65f88bcf、6bf6b244、2f9b4f87），当前没有 inline thread，也没有基于 fb5cfe62 的新 review。

旧反馈在当前实现中的闭环证据：

• raw client byte / digit-leading：scanNumber 的各个 ID promotion 路径使用 byte-preserving toLowerASCII（scanner.go:789,804,840），scanner 与 public ParseOne 覆盖 1\xe9A。
• supplementary Unicode：公共 scanIdentifier 入口调用 supplementaryUTF8SequenceSizeAt（scanner.go:844），正常入口及 0x/0b/@/@@/_utf8mb4 入口均有 scanner/public-parser 拒绝回归（scanner_test.go:150、mysql_sql_test.go:5130）。
• _utf8mb4 前缀：isCharsetIntroducer 只在后续确为单引号字符串时成立（scanner.go:330）；BMP/raw-byte/ASCII/数字后缀均保留完整 ID（scanner_test.go:116-120，mysql_sql_test.go:5104-5110），合法 introducer control 也保留（scanner_test.go:137，mysql_sql_test.go:5125）。

该 exact head 在 rebase 后已 fresh 通过 parser tree/mysql 的 list、build、vet、完整 deterministic tests，以及 TestTableFeatures race 回归；当前 exact-head check rollup 无 failure，CI 仍在运行。分支 mergeable、无冲突，worktree clean。烦请基于 fb5cfe62 重新 review 并更新旧 review 状态。