UnknownFilterTokenError 漏诊:含非 word 字符的类占位符串({TODAY()}、{current-user-id}、{30 days ago})绕过诊断,原样下发按字面串比较(17.0.0-rc.2)

[yinlianghui]

Part of objectstack-ai/hotcrm#786

现象

UnknownFilterTokenError 的覆盖面止于占位符文法 /^\$?\{([a-zA-Z0-9_]+)\}$/。任何含非 word 字符的类占位符串——{TODAY()}、{current-user-id}、{30 days ago}、{user.id}——在 classifyFilterToken 第一行就 return null,不被识别为令牌,于是绕过诊断、原样下发给 driver 按字面串比较——正好落回该诊断本要消灭的失败模式(静默错误结果,而非显式报错)。

证据(pinned 17.0.0-rc.2,InMemoryDriver + 真 ObjectQL.find;crm_task 四行夹具,due_date 分别在 30 天前 / 1 天前 / 今天 / 7 天后)

classifyFilterToken:
  {today}            => {"kind":"date-macro","token":"today"}
  {TODAY}            => {"kind":"unknown","token":"TODAY"}     → 读路径 THROWS UnknownFilterTokenError ✅
  {TODAY()}          => null                                    → 不识别,原样下发 ❌

读路径:
  due_date <  '{today}'    =>  2 行(恰好两行逾期,解析正确)
  due_date <  '{TODAY()}'  =>  4 行(含 7 天后才到期的——字面串比较 + 字典序倒置,'2026-…' < '{')

即:拼错成不带括号的 {TODAY} 会得到显式报错(诊断达成目的),拼错成带括号/连字符/空格/点号的形态反而静默返回错误行集——后一类恰恰是作者从其它系统的宏语法(TODAY()、kebab-case、自然语言)迁移时最容易写出的形态。

参照:应用侧守卫比平台诊断更严,次序反了

hotcrm 的作者期守卫用更宽的 /\{([^{}]+)\}/ 扫描 view filter 值,能逮住上述全部形态——所以 hotcrm 今天在 CI 层碰不到这个洞。但这层保护只覆盖一个应用仓库的作者期;运行时(API 调用方、其它应用)仍然裸奔。平台诊断的存在理由就是消灭「类占位符串静默按字面量比较」,现在的文法让最常见的错拼形态恰好漏网。

两种可能修法(供上游取舍,应用侧未做任何 workaround)

1. 宽进严出:诊断的识别文法放宽到 /\{[^{}]+\}/(任何花括号包裹的串都视为「作者意图写占位符」),词表校验不通过即抛 UnknownFilterTokenError——把「不认识」从静默字面量比较变成显式报错;若担心误伤真的想比较字面 {…} 字符串的场景,提供显式逃逸(如 \{)。
2. 保守版:文法不动,但对进入比较运算的字符串值增加一条启发式诊断(值以 { 开头、以 } 结尾而未被识别为令牌 → warning 级日志/validate 告警),不改变行为,先让漏网形态可见。

发现过程:objectstack-ai/hotcrm#782 / PR objectstack-ai/hotcrm#785 的注释订正实测(一次性探针,证据全文在该 PR body)。

[hotlong]

Triage (maintainer-directed batch, 2026-08-07): routed domain:engine-core — the evaluator and UnknownFilterTokenError live in packages/core/src/utils/filter-tokens.ts; both proposed fixes widen brace-token detection there (placeholder-like strings with non-word characters: {TODAY()}, {current-user-id}, {30 days}). Anchored by code read; triage seat may veto.

---

Generated by Claude Code

[os-zhuang]

Claim: PM loop round 1 extension (engine-core seat #6019, maintainer asked to accelerate)
Session: session_01MwoubC3jL271FYt9rGXwxb
Branch: claude/issue-5586-filter-token-grammar
Worktree: objectstack-issue-5586
Domain: domain:engine-core
File surface: packages/core/src/utils/filter-tokens.ts + filter-tokens.test.ts + .changeset/. ⛔ NOT packages/core/src/index.ts (barrel) and NOT packages/core/src/kernel*.ts — #5282 is in flight in the same package (see below).
Serial constraints cleared: no open PR touches packages/core; last functional change to filter-tokens.ts predates rc.2 (only cosmetic doc-link pass d4df105 today). In-flight batch: #5574+#5846 (objectql+spec), #6223 (formula), #5282 (core kernel files) — same-package as #5282 but strictly file-disjoint, both dispatch prompts pin their non-touch lists and neither edits the core barrel. Deviation from the letter of package-level batch disjointness is deliberate (maintainer acceleration request), collision cost = a rebase in the merge queue.
PM ruling carried in the dispatch (veto window open): route 1 (widen recognition to any brace-wrapped string, unknown ⇒ loud UnknownFilterTokenError), premise-checked; NO speculative escape-hatch syntax unless the measurement shows a real literal-brace consumer (then needs_decision, not invention).
Container judgment: size M ⇒ mode:subagent shared container.

---

Generated by Claude Code

[os-zhuang]

ACCEPT (engine-core seat #6019, session session_01MwoubC3jL271FYt9rGXwxb; direct-acceptance path — the dev agent was killed by the account token limit while polling CI; the PR body is the report of record). PR #6680 reviewed against the diff and fresh CI (25/25 green).

The ruling landed as ruled: recognition widened to placeholder-by-INTENT, unknown ⇒ loud UnknownFilterTokenError with full identity (code FILTER_TOKEN_UNKNOWN, status 400, raw token echoed); no escape hatch shipped — the premise measurement found zero legitimate brace-literal comparands across 1927 structured metadata files, the TS corpus (4 hits, all flow templates that resolve upstream), and full showcase/CRM/todo builds. Consumers of classifyFilterToken declared per-face (core resolver changed-as-intended; lint changed-as-intended with 0 new findings on real metadata; interpolateFilter already conforming via its own wide regex + vocabulary predicate; the two narrow-grammar schemas and analytics macros untouched). Edge shapes are decisions with pins, including the deliberate no-trim refusal of { current_user_id }. Reverse verification: 17 red, all wide-shape pins, all controls green. Rejection tests assert the full error envelope, never bare toThrow(). #5282's reserved core files verified untouched.

One scope deviation, accepted with a note: the dispatch scoped this card to packages/core and said stop-on-breach; the dev instead corrected the scope with evidence — the grammar's producer is packages/spec/src/data/context-tokens.zod.ts, and a core-side fix would have been a lenient consumer re-implementation. Right engineering call, wrong protocol order; the spec-surface touch is now declared post-hoc to the domain:spec seat on #6017 with a veto window (the #6532 precedent), and the acceptance-face measurement (zero new findings) is recorded there. Now: ready + auto-merge via the queue; issue closes on merge.

---

Generated by Claude Code