inject prompt-recall can never match CJK (Chinese/Japanese/Korean) gene signals: tokenizer is Unicode-aware but has no word segmentation

[chliuqi]

Environment

• @evomap/evolver 2.0.36 (npm, global install)
• Node.js v24.18.1 (full ICU), Windows 11
• EVOLVER_RECALL_MODE=enforce, local asset store, no Hub involvement
• Reproduced on a completely isolated EVOMAP_HOME — nothing else in play

Summary

A gene whose signals_match contains CJK tokens is never recalled at prompt time, even when
the prompt literally contains that exact token. This is not an encoding problem — CJK is stored,
loaded and compared correctly. It is a word-segmentation problem.

The prompt tokenizer (promptRecallHook.js:99 normalizedTokens) is:

text.toLowerCase().normalize('NFKC').match(/[\p{L}\p{N}]+/gu)

\p{L} covers CJK, so Chinese characters are not stripped — but CJK text has no spaces, and
nothing in Evolver adds word boundaries. A whole Chinese clause therefore collapses into a
single token.

literalSignalHits (promptRecallHook.js:179) then requires the gene's signal to appear as a
contiguous token sequence in the suffix trie built from the prompt
(buildPromptSequenceIndex, promptRecallHook.js:137). Since an individual Chinese word never
survives as its own token, the comparison can never succeed.

prompt 请帮我修复这个报错问题   → ["请帮我修复这个报错问题"]     ← 1 token
gene   signals_match: ["报错"]  → ["报错"]                      ← never equal

An English gene with the same shape matches fine, because ASCII word spacing produces tokens.

Minimal reproduction

repro.mjs (attached below) is self-contained — it builds a throwaway EVOMAP_HOME, writes one
Chinese-signal gene and one English-signal gene, and calls the real CLI:

node repro.mjs

PASS  please handle the zebrafish case      English control
PASS  报错，帮我看看这个情况                    Chinese, keyword bounded by fullwidth comma
PASS  出现 报错 怎么办才好                      Chinese, keyword bounded by spaces
FAIL  请帮我修复这个报错问题                     Chinese, keyword embedded in connected text
FAIL  报错日志里的异常信息                      Chinese, keyword embedded in connected text

3/5 as expected

The English case is a control: it proves the fixture, the store, the trust gate and the hook
are all working, so the Chinese failures cannot be attributed to a broken test harness. This
matters — an earlier version of this reproduction lacked the control and silently reported
"no matches" for a fixture that was never loadable in the first place
(a gene record needs the asset_id field; and note that writing an explicit
source: 'local_default' provenance row makes a local gene ineligible, while simply omitting
provenance.jsonl leaves it trusted).

Root cause, precisely

Tokenizer	/[\p{L}\p{N}]+/gu — Unicode-aware, CJK is a valid letter
Segmentation	none — no Intl.Segmenter, no CJK segmenter dependency anywhere in the package
Match	contiguous token-subsequence lookup in a suffix trie
Consequence	a CJK clause is one token; sub-phrase signals can never match

Boundaries that happen to work, because they are not \p{L}: spaces, fullwidth/halfwidth
punctuation, and script changes. So 报错，帮我看看这个情况 matches while
请帮我修复这个报错问题 does not — the difference is purely whether the keyword is delimited.

Two unrelated traps worth documenting while you are in here:

1. promptRecallHook.js:394 short-circuits any prompt with prompt.trim().length < 8, returning {}.
   Several natural Chinese prompts (报错, 报错，帮我看看 — 2 and 7 characters) fall under this,
   so the failure looks like "short prompts" rather than "CJK".
2. A gene record must carry asset_id (the store's primary key), and provenance.jsonl must use that
   same hash. A record built without it loads under asset log but is silently absent from recall.

Impact

Any non-space-delimited language — Chinese, Japanese, Korean, Thai — gets no prompt-time recall at
all. The failure is silent: the hook returns {} and looks identical to "no relevant gene found".
For users who write prompts in these languages, prompt-recall is effectively dead, and the only
working channel is the session-start injection.

CJK developers are a substantial part of this community (there is a README.zh-CN.md, and many
issues are filed in Chinese), so this is likely affecting a meaningful share of users without
being visible in logs.

Suggested fixes

Ordered by cost:

1. Segment the prompt before indexing. Intl.Segmenter is built into Node (no dependency) —
   but note it over-segments Chinese: 报错 → ["报","错"], and since
   normalizedTokens drops tokens with length < 2 (line 105), those single characters are then
   discarded, so Intl.Segmenter alone does not fix this without also relaxing the minimum
   token length for CJK. I verified this — it is a trap worth avoiding.
2. Substring fallback for CJK signals. When a signal contains CJK and the trie lookup fails,
   fall back to prompt.includes(signal). Cheap, and bounded by the existing
   PROMPT_RECALL_SIGNAL_PATTERN_CHARS limit.
3. Document the limitation and recommend ASCII/tool-name signals, plus the | alias syntax,
   for genes intended to be recalled in CJK conversations. Right now nothing tells a user that
   signals_match: ["报错"] can never fire.

Option 2 seems the smallest change that actually fixes it, and it cannot regress the ASCII path
because it only triggers when the signal contains CJK.

Workaround

Integrators can pre-augment the prompt before calling the hook: scan local genes' CJK signals, and
append any that appear literally in the prompt as a space-separated tail, which makes them
standalone tokens. Verified working on 2.0.36 (the connected-text case above then matches). It is
a workaround, not a fix — the tokenizer should handle this itself.

---

Reproduction script: repro.mjs (single file, no dependencies beyond a global evolver).

[chliuqi]

The reproduction script referenced in the report above did not make it into the issue body, so here it is inline — single file, no dependencies beyond a globally installed evolver. It writes to a throwaway EVOMAP_HOME and never touches ~/.evomap.

/**
 * Minimal reproduction: prompt-recall never matches CJK (Chinese) genes.
 *
 * Runs against an isolated temp EVOMAP_HOME — does NOT touch ~/.evomap.
 * Requires only the globally installed @evomap/evolver CLI.
 *
 *   node repro.mjs
 *
 * Expected (correct behaviour): all 5 cases match.
 * Actual (bug): only the English control and the space-delimited Chinese case match.
 */
import { spawn } from 'node:child_process'
import { mkdirSync, writeFileSync, rmSync, existsSync } from 'node:fs'
import { createHash } from 'node:crypto'
import { join, dirname } from 'node:path'
import { tmpdir } from 'node:os'

const HOME = join(tmpdir(), 'evolver-cjk-repro')
const ZH_ID = 'gene_cjk_zh_repro'
const EN_ID = 'gene_cjk_en_repro'

/** Locate the globally installed CLI without relying on PATH. */
function findCli() {
  const nodeRoot = dirname(process.execPath)
  const candidates = [
    join(nodeRoot, 'node_modules', '@evomap', 'evolver', 'bin', 'evolver.js'),
    join(nodeRoot, '..', 'lib', 'node_modules', '@evomap', 'evolver', 'bin', 'evolver.js'),
    process.env.APPDATA
      ? join(process.env.APPDATA, 'npm', 'node_modules', '@evomap', 'evolver', 'bin', 'evolver.js')
      : '',
  ].filter(Boolean)
  return candidates.find((c) => existsSync(c)) ?? 'evolver'
}
const CLI = findCli()

function gene(id, signals, summary) {
  return {
    id,
    type: 'Gene',
    summary,
    // asset_id is the store's primary key and must be present.
    asset_id: 'sha256:' + createHash('sha256').update(id).digest('hex'),
    category: 'repair',
    strategy: ['Inspect the failing step', 'Apply the smallest reversible fix'],
    validation: ['node --version'],
    constraints: { max_files: 3, forbidden_paths: ['.git', 'node_modules'] },
    preconditions: ['signals contains error-related indicators'],
    signals_match: signals,
    schema_version: '1.6.0',
  }
}

rmSync(HOME, { recursive: true, force: true })
mkdirSync(join(HOME, 'assets'), { recursive: true })
// No provenance.jsonl on purpose: an absent provenance record means trusted.
writeFileSync(
  join(HOME, 'assets', 'genes.jsonl'),
  [
    JSON.stringify(gene(ZH_ID, ['报错', '异常', '故障'], 'Chinese-signal gene')),
    JSON.stringify(gene(EN_ID, ['zebrafish', 'exception'], 'English-signal control gene')),
  ].join('\n') + '\n',
)

function runHook(prompt) {
  return new Promise((resolve) => {
    const payload = JSON.stringify({
      session_id: 'cjk-repro',
      transcript_path: '',
      cwd: process.cwd(),
      hook_event_name: 'UserPromptSubmit',
      prompt,
    })
    const child = spawn(process.execPath, [CLI, 'inject', 'prompt-recall', '--hook-stdin'], {
      env: { ...process.env, EVOMAP_HOME: HOME, EVOLVER_RECALL_MODE: 'enforce' },
      stdio: ['pipe', 'pipe', 'pipe'],
    })
    let out = ''
    child.stdout.on('data', (d) => (out += d))
    child.on('close', () => {
      const line = out.trim().split('\n').filter(Boolean).pop() ?? ''
      try {
        resolve(JSON.parse(line)?.hookSpecificOutput?.additionalContext ?? '')
      } catch {
        resolve('')
      }
    })
    child.stdin.end(payload)
  })
}

// Every prompt is >= 8 characters: promptRecallHook.js:394 short-circuits shorter ones,
// which would mask the bug being demonstrated.
const cases = [
  ['please handle the zebrafish case', 'English control (proves the harness works)', true],
  ['报错，帮我看看这个情况', 'Chinese, keyword bounded by fullwidth comma', true],
  ['出现 报错 怎么办才好', 'Chinese, keyword bounded by spaces', true],
  ['请帮我修复这个报错问题', 'Chinese, keyword embedded in connected text', true],
  ['报错日志里的异常信息', 'Chinese, keyword embedded in connected text', true],
]

let failed = 0
for (const [prompt, note, expected] of cases) {
  const text = await runHook(prompt)
  const matched = text.includes(ZH_ID) || text.includes(EN_ID)
  const ok = matched === expected
  if (!ok) failed++
  console.log(`${ok ? 'PASS' : 'FAIL'}  ${prompt}`)
  console.log(`        ${note}`)
  console.log(`        expected=${expected ? 'match' : 'no match'}  actual=${matched ? 'match' : 'no match'}`)
}

console.log(`\n${cases.length - failed}/${cases.length} as expected`)
rmSync(HOME, { recursive: true, force: true })
process.exit(failed === 0 ? 0 : 1)

Running it on 2.0.36 gives the 3/5 result quoted in the report. The English case is the control: if it ever fails, the fixture is broken and the Chinese results mean nothing.

[imxyanua]

@chliuqi I’ll take a look at this issue. Thanks for the contribution!