[p-diogo]

Summary

Harrier (harrier-oss-v1-270m) pools the last token, so every input must end with <eos>. The ONNX repo's tokenizer_config.json has no add_eos_token, and transformers' Gemma tokenizer rebuilds its post-processor from that flag, so the Python client tokenized "hello world" as [2, 23391, 1902]. The vector was effectively the last word's hidden state. transformers.js reads tokenizer.json directly and adds <eos>, so the TS clients were fine.

• Loader passes add_eos_token=True to AutoTokenizer.from_pretrained.
• get_embedding appends <eos> itself if a transformers version ignores the flag (overwrites the last token at the 512 limit).
• Docstring explains why.

Evidence

Measured on a 13-fact vault, 12 queries (offline) and on a throwaway staging account (end-to-end):

	Right fact ranked first
No <eos> (shipped)	4/12 ("vegetarian" put "User is vegetarian." 11th of 13)
With <eos> (q4, quantized, fp32 all agree)	12/12
Staging e2e, 5 phrasings of the QA's diet question	0/5 → 5/5

• Python ↔ TS embedding parity: cosine 0.40 → 1.0000.
• Mixed rollout is safe: old (no-eos) facts with fixed queries still rank first 9/12; every old/new mix beats today.
• Staging write → recall-visible latency is 7–14 s, so the rc7 QA's "stored fact not found" (internal chore(cleanup #650 phase 3): DEFAULT_CHAIN_ID 84532 -> 100 (client + app) #656) was ranking, not index lag.

Also affects near-duplicate and contradiction checks, which use the same embeddings.

Tests

• tests/test_embedding_eos.py (new, runs in CI, no model): appends EOS when dropped, leaves it when present, full-length input, loader passes the flag. 3 of 4 fail on the old code.
• tests/test_embedding.py::TestRetrievalQuality (model-dependent, skipped in CI): passed on a machine with the model.
• Full suite: 2742 passed, 26 skipped, 1 xfailed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bd3G1BkVMFSXkcdAQ7nFax