fix(llama): distinguish zephyr/tinyllama from phi chat template

[dundysm]

Background

Fixes #1271.

families/llama/tests/manifests/tinyllama-1.1b.json e2e fails because the native
runtime's first decode step predicts EOS immediately (token_ids: [2], empty
text). HF reference greedy generation for the same prompt and chat template
returns a real answer (token_ids: [12415, 2410, 29901, 3681, 2], text
Capital: Paris).

root cause (confirmed via template + detection/render comparison, not GPU
logits): llama_detect_chat_template_format treated any jinja containing
<|user|> / <|assistant|> as phi. TinyLlama-1.1B-Chat (and Zephyr) use
those role tags but end turns with eos_token (</s>), not Phi's literal
<|end|>. apply_phi() therefore injected <|end|> separators the model was
never trained on, which matches the observed out-of-distribution first-step EOS.

unrelated to open PR #1269 (eos_token_id list); that work does not touch
chat-template detection/rendering.

Exit Criteria

• TinyLlama / Zephyr jinja (role tags + eos_token, no <|end|>) detects as
  zephyr, not phi.
• Phi jinja (role tags + <|end|>) still detects as phi.
• llama_apply_chat_template("zephyr", ...) renders
  <|user|>\n{prompt}</s>\n<|assistant|>\n (matches HF apply_chat_template
  with trim_blocks/lstrip_blocks).
• non-goal: does not change eos_token_id list handling (fix(llama): preserve full eos_token_id list through to the runtime #1269).
• non-goal of this PR alone: full A40 GPU e2e (no local NVIDIA GPU here); ask
  maintainers/CI GPU to confirm the e2e harness.

Implementation

in families/llama/runtime/chat_templates.cpp:

• require <|end|> (in addition to role tags) to classify phi.
• otherwise classify role-tag templates without <|end|> as zephyr.
• add apply_zephyr() that uses </s> turn ends instead of <|end|>.

regression coverage in families/llama/tests/cpp/test_llama_chat_template.cpp
(detect + apply + e2e prompt string; asserts no <|end|> in zephyr render).

affected: llama family chat-template detection/apply only. no public API, ABI,
or bundle format change (format string is internal; bundles still store the
same chat_template.jinja section).

Change categories

• Model or runtime behavior
• Public API
• ABI
• Bundle or artifact format
• Dependencies
• Documentation only
• CI or developer tooling

Validation

Commands and Results

g++ -std=c++17 -Wall -Wextra -Wpedantic -I. \
  -o /tmp/test_llama_chat_template \
  families/llama/tests/cpp/test_llama_chat_template.cpp \
  families/llama/runtime/chat_templates.cpp
/tmp/test_llama_chat_template
# All llama chat_template tests passed.

also verified detect/apply on the real TinyLlama and Phi-3 jinja strings:

• TinyLlama tokenizer_config chat_template -> zephyr
• Phi-3 (<|end|> present) -> phi
• zephyr apply for the issue prompt:
  <|user|>\nWhat is the capital of France? Answer in one word.</s>\n<|assistant|>\n
• phi apply (broken path TinyLlama was hitting):
  <|user|>\nWhat is the capital of France? Answer in one word.<|end|>\n<|assistant|>\n

HF-equivalent jinja render (jinja2 trim_blocks=True, lstrip_blocks=True,
same as transformers) matches the zephyr apply string above.

Hardware, Environment, and Revisions

• tested head: this branch (fix/tinyllama-chat-template-format)
• CPU-only box; no NVIDIA GPU / no nvidia-smi; cannot run A40 e2e or
  trtmc run / pytest families/llama/tests/test_e2e.py --e2e-model=tinyllama-1.1b
• TinyLlama template taken from TinyLlama/TinyLlama-1.1B-Chat-v1.0 tokenizer_config
  on HF main; Phi comparison from microsoft/Phi-3-mini-4k-instruct

Not Run / Remaining Gaps

• full GPU e2e for tinyllama-1.1b (build bundle + native vs HF reference) was
  not run locally. please confirm on community GPU CI / a machine with CUDA
  that pytest families/llama/tests/test_e2e.py --e2e-model=tinyllama-1.1b
  now produces non-empty text aligned with the HF reference cited in Bug: TinyLlama-1.1B-Chat's e2e case produces empty output — first decode step predicts EOS immediately #1271.
• other llama e2e manifests not re-run (chat-template path for phi-tagged models
  unchanged aside from requiring <|end|> for phi classification).

Contributor Self-Review

• I have completed a self-review of this change.

Notes For Future Readers

• suggested review order: detection change in chat_templates.cpp, then
  apply_zephyr, then new unit tests.
• if a future model uses <|user|> with a non-</s> eos, detection may need a
  richer signal; TinyLlama/Zephyr eos is </s> and matches this apply path.

Risk level

• Low
• Medium
• High

Risk rationale: surgical heuristic tightening plus one new format path; phi
behavior preserved when <|end|> is present; covered by non-GPU unit tests.

[dundysm]

@yifeif-nv @chaofengw-nv friendly ping for review when you have a moment; this is a small llama chat-template detection fix for #1271 (tinyllama misclassified as phi). no local gpu here so the unit tests are cpu-only; would appreciate a gpu ci / a40 pass on pytest families/llama/tests/test_e2e.py --e2e-model=tinyllama-1.1b if convenient.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4f04c574-4e97-4100-a96d-f9a4a4ed351c

📥 Commits

Reviewing files that changed from the base of the PR and between 798573f and 30c99c9.

📒 Files selected for processing (1)

• families/llama/tests/cpp/test_llama_chat_template.cpp

🚧 Files skipped from review as they are similar to previous changes (1)

• families/llama/tests/cpp/test_llama_chat_template.cpp

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Summary

Summary

• Fixes Llama chat-template detection for TinyLlama and Zephyr models.
• Classifies role-tag templates as Phi only when they contain <|end|>.
• Adds Zephyr rendering with </s> turn separators.
• Adds regression tests for detection and prompt rendering.
• Public APIs, ABI, bundle format, and eos_token_id handling remain unchanged.

Architecture impact

• Family-owned files: Changes are limited to families/llama/runtime and families/llama/tests/cpp.
• Shared surfaces: No shared implementation or public surface changes.
• Dependency direction: Tests use existing Llama runtime chat-template behavior. No new dependency direction is introduced.
• Affected consumers: Llama runtime consumers that detect and apply chat templates, especially TinyLlama and Zephyr models.
• Unresolved blast-radius question: GPU end-to-end validation must confirm non-empty TinyLlama output and alignment with the Hugging Face reference. CPU unit tests pass.

HUMAN REVIEW REQUIRED — Complete GPU validation before merge.

Walkthrough

The Llama runtime now detects Zephyr templates separately from Phi templates and renders Zephyr turns with </s> separators. Tests cover TinyLlama detection, prompt rendering, assistant prompting, and test registration.

Changes

Zephyr chat-template support

Layer / File(s)	Summary
Detection and rendering families/llama/runtime/chat_templates.cpp	The runtime classifies role-tagged templates containing `<
Detection and rendering tests families/llama/tests/cpp/test_llama_chat_template.cpp	Tests verify Zephyr classification, </s> output, assistant prompting, and exclusion of injected `<

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 30c99

The reviewed change adds the intended Zephyr template detection and regression coverage without a supported unresolved merge-blocking risk.

🚥 Pre-merge checks | ✅ 7 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Linked Issues check	❓ Inconclusive	For [#1271], the implementation now classifies role-tag templates with literal <|end|> as phi and other role-tag templates as zephyr. The zephyr formatter emits </s> before the assistant tag…	Provide the GPU E2E result for families/llama/tests/test_e2e.py::test_e2e[tinyllama-1.1b], including the non-empty output and reference comparison.

✅ Passed checks (7 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description is complete and follows the repository template. It documents the background, exit criteria, implementation, change category, validation results, environment limits, remaining gaps, se…
Title check	✅ Passed	The title clearly identifies the primary change: correcting Llama chat-template classification for Zephyr and TinyLlama models instead of treating them as Phi.
Out of Scope Changes check	✅ Passed	The diff changes only Llama chat-template detection and rendering in families/llama/runtime/chat_templates.cpp and adds focused regression tests in `families/llama/tests/cpp/test_llama_chat_template…
Family Ownership Boundary	✅ Passed	PASS. The authoritative diff contains only families/llama/runtime/chat_templates.cpp and families/llama/tests/cpp/test_llama_chat_template.cpp. The changed files include only the Llama header and …
Shared Semantic Neutrality	✅ Passed	PASS. The authoritative pull-request diff changes only families/llama/runtime/chat_templates.cpp and families/llama/tests/cpp/test_llama_chat_template.cpp. The custom check excludes runtime and C+…
Benchmark Validation Integrity	✅ Passed	PASS. The authoritative diff changes only Llama chat-template detection/rendering and adds deterministic string assertions. It does not change a benchmark, performance measurement, workload, metric, g…
Shared Change Blast Radius	✅ Passed	PASS: The pull request does not alter a shared surface. The authoritative diff contains only families/llama/runtime/chat_templates.cpp and families/llama/tests/cpp/test_llama_chat_template.cpp. Th…

Full details: Linked Issues check

Explanation

For [#1271], the implementation now classifies role-tag templates with literal <|end|> as phi and other role-tag templates as zephyr. The zephyr formatter emits </s> before the assistant tag. Regression tests cover TinyLlama detection, rendering, and the expected issue prompt without <|end|>. The reported TinyLlama GPU E2E case and its non-empty output against the Hugging Face reference remain unvalidated, so the fix cannot be confirmed at runtime.

---

_{Comment @coderabbitai help to get the list of available commands.}

[yifeif-nv]

@yifeif-nv @chaofengw-nv friendly ping for review when you have a moment; this is a small llama chat-template detection fix for #1271 (tinyllama misclassified as phi). no local gpu here so the unit tests are cpu-only; would appreciate a gpu ci / a40 pass on pytest families/llama/tests/test_e2e.py --e2e-model=tinyllama-1.1b if convenient.

Hello, thanks for the contribution. I will take a look today and trigger the CI for you if it all looks good. Sorry I missed that PR earlier!

[yifeif-nv]

@yifeif-nv @chaofengw-nv friendly ping for review when you have a moment; this is a small llama chat-template detection fix for #1271 (tinyllama misclassified as phi). no local gpu here so the unit tests are cpu-only; would appreciate a gpu ci / a40 pass on pytest families/llama/tests/test_e2e.py --e2e-model=tinyllama-1.1b if convenient.

Looks like a legitimate fix. I'm triggering the CI for you! Thanks for the contribution, and looking forward to future contributions as well!

[dundysm]

@yifeif-nv thanks for taking a look and kicking ci; really appreciate it.

pushed a small follow-up for the source-quality fail (30c99c9, clang-format on the zephyr/tinyllama string literal wrapping). community cpu is green on the new head now; still no local gpu here so would love the a40 / e2e pass when you get a chance. happy to iterate on anything else you spot.