TextLLMRunner degenerates on an LFM2.5 export that steps correctly token by token

[john-rocky]

TextLLMRunner produces degenerate output for an LFM2.5 export that generates correctly when the same .pte is stepped token by token. Reproduces on macOS with the pip wheel and on Android through LlmModule, and does not depend on the export configuration.

Environment: executorch 1.4.0 (pip), torch 2.13.0, macOS arm64; also reproduced on a Pixel 8a with an AAR built from main.

What happens

Same .pte, same prompt, same machine, two ways of driving it:

token-by-token, explicit input_pos   "Here are three colors:\n\n1. **Red**\n2. **Green**"
TextLLMRunner                        "s::::<|im_end|>"

Shorter prompts make the shape clearer — the runner repeats one token:

prompt "Hi"                    -> "HiHiHiHiHiHiHiHiHiHiHiHiHiHiHiHi"
prompt "Name three colours."   -> "<|im_end|><|im_end|>"

Repro

from executorch.extension.llm.custom_ops import custom_ops  # noqa
from executorch.kernels import quantized  # noqa
from executorch.extension.llm.runner import TextLLMRunner, GenerationConfig

r = TextLLMRunner("lfm2_5_350m_xnnpack_8da4w.pte", "tokenizer.json")
out = []
r.generate("Hi", GenerationConfig(max_new_tokens=16, echo=False), lambda t: out.append(t))
print("".join(out))     # HiHiHiHiHiHiHiHiHiHiHiHiHiHiHiHi

The same file through a loop that steps one token at a time answers normally:

import torch
from executorch.extension.pybindings.portable_lib import _load_for_executorch

model = _load_for_executorch("lfm2_5_350m_xnnpack_8da4w.pte")
for i, tok in enumerate(prompt_tokens):
    logits = model.forward((torch.tensor([[tok]]), torch.tensor([i])))[0]
# then greedy-decode from logits, advancing input_pos each step

The .pte is public: https://huggingface.co/mlboydaisuke/LFM2.5-350M-ExecuTorch — exported with
executorch.extension.llm.export.export_llm, qmode: 8da4w, embedding_quantize: "8,0",
XNNPACK.

What it is not

I re-exported the model to remove each candidate in turn and drove every build both ways:

build	forward input	enable_dynamic_shape	token-by-token	TextLLMRunner
shipped (use_sdpa_with_kv_cache: True)	[1, 2047]	True	correct	degenerate
control (use_sdpa_with_kv_cache: False, enable_dynamic_shape: False)	[1, 1]	False	correct	degenerate

So it is not the custom SDPA path, and not dynamic versus static shape — a static [1, 1] export,
which the runner should drive sequentially, fails the same way.

It is not the tokenizer either: the same tokenizer.json encodes the prompt to the expected ids
through a standalone HFTokenizer harness on the same machine, special tokens included, and the
.pte reports get_bos_id = 1, get_eos_ids = [7], matching the tokenizer.

And it is not the runner in general — Qwen3.5-0.8B exported the same way (8da4w,
embedding_quantize: "8,0", XNNPACK) generates correctly through TextLLMRunner on the same
build, so whatever this is interacts with something specific to LFM2.5.

A guess I could not confirm

LFM2.5 is a hybrid: short-convolution blocks interleaved with attention. Convolution blocks carry
state along the sequence, so anything that drives positions differently from a plain
one-token-at-a-time walk — a warmup pass, a re-prefill, a batched first step — would corrupt that
state while leaving a pure-attention model like Qwen unharmed. I could not confirm which of those
the runner does, so I am reporting the isolation rather than a diagnosis.

Happy to run further experiments against this model, or to test a patch.

This report was written with Claude.

cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng @digantdesai