Do not hand the tokenizer loader an empty special token list on Apple

[shoumikhin]

Part of #21805.

The Apple frameworks are built from the extension/llm/tokenizers submodule
pin. The tokenizers half of this bug is meta-pytorch/tokenizers#210, and it
reaches iOS users only once that pin moves forward. This PR does not move it.

Problem

ExecuTorchLLMTextRunner's two argument initializer passes @[] for special
tokens, and the three argument one accepts whatever array the caller gives it,
including an empty one. In both of those cases the runner ends up holding an
empty std::vector<std::string> and hands that to load_tokenizer.

An empty vector is not the same thing as no vector. load_tokenizer takes it at
face value and builds a Tiktoken whose BOS index 0 and EOS index 1 point past
the end of an empty list. In the 1.4.0 prebuilts that reached an abort() in
the Tiktoken constructor, so the app died with signal 6 rather than returning
an error. It happens whenever the HuggingFace loader fails first and the chain
falls through to Tiktoken, which is the path a Qwen style tokenizer.json takes
today.

There is a second, quieter bug in the same lines. The special tokens were moved
into load_tokenizer, which leaves the runner's own pointer null, not empty. A
caller that retried load after a failure therefore took the branch that means
"no list was given", and Tiktoken filled in its own 256 built in Llama 3
special tokens. The retry looks like it succeeded while quietly producing token
ids from the wrong list.

Solution

Pass nothing when the list is empty, so the tokenizer falls back to its own
defaults, and copy the list instead of moving it so a retry still has it.

This covers the empty case only, which is the one the runner itself creates. An
empty array means the caller never specified a list, because the two argument
initializer manufactures @[]. A caller that passes a list with fewer than two
entries is stating a list that cannot work, since this API fixes the BOS index at
0 and the EOS index at 1. That stays an error, and after
meta-pytorch/tokenizers#210 it is reported as an error instead of killing the
process.

Test plan

Copied load_tokenizer into a small C++ program and called it three ways
against a Tiktoken model file, on Linux x86_64, built against
meta-pytorch/tokenizers#210:

what is passed	result
an empty list, as the runner did before	bos index 0 and eos index 1 must both be below the special token count 0, the chain returns null, no abort
nothing, as the runner does now	loads, vocab 128256
a moved from pointer, as a retry saw	loads with the tokenizer's own defaults, BOS piece is the built in begin of text token, not the caller's

CI compiles the changed file clean for iOS device, iOS simulator and macOS, in
the apple (ios), apple (ios-simulator) and apple (macos) builds. The one
warning reported against it comes from a header it includes and shows up on the
unchanged files in the same target too.

Not tested: there is no run on a device, and no automated test. The repository
has no host runnable harness that exercises this binding, and the existing Swift
tests cover a successful load, not this fall through.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22047

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f8e2f17 with merge base fbd4bbf ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.