Skip to content

fix: count literal special tokens like <|endoftext|> as plain text - #129

Merged
anhnh2002 merged 1 commit into
mainfrom
fix/tiktoken-special-tokens
Oct 8, 2026
Merged

fix: count literal special tokens like <|endoftext|> as plain text#129
anhnh2002 merged 1 commit into
mainfrom
fix/tiktoken-special-tokens

fix: count literal special tokens like <|endoftext|> as plain text

28d3734
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar