Skip to content

fix(train): drop unknown tokens during training and inference - #359

Open
serhiizghama wants to merge 2 commits into
MinishLab:mainfrom
serhiizghama:fix/train-drop-unk-tokens
Open

fix(train): drop unknown tokens during training and inference#359
serhiizghama wants to merge 2 commits into
MinishLab:mainfrom
serhiizghama:fix/train-drop-unk-tokens

test: cover unk stripping against StaticModel tokenization

97beaae
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 warning
Run tests with pytest (ubuntu-latest, 3.10)
succeeded Sep 3, 2026 in 8m 31s