Delete libonig from the build: tokenizers on rusty_expressions - #20
Delete libonig from the build: tokenizers on rusty_expressions#20Ttimmahlax wants to merge 1 commit into
Conversation
This workspace compiled libonig C on every non-wasm build, from two independent
sources:
* our own `tokenizers 0.23` with `features = ["onig"]`, and
* `candle-core 0.11`, which pins `tokenizers 0.22` with `features = ["onig"]`
as a non-optional dependency on every non-wasm target.
Because Cargo features are additive across the graph, that reached every
downstream consumer too -- including rusty_RAG, which had explicitly selected
`unstable_wasm` with the comment "set pulls onig_sys, which is C" and got
onig_sys anyway.
`rusty_expressions` is Oniguruma remade in pure Rust. Its pre-tokenizer output
is byte-identical to the `onig` backend across 95 dumps (ByteLevel, Whitespace,
Punctuation, a negative-lookahead Split and a backreference Split) over ASCII,
CJK, Cyrillic, Greek, emoji with modifiers, NFD vs NFC, zero-width characters
and long repeats -- so tokenization does not change.
A single [patch] entry cannot cover two semver-incompatible versions, so both
are patched. Result: `cargo tree -i onig` and `-i onig_sys` are both empty.
ffai-carmenta no longer needs a different engine on wasm32 than on native --
only the getrandom plumbing, which is now a `wasm` feature separate from the
engine choice.
Upstream: huggingface/tokenizers#2368 adds the backend. Drop the 0.23 patch
when that lands; drop the 0.22 patch when candle stops hardcoding `onig`.
Verified: `cargo check --workspace` clean; onig and onig_sys absent from the
graph; rusty_expressions v0.1.3 present. Not verified here: anything needing
model downloads or a GPU.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Follow-up evidence, since the candle half of this is the part worth being sure I reproduced it in isolation — a one-file crate whose only dependency is Before After, with the only change being this PR's So the redirect removes libonig from a candle graph without forking candle, To be explicit about what the deploy branches do, since it is a real trade: The durable fix is candle not hardcoding the backend. Happy to open that |
What
This workspace compiles libonig C on every non-wasm build. This removes it.
cargo tree -i onigand-i onig_sysare both empty after this change, andcargo check --workspaceis clean.Where the C came from — two places, not one
tokenizers 0.23withfeatures = ["onig"].candle-core 0.11, which pinstokenizers 0.22withfeatures = ["onig"]as a non-optional dependency on every non-wasmtarget.
The second one explains something odd:
rusty_RAGhad already selectedunstable_wasmwith the comment "set pulls onig_sys, which is C" — andshipped
onig_sysanyway. Cargo features are additive across the graph, so anexplicit opt-out downstream cannot win against a hardcoded request upstream.
How
rusty_expressionsis Onigurumaremade in pure Rust. Same engine semantics, no C, works on wasm32.
Tokenization does not change. I built the same pre-tokenizer dumper under
both backends and diffed: 95 dumps, byte-identical (ByteLevel, Whitespace,
Punctuation, a
\s+(?!\S)negative-lookahead Split, a(\w)\1backreferenceSplit) over ASCII, CJK, Cyrillic, Greek, emoji with skin-tone modifiers, NFD vs
NFC, zero-width characters, URLs and long repeats.
A single
[patch.crates-io]entry cannot cover two semver-incompatibleversions, so both are patched.
ffai-carmentano longer needs a different engine on wasm32 than on native —only the getrandom plumbing, which is now a
wasmfeature separate from theengine choice.
Temporary bits, and how they retire
rusty-expressionsregex backend (Oniguruma semantics, pure Rust, wasm-capable) huggingface/tokenizers#2368 lands.onigbackend —worth an upstream issue there.
What I did not verify
cargo check --workspaceand the dependency graph. Not anything requiringmodel downloads or a GPU — please run the model-backed tests before merging.
Note:
ccis still in the graph, but viaaws-lc-sys←rustls, unrelated tothis change.