Skip to content

Delete libonig from the build: tokenizers on rusty_expressions - #20

Open
Ttimmahlax wants to merge 1 commit into
masterfrom
drop-libonig-for-rusty-expressions
Open

Delete libonig from the build: tokenizers on rusty_expressions#20
Ttimmahlax wants to merge 1 commit into
masterfrom
drop-libonig-for-rusty-expressions

Conversation

@Ttimmahlax

Copy link
Copy Markdown
Collaborator

What

This workspace compiles libonig C on every non-wasm build. This removes it.

cargo tree -i onig and -i onig_sys are both empty after this change, and
cargo check --workspace is clean.

Where the C came from — two places, not one

  1. Our own tokenizers 0.23 with features = ["onig"].
  2. candle-core 0.11, which pins tokenizers 0.22 with
    features = ["onig"] as a non-optional dependency on every non-wasm
    target.

The second one explains something odd: rusty_RAG had already selected
unstable_wasm with the comment "set pulls onig_sys, which is C" — and
shipped onig_sys anyway. Cargo features are additive across the graph, so an
explicit opt-out downstream cannot win against a hardcoded request upstream.

How

rusty_expressions is Oniguruma
remade in pure Rust. Same engine semantics, no C, works on wasm32.

Tokenization does not change. I built the same pre-tokenizer dumper under
both backends and diffed: 95 dumps, byte-identical (ByteLevel, Whitespace,
Punctuation, a \s+(?!\S) negative-lookahead Split, a (\w)\1 backreference
Split) over ASCII, CJK, Cyrillic, Greek, emoji with skin-tone modifiers, NFD vs
NFC, zero-width characters, URLs and long repeats.

A single [patch.crates-io] entry cannot cover two semver-incompatible
versions, so both are patched.

ffai-carmenta no longer needs a different engine on wasm32 than on native —
only the getrandom plumbing, which is now a wasm feature separate from the
engine choice.

Temporary bits, and how they retire

What I did not verify

cargo check --workspace and the dependency graph. Not anything requiring
model downloads or a GPU — please run the model-backed tests before merging.

Note: cc is still in the graph, but via aws-lc-sysrustls, unrelated to
this change.

This workspace compiled libonig C on every non-wasm build, from two independent
sources:

  * our own `tokenizers 0.23` with `features = ["onig"]`, and
  * `candle-core 0.11`, which pins `tokenizers 0.22` with `features = ["onig"]`
    as a non-optional dependency on every non-wasm target.

Because Cargo features are additive across the graph, that reached every
downstream consumer too -- including rusty_RAG, which had explicitly selected
`unstable_wasm` with the comment "set pulls onig_sys, which is C" and got
onig_sys anyway.

`rusty_expressions` is Oniguruma remade in pure Rust. Its pre-tokenizer output
is byte-identical to the `onig` backend across 95 dumps (ByteLevel, Whitespace,
Punctuation, a negative-lookahead Split and a backreference Split) over ASCII,
CJK, Cyrillic, Greek, emoji with modifiers, NFD vs NFC, zero-width characters
and long repeats -- so tokenization does not change.

A single [patch] entry cannot cover two semver-incompatible versions, so both
are patched. Result: `cargo tree -i onig` and `-i onig_sys` are both empty.

ffai-carmenta no longer needs a different engine on wasm32 than on native --
only the getrandom plumbing, which is now a `wasm` feature separate from the
engine choice.

Upstream: huggingface/tokenizers#2368 adds the backend. Drop the 0.23 patch
when that lands; drop the 0.22 patch when candle stops hardcoding `onig`.

Verified: `cargo check --workspace` clean; onig and onig_sys absent from the
graph; rusty_expressions v0.1.3 present. Not verified here: anything needing
model downloads or a GPU.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Ttimmahlax

Copy link
Copy Markdown
Collaborator Author

Follow-up evidence, since the candle half of this is the part worth being sure
about.

I reproduced it in isolation — a one-file crate whose only dependency is
candle-core = "0.11":

Before

onig v6.5.3
└── tokenizers v0.22.2
    └── candle-core v0.11.0
onig_sys v69.9.3

After, with the only change being this PR's [patch.crates-io] block:

onig      -> did not match any packages
onig_sys  -> did not match any packages
rusty_expressions v0.1.3
candle-core still builds and runs

So the redirect removes libonig from a candle graph without forking candle,
and candle keeps working.

To be explicit about what the deploy branches do, since it is a real trade:
they redirect the onig feature to the pure-Rust backend and drop the
optional onig dependency entirely. That means features = ["onig"] no longer
means onig in this graph — a deliberate lie, deploy-only, never proposed
upstream. It is the only way a downstream patch can win against a hardcoded
features = ["onig"] without forking candle.

The durable fix is candle not hardcoding the backend. Happy to open that
upstream issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants