Skip to content

fix(downloader): offload download_repo to a dedicated thread when a Tokio runtime is active#668

Merged
inureyes merged 1 commit into
mainfrom
fix/issue-463-serve-nested-runtime
Jul 5, 2026
Merged

fix(downloader): offload download_repo to a dedicated thread when a Tokio runtime is active#668
inureyes merged 1 commit into
mainfrom
fix/issue-463-serve-nested-runtime

Conversation

@inureyes

@inureyes inureyes commented Jul 5, 2026

Copy link
Copy Markdown
Member

Summary

mlxcel serve -m <repo-id> aborted with Tokio's "Cannot start a runtime from within a runtime" whenever the model had to be auto-downloaded, because download_repo builds its own runtime and calls block_on, which is forbidden on a thread already driving a runtime (run_serve is async). The synchronous mlxcel run path never hit it.

Fix

The download_repo entry point now detects an active runtime via tokio::runtime::Handle::try_current() and runs the unchanged blocking body (download_repo_blocking) on a dedicated OS thread (joined, panics propagated); synchronous callers keep the direct call. This is the issue's "runtime-aware download entry" option: one change makes every current and future caller safe — mlxcel serve, mlxcel-server's #[tokio::main] download path near startup, and the CLI download / run verbs — with identical progress output (the bars write to the same stderr).

Validation

End to end with the release binary and an empty store:

mlxcel serve -m mlx-community/SmolLM2-135M-Instruct-8bit --port 18463
  [mlxcel download] 8 files queued (filtered from 10 total siblings)
  [mlxcel download] complete: downloaded=8 cached=0 total_size=141.02 MiB dest=~/.cache/mlxcel/models/...
  ... Warmup complete

The server starts and answers a /v1/chat/completions request; the same invocation previously aborted at the first internal block_on (the issue's exact repro shape). A missing-repo variant surfaces a clean HTTP 404 error through the serve path instead of the abort. A regression test pins that download_repo called from inside a multi-thread runtime returns a normal error (closed local endpoint, no network access). 123 downloader unit tests, cargo fmt, and cargo clippy --all-targets -- -D warnings are clean.

Closes #463

…okio runtime is active

download_repo creates its own Tokio runtime and drives the HTTP work with block_on, which Tokio forbids on a thread that is already inside a runtime; mlxcel serve -m <repo-id> (async startup) therefore aborted with "Cannot start a runtime from within a runtime" whenever the model had to be auto-downloaded, while the synchronous mlxcel run path worked (issue #463). The entry point now detects an active runtime via tokio::runtime::Handle::try_current() and runs the unchanged blocking body (download_repo_blocking) on a dedicated OS thread, joining it and propagating panics; sync callers keep the direct call. This makes every current and future caller safe (mlxcel serve, mlxcel-server's #[tokio::main] download path, and the CLI download/run verbs) with identical progress output. Validated end to end with the release binary: mlxcel serve -m mlx-community/SmolLM2-135M-Instruct-8bit with an empty store downloads the model (8 files, 141 MiB, same progress lines as run), starts serving, and answers a /v1/chat/completions request; the same invocation previously aborted at the first internal block_on. A regression test pins that download_repo called from inside a multi-thread runtime returns a normal error (against a closed local endpoint, no network access).
@inureyes inureyes added type:bug Bug fixes, error corrections, or issue resolutions area:core mlxcel-core: MLX FFI, primitives, KV cache, layers area:cli Command-line interface / CLI flags priority:high High priority labels Jul 5, 2026
@inureyes inureyes merged commit 0dc80d5 into main Jul 5, 2026
5 checks passed
@inureyes inureyes deleted the fix/issue-463-serve-nested-runtime branch July 5, 2026 23:27
inureyes added a commit that referenced this pull request Jul 6, 2026
…s) (#673)

Full text (169 models) and VLM re-benchmark on mlxcel 0.3.3 after the five-family VLM port batch (#660-#664), the slice_update/causal-mask/downloader fix batch (#666-#668), and the qwen3.5-MoE sanitize fix (#671). Adds metal_m1ultra_2026-07-06.csv, metal_m1ultra_vlm_2026-07-06.csv, the two post-#671 single-model re-bench CSVs (qwen3.5/3.6-35b-a3b back at 82.25/78.84 tok/s), and per-model mlx-lm/mlx-vlm baselines for 21 newly added models (13 measured, 8 python-side FAIL:warmup), plus the previously uncommitted June baseline CSVs (2026-06-13/14 singles and the 2026-06-16 repfresh pair). Versus 2026-06-15 there are no decode regressions; the qwen VLM family and small text models are 10-16% faster from the intervening fused-MoE and related work. The sweep caught issue #670 (qwen3.5/3.6-MoE silently failing to load since #588), fixed the same day in #671. New-model baseline highlights: ernie-4.5-vl 160% of mlx-vlm decode, deepseek-vl2 112%, idefics3 119%, qwen3-next-80b 118% of mlx-lm; mistral-small-4-119b runs only under mlxcel. Documentation: model_tests_m1ultra.md gets 110 refreshed rows, 14 new VLM rows, 9 new text rows, and updated environment/run-over-run notes; model_tests.md index registers the new CSVs. Also updates the ernie4_5_moe_vl and hunyuan_vl parity tests to the lowercase local model-directory names so the suites keep finding their checkpoints.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:cli Command-line interface / CLI flags area:core mlxcel-core: MLX FFI, primitives, KV cache, layers priority:high High priority type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(serve): mlxcel serve aborts with a nested-runtime panic when auto-downloading a missing model

1 participant