fix(downloader): offload download_repo to a dedicated thread when a Tokio runtime is active#668
Merged
Merged
Conversation
…okio runtime is active download_repo creates its own Tokio runtime and drives the HTTP work with block_on, which Tokio forbids on a thread that is already inside a runtime; mlxcel serve -m <repo-id> (async startup) therefore aborted with "Cannot start a runtime from within a runtime" whenever the model had to be auto-downloaded, while the synchronous mlxcel run path worked (issue #463). The entry point now detects an active runtime via tokio::runtime::Handle::try_current() and runs the unchanged blocking body (download_repo_blocking) on a dedicated OS thread, joining it and propagating panics; sync callers keep the direct call. This makes every current and future caller safe (mlxcel serve, mlxcel-server's #[tokio::main] download path, and the CLI download/run verbs) with identical progress output. Validated end to end with the release binary: mlxcel serve -m mlx-community/SmolLM2-135M-Instruct-8bit with an empty store downloads the model (8 files, 141 MiB, same progress lines as run), starts serving, and answers a /v1/chat/completions request; the same invocation previously aborted at the first internal block_on. A regression test pins that download_repo called from inside a multi-thread runtime returns a normal error (against a closed local endpoint, no network access).
inureyes
added a commit
that referenced
this pull request
Jul 6, 2026
…s) (#673) Full text (169 models) and VLM re-benchmark on mlxcel 0.3.3 after the five-family VLM port batch (#660-#664), the slice_update/causal-mask/downloader fix batch (#666-#668), and the qwen3.5-MoE sanitize fix (#671). Adds metal_m1ultra_2026-07-06.csv, metal_m1ultra_vlm_2026-07-06.csv, the two post-#671 single-model re-bench CSVs (qwen3.5/3.6-35b-a3b back at 82.25/78.84 tok/s), and per-model mlx-lm/mlx-vlm baselines for 21 newly added models (13 measured, 8 python-side FAIL:warmup), plus the previously uncommitted June baseline CSVs (2026-06-13/14 singles and the 2026-06-16 repfresh pair). Versus 2026-06-15 there are no decode regressions; the qwen VLM family and small text models are 10-16% faster from the intervening fused-MoE and related work. The sweep caught issue #670 (qwen3.5/3.6-MoE silently failing to load since #588), fixed the same day in #671. New-model baseline highlights: ernie-4.5-vl 160% of mlx-vlm decode, deepseek-vl2 112%, idefics3 119%, qwen3-next-80b 118% of mlx-lm; mistral-small-4-119b runs only under mlxcel. Documentation: model_tests_m1ultra.md gets 110 refreshed rows, 14 new VLM rows, 9 new text rows, and updated environment/run-over-run notes; model_tests.md index registers the new CSVs. Also updates the ernie4_5_moe_vl and hunyuan_vl parity tests to the lowercase local model-directory names so the suites keep finding their checkpoints.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mlxcel serve -m <repo-id>aborted with Tokio's "Cannot start a runtime from within a runtime" whenever the model had to be auto-downloaded, becausedownload_repobuilds its own runtime and callsblock_on, which is forbidden on a thread already driving a runtime (run_serveis async). The synchronousmlxcel runpath never hit it.Fix
The
download_repoentry point now detects an active runtime viatokio::runtime::Handle::try_current()and runs the unchanged blocking body (download_repo_blocking) on a dedicated OS thread (joined, panics propagated); synchronous callers keep the direct call. This is the issue's "runtime-aware download entry" option: one change makes every current and future caller safe —mlxcel serve,mlxcel-server's#[tokio::main]download path near startup, and the CLIdownload/runverbs — with identical progress output (the bars write to the same stderr).Validation
End to end with the release binary and an empty store:
The server starts and answers a
/v1/chat/completionsrequest; the same invocation previously aborted at the first internalblock_on(the issue's exact repro shape). A missing-repo variant surfaces a clean HTTP 404 error through the serve path instead of the abort. A regression test pins thatdownload_repocalled from inside a multi-thread runtime returns a normal error (closed local endpoint, no network access). 123 downloader unit tests,cargo fmt, andcargo clippy --all-targets -- -D warningsare clean.Closes #463