fix(model-artifacts): materialize longcat-video on the controller, and support companion repos#10949
Merged
Merged
Conversation
…ntroller longcat-video loads a checkpoint directory: its backend.py takes request.ModelFile when os.path.isdir(request.ModelFile) and otherwise falls back to snapshot_download. That places it in the same class as transformers/vllm/diffusers/sglang, but the allow-list added in #10910 did not enumerate it, so PrimaryArtifactSpec returned no managed artifact for a bare HuggingFace repo id. The consequence in distributed mode: nothing was acquired on the controller, ModelFileName fell through to the raw repo id, and staging skipped the resulting phantom /models/<owner>/<repo> path. The worker received a blank ModelFile, fell back to request.Model, and downloaded ~83GB from HuggingFace inside the remote LoadModel deadline - so the load could only ever fail with DeadlineExceeded while an abandoned backend process kept downloading. Note this materializes the full repository. The backend restricts its own snapshot_download with allow_patterns, and the avatar repo ships both base_model/ and base_model_int8/ where only one is ever loaded; inferred specs have no way to carry patterns today. Tracked separately. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
stageModelFiles logs "Staging model files for remote node" up front, then silently drops any path field that does not exist on the controller. The skip itself is legitimate and must stay: a backend outside managedArtifactBackends that takes a bare HuggingFace repo id gets an optimistically constructed path (ModelFileName falls through to the raw model reference) that was never materialized, and sources its own weights on the worker. Erroring would break those configs. But at debug level the operator is left with a reassuring staging line and no trace of the skip, so a genuine controller-side acquisition gap is indistinguishable from a healthy pass-through - it surfaces much later as a remote LoadModel timeout, on a worker that is quietly downloading tens of gigabytes. Raise the skip to warn and name the field, path, node and tracking key. Behavior is unchanged. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A composed pipeline needs more than one HuggingFace snapshot.
LongCat-Video-Avatar-1.5 loads its own transformer but takes the
tokenizer, text encoder and VAE from the separate LongCat-Video base
repo, so a single-artifact config cannot express it and the backend is
left to fetch the second repo itself at load time.
Widen the artifact model to target: model plus any number of named
target: companion entries. Normalize accepts the new target and
constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that
name is the option key the backend later receives; a companion may not
claim primary_file, which only means anything for a load target.
ModelConfig.Validate requires exactly one primary and requires it first,
since Artifacts[0] is what ModelFileName, size estimation and staging all
resolve from.
Both acquisition paths now loop instead of touching index 0 alone:
preloadOne for an already-installed config, bindPrimaryArtifact for a
gallery install. Failure policy differs by provenance. An inferred
primary keeps its warn-and-fall-back, because the legacy download path
still exists for it. Companions are explicit by construction, so they are
all-or-nothing: a config naming one is asserting the backend needs it,
and failing at the acquisition boundary is far more legible than a
missing-weights error surfacing later inside the backend.
The cache key is deliberately unchanged. It hashes source identity only,
never name or target, so every already-installed managed model still hits
its existing snapshot instead of silently re-downloading. Two specs pin
that: one proving a companion and a primary with identical sources agree
on the key, and one pinning the digest of a known primary outright.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A materialized companion is useless until the backend can find it, and its location is a content-addressed cache key that does not exist until the artifact resolves. A static gallery override cannot carry that, and persisting it into the config YAML would rot the moment a re-resolve produced a new key. Synthesize it instead at load time: each resolved companion becomes "<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the key:value convention backends already parse for options like attention_backend. The value stays relative to the models directory so a remote worker can resolve it under its own ModelPath once staging has rewritten the model root. An option the author set explicitly always wins, so pinning a companion to a local checkout still beats the managed snapshot. longcat-video resolves base_model through ModelPath, the same convention qwen-tts, voxcpm, outetts and ace-step already use for companion assets. Its sibling-directory heuristic is deleted: it looked for a LongCat-Video directory next to the model, which cannot exist under the content addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead code the moment the model became managed. The gallery entry declares both repositories and restricts each with allow_patterns. The avatar repo ships base_model/ and base_model_int8/ and only ever loads one, so fetching the whole repo would roughly double the download. The patterns match the entry's own options (use_distill true, use_int8 default false); enabling use_int8 here also requires adding base_model_int8/**, which is called out in the entry. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Staging anchored the worker's models directory on the primary snapshot whenever a model was managed, so a companion snapshot could not reach the worker at all. frontendModelsDir was derived by stripping the Model relative path off the end of ModelFile. For a managed artifact nothing matches: ModelFile is .artifacts/huggingface/<key>/snapshot while Model stays a bare HuggingFace repo id, so the strip was a no-op and the "models directory" came out as the snapshot itself. Two consequences, both silent. Staging keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two snapshots of one model were indistinguishable on the worker. And a companion, which lives in a sibling snapshot directory outside the primary, fell outside that directory entirely: StagingKeyMapper.Key collapsed its files to bare basenames and resolveOptionPath could not resolve the relative option at all, so it was skipped without a word. Derive the models root from the artifact tree instead when the path runs through it, and compute the worker's ModelPath from the file's path relative to that root rather than from the Model field. The legacy layout is unaffected: where Model really is the relative path, the new derivation reduces to the old one, which a regression spec pins. This deliberately changes an invariant that router_dirstage_test.go pinned: for a managed primary, ModelFile and ModelPath were both the snapshot directory, and staging keys were relative to it. Now ModelFile is the snapshot, ModelPath is the models root above it, and keys keep the full relative path. That spec is updated rather than accommodated, with the reasoning recorded inline, because the old invariant is exactly what made a sibling companion unreachable. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Loading
longcat-video-avatar-1.5in distributed mode failed withDeadlineExceededafter exactly 302s. The apparent cause was a short timeout. The actual cause was that the controller never acquired the weights at all, so the worker's Python backend downloaded ~57 GB from HuggingFace inside theLoadModelRPC.Observed on a live cluster:
/models/meituan-longcatexisted on neither the controller nor the worker. Meanwhile the worker's HF cache grew to 57 GB:Root cause — a regression from #10910
core/config/model_artifact_inference.gogates artifact inference on a backend allow-list.longcat-videowas absent, soPrimaryArtifactSpecreturnedhasArtifact=false,bindPrimaryArtifactwas never reached, andEnsurewas never called.That gate arrived in
2dade4a9f(#10910), which post-dates the longcat backend (#10792). #10910 was itself a correctness fix — single-file backends like llama.cpp were being handed a snapshot directory — and it enumerated the directory-consuming backends known at the time. longcat-video was simply never added.Confirmed by three independent checks on the live controller:
/models/.artifacts/huggingface/exists and holds snapshots for other models; the longcat config carries noartifacts:key; and no "falling back to legacy" log line was ever emitted for it.Bare
owner/reporeferences were never the problem —PrimaryArtifactSpecsynthesizes a spec viaCanonicalRepofor 2-part references. The allow-list was the only thing blocking it.The backend already implements the intended contract (
backend/python/longcat-video/backend.py:117-119): useModelFilewhen it is a directory, otherwise fall back tosnapshot_download. The self-download is the fallback, not the design.Changes
1. Add
longcat-videotomanagedArtifactBackends. Same criterion astransformers/vllm/diffusers— it consumes a snapshot directory.2. Warn when staging skips a missing source.
stageModelFilesskipped a non-existent path atxlog.Debugand blanked the field, while a transport failure on the same field 40 lines later isxlog.Error+ hard failure. That asymmetry is why this looked like a timeout bug for hours: the reassuring "Staging model files for remote node" line was emitted for a path that existed nowhere. Behavior is unchanged (a missingModelFileis legitimate for backends that fetch their own weights) — only visibility.3. Companion artifacts. Avatar 1.5 needs two repos: its own (35 GB) and the base
LongCat-Video(22 GB) for tokenizer, text encoder and VAE.Spec.Normalizepreviously rejected any artifact not namedmodel, and onlyArtifacts[0]was ever materialized, so the second repo could not be expressed and kept being fetched in-band.target ∈ {model, companion}is now accepted, with exactly one primary at index 0. BothpreloadOneandbindPrimaryArtifactloop. Failure policy: the inferred primary keeps warn-and-fall-back; declared companions are all-or-nothing.Cache-key stability is pinned two ways, since a key change would silently re-download every installed managed model: one spec proves a companion and a primary with identical sources yield the same key (name/target are not in the identity struct), another pins a known primary's literal digest.
4. Hand the resolved snapshot to the backend.
withCompanionArtifactOptionssynthesizes<name>:<relative snapshot path>intopb.ModelOptions.Optionsat load time, never persisted — a re-resolve to a new revision would otherwise leave a stale path in the on-disk YAML. Author-set keys win.backend.pyresolvesbase_modelthroughrequest.ModelPath, the conventionqwen-tts/voxcpm/outetts/ace-stepalready use, and the sibling heuristic at:443-450is deleted — it looked for<snapshot>/../LongCat-Video, which cannot exist under a cache-key layout.5. Stage from the models root.
ModelsRootForModelFileanchors on the directory containing the.artifactstree;DeriveRemoteModelPathtakes the models-root-relative path instead of theModelfield. PreviouslyfrontendModelsDir = TrimSuffix(ModelFile, Model)never matched for a managed primary (ModelFileends in.artifacts/huggingface/<key>/snapshot,Modelis a bare repo id), so it collapsed to the snapshot dir itself — self-consistent for the primary, fatal for a companion, whichstageGenericOptionsthen skipped silently. That is the same silent-skip class as (2), in a second location.router_dirstage_test.gois updated deliberately: it pinnedModelFile == ModelPath == snapshot, and nowModelPathis the models root with keys keeping their full relative path. Reasoning is recorded inline and in the commit message. A legacy-layout regression spec was added that passes before and after, as evidence the old path is untouched.Tradeoff worth reviewing
The gallery
allow_patternspinbase_model/**, matching this entry'suse_distill:trueanduse_int8defaulting false. Settinguse_int8:trueon this entry would find the int8 weights absent from the snapshot. Documented inline in the gallery entry rather than fetching both subfolders and forfeiting roughly half the download.Testing
Red-first throughout: 7 functional failures across
pkg/modelartifactsandcore/configfor step 1, 2 for step 2, and step 3's sole failure was exactly therouter_dirstage_test.goinvariant.One test was corrected rather than papered over: a "rejects two primaries" spec asserted an unreachable path (two primaries must both be named
model, so duplicate-name fires first). It was split into the genuinely reachable zero-primary and second-artifact-claiming-primary cases.make lint→ 0 issues. Coverage 53.0% (baseline 48.5%).Merge notes
core/services/nodes/router.gois touched in 4 small hunks (net +15/−10), all insidestageModelFilesat lines ~1033-1173. #10948 touches the same file at lines 18-336.git merge-treebetween the two branches reports zero conflicts.Future cleanup (not in this PR)
diffusers/backend.py:259-263hardcodes abfl_repoto pulltext_encoder_2from a FLUX base repo — the same companion pattern, now expressible declaratively.🤖 Generated with Claude Code