[Bug] Anima - LoRA shape mismatch handled ungracefully

[nphuracm]

Git commit

ea4e566

Operating System & Version

Arch Linux

GGML backends

Vulkan

Command-line arguments used

-p "lora:00_Cosmos_DMD2_Distill:0.8score_9, black background, 1other, kris (dark world) (deltarune), aqua skin, hair over eyes, armor, pink ascot, upper body, arm up, holding sword, pointing sword, pink sword, facing viewer, looking at viewer, (deformed:2.3), (flat color:1.9), (no lineart:2.7), vector art, expressionless. Kris stands in the midst of great darkness. They hold their sword to their side pointing away downwards. There is white sans-serif text to the top-left that reads "To be continued in Chapter 6"" --diffusion-model ~/Pictures/SD.cpp/anima_aestheticV11.safetensors --llm ~/Pictures/SD.cpp/Qwen3-0.6B-Base-Q8_0.gguf --vae /mnt/ntfshdd/SD_REPOS_MIRROR/comfyui/model/vae/qwen_image_vae.safetensors --cfg-scale 1 --steps 12 --sampling-method er_sde --scheduler sgm_uniform -W 1152 -H 864 --output /tmp/output.png --lora-model-dir ~/Pictures/SD.cpp/loras/ --threads 24 --clip-on-cpu --vae-on-cpu

Steps to reproduce

1. Obtain this LoRA: https://civitai.red/models/2466415/cosmos-predict25-2b-base-distilled-extracted-dmd2-lora
2. Apply this on Anima-Aesthetic v1.1
3. Run the full pipeline once

What you expected to happen

The LoRA, unmodified, should be able to load, and the image would generate correctly with the LoRA in effect.

What actually happened

The program aborts (SIGABRT) just before denoising with message: /home/nphuracm/Applications/stable-diffusion.cpp/ggml/src/ggml.c:3282: GGML_ASSERT(ggml_can_mul_mat(a, b)) failed

Logs / error messages / stack trace

#0  0x00007ffff769981c in ?? () from /usr/lib/libc.so.6
#1  0x00007ffff763e150 in raise () from /usr/lib/libc.so.6
#2  0x00007ffff762567d in abort () from /usr/lib/libc.so.6
#3  0x0000555555fc3ab6 in ggml_abort (file=0x55555bdcbea8 "/home/nphuracm/Applications/stable-diffusion.cpp/ggml/src/ggml.c", line=3282, fmt=0x55555bdcbbd7 "GGML_ASSERT(%s) failed")
    at /home/nphuracm/Applications/stable-diffusion.cpp/ggml/src/ggml.c:271
#4  0x0000555555fc92a0 in ggml_mul_mat (ctx=0x55555d3e8b70, a=0x555564208a40, b=0x7fff4060a1a0) at /home/nphuracm/Applications/stable-diffusion.cpp/ggml/src/ggml.c:3282
#5  0x00005555557c00e9 in ggml_ext_linear (ctx=0x55555d3e8b70, x=0x7fff4060a1a0, w=0x555564208a40, b=0x0, force_prec_f32=false, scale=1)
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/core/ggml_extend.hpp:1028
#6  0x00005555558399e9 in LoraModel::get_out_diff (this=0x55555d333270, ctx=0x55555d3e8b70, backend=0x55555c1ae0d0, x=0x7fff4060a1a0, forward_params=..., 
    model_tensor_name="model.diffusion_model.net.x_embedder.proj.1.weight") at /home/nphuracm/Applications/stable-diffusion.cpp/src/model/adapter/lora.hpp:737
#7  0x000055555583a455 in MultiLoraAdapter::forward_with_lora (this=0x55555cbb3a80, ctx=0x55555d3e8b70, backend=0x55555c1ae0d0, x=0x7fff4060a1a0, w=0x5555624ebf30, b=0x0, 
    prefix="model.diffusion_model.net.x_embedder.proj.1.", forward_params=...) at /home/nphuracm/Applications/stable-diffusion.cpp/src/model/adapter/lora.hpp:956
#8  0x00005555557f7fba in Linear::forward (this=0x55555ca01850, ctx=0x7fffffff9450, x=0x7fff4060a1a0)
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/core/ggml_extend.hpp:3384
#9  0x0000555555855f3f in Anima::XEmbedder::forward (this=0x55555cacdf10, ctx=0x7fffffff9450, x=0x7fff4060a1a0)
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/model/diffusion/anima.hpp:74
#10 0x000055555585c4b0 in Anima::AnimaNet::forward (this=0x55555ca01760, ctx=0x7fffffff9450, x=0x7fff4060a1a0, timestep=0x7fff401b05d0, encoder_hidden_states=0x7fff401b07a0, 
    image_pe=0x7fff40608be0, t5_ids=0x7fff401b0970, t5_weights=0x7fff401b0b40, adapter_q_pe=0x7fff40608db0, adapter_k_pe=0x7fff40608f80, 
    ref_latents=std::vector of length 0, capacity 0) at /home/nphuracm/Applications/stable-diffusion.cpp/src/model/diffusion/anima.hpp:516
#11 0x000055555585e387 in Anima::AnimaRunner::build_graph (this=0x55555ca01330, x_tensor=..., timesteps_tensor=..., context_tensor=..., t5_ids_tensor=..., t5_weights_tensor=..., 
    ref_latents_tensor=std::vector of length 0, capacity 0) at /home/nphuracm/Applications/stable-diffusion.cpp/src/model/diffusion/anima.hpp:694
#12 0x000055555585e4a8 in Anima::AnimaRunner::compute(int, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<int> const&, sd::Tensor<float> const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&)::{lambda()#1}::operator()() const (__closure=0x55555cbb65b0)
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/model/diffusion/anima.hpp:718
#13 0x00005555559febca in std::__invoke_impl<ggml_cgraph*, Anima::AnimaRunner::compute(int, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<int> const&, sd::Tensor<float> const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&)::{lambda()#1}&>(std::__invoke_other, Anima::AnimaRunner::compute(int, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<int> const&, sd::Tensor<float> const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&)::{lambda()#1}&) (__f=...) at /usr/include/c++/16/bits/invoke.h:63
#14 0x00005555559d452f in std::__invoke_r<ggml_cgraph*, Anima::AnimaRunner::compute(int, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<int> const&, sd::Tensor<float> const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&)::{lambda()#1}&>(Anima::AnimaRunner::compute(int, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<int> const&, sd::Tensor<float> const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&)::{lambda()#1}&) (__fn=...) at /usr/include/c++/16/bits/invoke.h:116
#15 0x00005555559a66df in std::_Function_handler<ggml_cgraph* (), Anima::AnimaRunner::compute(int, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<float> const&, sd::Tensor<int> const&, sd::Tensor<float> const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/16/bits/std_function.h:295
#16 0x000055555570f774 in std::function<ggml_cgraph* ()>::operator()() const (this=0x7fffffff9690) at /usr/include/c++/16/bits/std_function.h:581
#17 0x0000555555702033 in GGMLRunner::get_compute_graph(std::function<ggml_cgraph* ()>) (this=0x55555ca01330, get_graph=...)
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/core/ggml_extend.hpp:2004
#18 0x000055555570227c in GGMLRunner::prepare_compute_graph(std::function<ggml_cgraph* ()>, ggml_cgraph**) (this=0x55555ca01330, get_graph=..., gf_out=0x7fffffff9708)
--Type <RET> for more, q to quit, c to continue without paging--c
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/core/ggml_extend.hpp:2028
#19 0x0000555555712698 in GGMLRunner::compute<float>(std::function<ggml_cgraph* ()>, int, bool, bool, bool, bool) (this=0x55555ca01330, get_graph=..., n_threads=18, 
    auto_free=false, free_compute_buffer=false, free_compute_params=false, no_return=false) at /home/nphuracm/Applications/stable-diffusion.cpp/src/core/ggml_extend.hpp:3109
#20 0x000055555585e5da in Anima::AnimaRunner::compute (this=0x55555ca01330, n_threads=18, x=..., timesteps=..., context=..., t5_ids=..., t5_weights=..., 
    ref_latents=std::vector of length 0, capacity 0, ref_image_params=...) at /home/nphuracm/Applications/stable-diffusion.cpp/src/model/diffusion/anima.hpp:720
#21 0x000055555585e83e in Anima::AnimaRunner::compute (this=0x55555ca01330, n_threads=18, diffusion_params=...)
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/model/diffusion/anima.hpp:736
#22 0x0000555555961739 in StableDiffusionGGML::sample(std::shared_ptr<DiffusionModelRunner> const&, bool, sd::Tensor<float> const&, sd::Tensor<float>, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor<float> const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator<float> > const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&, sd::Tensor<float> const&, sd::Tensor<float> const&, float, int, float, sd_cache_params_t const*, sd::Tensor<float> const&)::{lambda(sd::Tensor<float> const&, float, int)#1}::operator()(sd::Tensor<float> const&, float, int) const::{lambda(SDCondition const&, sd::Tensor<float> const*, std::vector<int, std::allocator<int> > const*, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const*)#1}::operator()(SDCondition const&, sd::Tensor<float> const*, std::vector<int, std::allocator<int> > const*, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const*) const (
    __closure=0x7fffffff9ed0, condition=..., c_concat_override=0x0, local_skip_layers=0x0, ref_latents_override=0x0)
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/stable-diffusion.cpp:2588
#23 0x000055555595ef26 in StableDiffusionGGML::sample(std::shared_ptr<DiffusionModelRunner> const&, bool, sd::Tensor<float> const&, sd::Tensor<float>, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor<float> const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator<float> > const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&, sd::Tensor<float> const&, sd::Tensor<float> const&, float, int, float, sd_cache_params_t const*, sd::Tensor<float> const&)::{lambda(sd::Tensor<float> const&, float, int)#1}::operator()(sd::Tensor<float> const&, float, int) const (__closure=0x55555c85a470, 
    x=..., sigma=1, step=1) at /home/nphuracm/Applications/stable-diffusion.cpp/src/stable-diffusion.cpp:2611
#24 0x0000555555a2449d in std::__invoke_impl<sd::guidance::GuiderOutput, StableDiffusionGGML::sample(std::shared_ptr<DiffusionModelRunner> const&, bool, sd::Tensor<float> const&, sd::Tensor<float>, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor<float> const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator<float> > const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&, sd::Tensor<float> const&, sd::Tensor<float> const&, float, int, float, sd_cache_params_t const*, sd::Tensor<float> const&)::{lambda(sd::Tensor<float> const&, float, int)#1}&, sd::Tensor<float> const&, float, int>(std::__invoke_other, StableDiffusionGGML::sample(std::shared_ptr<DiffusionModelRunner> const&, bool, sd::Tensor<float> const&, sd::Tensor<float>, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor<float> const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator<float> > const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&, sd::Tensor<float> const&, sd::Tensor<float> const&, float, int, float, sd_cache_params_t const*, sd::Tensor<float> const&)::{lambda(sd::Tensor<float> const&, float, int)#1}&, sd::Tensor<float> const&, float&&, int&&) (__f=...)
    at /usr/include/c++/16/bits/invoke.h:63
#25 0x00005555559f006d in std::__invoke_r<sd::guidance::GuiderOutput, StableDiffusionGGML::sample(std::shared_ptr<DiffusionModelRunner> const&, bool, sd::Tensor<float> const&, sd::Tensor<float>, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor<float> const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator<float> > const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&, sd::Tensor<float> const&, sd::Tensor<float> const&, float, int, float, sd_cache_params_t const*, sd::Tensor<float> const&)::{lambda(sd::Tensor<float> const&, float, int)#1}&, sd::Tensor<float> const&, float, int>(StableDiffusionGGML::sample(std::shared_ptr<DiffusionModelRunner> const&, bool, sd::Tensor<float> const&, sd::Tensor<float>, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor<float> const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator<float> > const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&, sd::Tensor<float> const&, sd::Tensor<float> const&, float, int, float, sd_cache_params_t const*, sd::Tensor<float> const&)::{lambda(sd::Tensor<float> const&, float, int)#1}&, sd::Tensor<float> const&, float&&, int&&) (__fn=...) at /usr/include/c++/16/bits/invoke.h:118
#26 0x00005555559c74b7 in std::_Function_handler<sd::guidance::GuiderOutput (sd::Tensor<float> const&, float, int), StableDiffusionGGML::sample(std::shared_ptr<DiffusionModelRunner> const&, bool, sd::Tensor<float> const&, sd::Tensor<float>, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor<float> const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator<float> > const&, std::vector<sd::Tensor<float>, std::allocator<sd::Tensor<float> > > const&, RefImageParams const&, sd::Tensor<float> const&, sd::Tensor<float> const&, float, int, float, sd_cache_params_t const*, sd::Tensor<float> const&)::{lambda(sd::Tensor<float> const&, float, int)#1}>::_M_invoke(std::_Any_data const&, sd::Tensor<float> const&, float&&, int&&) (__functor=..., __args#0=..., __args#1=@0x7fffffffa674: 1, __args#2=@0x7fffffffa670: 1)
    at /usr/include/c++/16/bits/std_function.h:296
#27 0x000055555598c98b in std::function<sd::guidance::GuiderOutput(sd::Tensor<float> const&, float, int)>::operator() (this=0x7fffffffabb0, __args#0=..., __args#1=1, __args#2=1)
    at /usr/include/c++/16/bits/std_function.h:581
#28 0x00005555557d2a5e in sample_er_sde (model=..., x=..., sigmas=std::vector of length 13, capacity 13 = {...}, rng=std::shared_ptr<RNG> (use count 4, weak count 0) = {...}, 
    is_flow_denoiser=true, eta=1) at /home/nphuracm/Applications/stable-diffusion.cpp/src/runtime/denoiser.hpp:2413
#29 0x00005555557d60e8 in sample_k_diffusion (method=ER_SDE_SAMPLE_METHOD, model=..., x=..., sigmas=std::vector of length 13, capacity 13 = {...}, 
    rng=std::shared_ptr<RNG> (use count 4, weak count 0) = {...}, eta=1, is_flow_denoiser=true, extra_sample_args=0x0, 
    denoiser_for_dispatch=std::shared_ptr<Denoiser> (use count 2, weak count 0) = {...}) at /home/nphuracm/Applications/stable-diffusion.cpp/src/runtime/denoiser.hpp:2707
#30 0x00005555559625e8 in StableDiffusionGGML::sample (this=0x55555c1287e0, work_diffusion_model=std::shared_ptr<DiffusionModelRunner> (use count 1, weak count 0) = {...}, 
    inverse_noise_scaling=true, init_latent=..., noise=..., cond=..., uncond=..., img_uncond=..., control_image=..., control_strength=0.899999976, guidance=..., eta=1, 
    shifted_timestep=0, method=ER_SDE_SAMPLE_METHOD, is_flow_denoiser=true, extra_sample_args=0x0, sigmas=std::vector of length 13, capacity 13 = {...}, 
    ref_latents=std::vector of length 0, capacity 0, ref_image_params=..., denoise_mask=..., vace_context=..., vace_strength=1, audio_length=0, frame_rate=16, 
    cache_params=0x7fffffffcf10, video_positions=...) at /home/nphuracm/Applications/stable-diffusion.cpp/src/stable-diffusion.cpp:2696
#31 0x00005555557e2e30 in generate_image (sd_ctx=0x55555c18a400, sd_img_gen_params=0x7fffffffcdb0, images_out=0x7fffffffcfb0, num_images_out=0x7fffffffcb64)
    at /home/nphuracm/Applications/stable-diffusion.cpp/src/stable-diffusion.cpp:5387
#32 0x0000555555694fb1 in main (argc=25, argv=0x7fffffffdfc8) at /home/nphuracm/Applications/stable-diffusion.cpp/examples/cli/main.cpp:881

Additional context / environment details

GPU: AMD Radeon RX 580 2048SP.

Before reporting, I have already run OpenAI Codex locally to pinpoint the issue and decide whether to report. Excerpts from its final responses:

• This is not an AMD, Vulkan, compiler, or ggml bug.

  The immediate cause is one incompatible tensor between the Anima checkpoint and Cosmos2.5 LoRA:

  Anima x_embedder weight: [2048, 68]
  LoRA down:               [64, 72]
  LoRA up:                 [2048, 64]
  Resulting LoRA update:   [2048, 72]

  Anima constructs 68 input features from (16 latent channels + 1 mask) * 2 * 2 in src/model/diffusion/anima.hpp:469. The Cosmos2.5 LoRA expects 72. At runtime, sd.cpp passes the 68-
  wide input through the 72-wide LoRA projection at src/model/adapter/lora.hpp:737, causing ggml’s correct dimension check to fail at ggml/src/ggml.c:3282.

  I compared every tensor using only the safetensors headers:

  - 454 LoRA matrix pairs
  - 113 full-difference tensors
  - 566 of 567 target weights are compatible
  - diffusion_model.x_embedder.proj.1 is the sole mismatch

  So there are two aspects to the diagnosis:

  - The assets are not completely shape-compatible because Cosmos2 and Cosmos2.5 differ at x_embedder.
  - sd.cpp has a robustness bug: it should warn and skip this incompatible pair, allowing the other 1,019 LoRA tensors to apply. Instead, runtime mode reaches an invalid matrix
    multiply; immediate mode would hit the element-count assertion in src/model/adapter/lora.hpp:549.

  Civitai working is consistent with systems that skip shape-mismatched LoRA patches. A practical workaround is to remove these two tensors from a copy of the LoRA:

  diffusion_model.x_embedder.proj.1.lora_down.weight
  diffusion_model.x_embedder.proj.1.lora_up.weight

  Updating ggml or changing Vulkan drivers cannot resolve this. The pinned ggml revision performs the same dimension check as your updated submodule. No repository files were
  changed.

...

  The current abort is not useful strict validation. A library-level GGML_ASSERT provides no tensor name, dimensions, or recovery path and terminates the entire host process. Even if
  maintainers prefer strict compatibility, sd.cpp should detect this earlier and return a descriptive error.

  I recommend opening an issue first because the desired policy needs maintainer agreement:

  - Default permissive: warn and skip incompatible LoRA targets, consistent with existing partial/unused-tensor handling.
  - Optional strict mode: reject the LoRA cleanly before generation.
  - Never silently truncate a 72-wide adapter to 68; skipping the complete target pair is safer.