add modular blockset for wan 2.2 vace - #14432
Conversation
|
Hi @akshan-main, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. |
|
Hi @akshan-main, a follow-up on the reminder above: this PR still does not link an issue it fixes. Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice. Adding a closing keyword (e.g. |
| @property | ||
| def expected_components(self) -> list[ComponentSpec]: | ||
| return [ | ||
| ComponentSpec("transformer", WanVACETransformer3DModel), |
There was a problem hiding this comment.
can we not list transformer as components here? if you just need the patch size, make it a property on pipeline like vae_scale_factor_temporal
There was a problem hiding this comment.
Removed it. Also adapted the test file to the new modular mixin structure.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@akshan-main can you take a look at the failing CI? |
|
fixed! |
| return components, block_state | ||
|
|
||
|
|
||
| class Wan22VaceLoopDenoiser(ModularPipelineBlocks): |
There was a problem hiding this comment.
oh, can we try to see if we can reuse Wan22LoopDenoiser here and make it more general instead of adding a new block? There is a little bit of a write-up on this here:
https://github.com/huggingface/diffusers/blob/main/.ai/references/modular.md#growing-a-pipeline-one-workflow-at-a-time
I think the only differences are:
- the transformer type — we can make it dynamic in init
- the extra inputs passed to the denoiser, we can use
kwargs_type="denoiser_input_fields"pattern
let me know what you think!
There was a problem hiding this comment.
yeah that works, the control tensors just need to stay shared across the guider batches. Will rework
What does this PR do?
Adds a modular blockset for Wan 2.2 VACE:
Wan22VaceBlockswith aWanVaceEncoderStepfor the VACE conditioning latents (control video, mask, reference images), a two-transformerWan22VaceLoopDenoiserthat forwardscontrol_hidden_statesas shared kwargs across the guider batches (same shape as the qwenimage controlnet denoise step), and a post-denoise reference-frame trim. Existing wan blocks are reused unchanged.Outputs match
WanVACEPipelineexactly with shared components and seed. One deliberate divergence:output_type="latent"returns the trimmed latents so they are directly decodable. Test components live in akshan-main/tiny-wan22-vace-modular-pipe.Before submitting
Who can review?
@yiyixuxu @sayakpaul