Skip to content

add modular blockset for wan 2.2 vace - #14432

Open
akshan-main wants to merge 6 commits into
huggingface:mainfrom
akshan-main:wan-vace-modular
Open

add modular blockset for wan 2.2 vace#14432
akshan-main wants to merge 6 commits into
huggingface:mainfrom
akshan-main:wan-vace-modular

Conversation

@akshan-main

@akshan-main akshan-main commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a modular blockset for Wan 2.2 VACE: Wan22VaceBlocks with a WanVaceEncoderStep for the VACE conditioning latents (control video, mask, reference images), a two-transformer Wan22VaceLoopDenoiser that forwards control_hidden_states as shared kwargs across the guider batches (same shape as the qwenimage controlnet denoise step), and a post-denoise reference-frame trim. Existing wan blocks are reused unchanged.

Outputs match WanVACEPipeline exactly with shared components and seed. One deliberate divergence: output_type="latent" returns the trimmed latents so they are directly decodable. Test components live in akshan-main/tiny-wan22-vace-modular-pipe.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline?
  • Did you read our philosophy doc (important for complex PRs)?
  • Was this discussed/approved via a GitHub issue or the forum? Discussed on Slack with @sayakpaul
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Who can review?

@yiyixuxu @sayakpaul

@github-actions

Copy link
Copy Markdown
Contributor

Hi @akshan-main, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Hi @akshan-main, a follow-up on the reminder above: this PR still does not link an issue it fixes.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice. Adding a closing keyword (e.g. Fixes #1234) to the PR description, or a maintainer adding the no-issue-needed label, will prevent that.

@sayakpaul
sayakpaul requested a review from yiyixuxu August 27, 2026 10:57

@yiyixuxu yiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@property
def expected_components(self) -> list[ComponentSpec]:
return [
ComponentSpec("transformer", WanVACETransformer3DModel),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we not list transformer as components here? if you just need the patch size, make it a property on pipeline like vae_scale_factor_temporal

@akshan-main akshan-main Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed it. Also adapted the test file to the new modular mixin structure.

@yiyixuxu yiyixuxu added close-to-merge no-issue-needed for PRs that do not require link to an issue and removed close-to-merge labels Aug 27, 2026
@akshan-main
akshan-main requested a review from yiyixuxu August 30, 2026 21:39
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yiyixuxu

yiyixuxu commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

@akshan-main can you take a look at the failing CI?

@akshan-main

Copy link
Copy Markdown
Contributor Author

fixed!

return components, block_state


class Wan22VaceLoopDenoiser(ModularPipelineBlocks):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, can we try to see if we can reuse Wan22LoopDenoiser here and make it more general instead of adding a new block? There is a little bit of a write-up on this here:
https://github.com/huggingface/diffusers/blob/main/.ai/references/modular.md#growing-a-pipeline-one-workflow-at-a-time

I think the only differences are:

  1. the transformer type — we can make it dynamic in init
  2. the extra inputs passed to the denoiser, we can use kwargs_type="denoiser_input_fields" pattern

let me know what you think!

@akshan-main akshan-main Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that works, the control tensors just need to stay shared across the guider batches. Will rework

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

modular-pipelines no-issue-needed for PRs that do not require link to an issue size/L PR with diff > 200 LOC tests utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants