Skip to content

[Feat] support JoyAIImagePipeline #13437

Open
RuixiangMa wants to merge 6 commits intohuggingface:mainfrom
RuixiangMa:joyaiimage
Open

[Feat] support JoyAIImagePipeline #13437
RuixiangMa wants to merge 6 commits intohuggingface:mainfrom
RuixiangMa:joyaiimage

Conversation

@RuixiangMa
Copy link
Copy Markdown

@RuixiangMa RuixiangMa commented Apr 9, 2026

Support JoyAIImagePipeline

Prompt image
NO dog
add hat in the dog joyai_image_seqoffload_out
Rotate the dog to show the left side view left
Rotate the dog to show the right side view right

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Signed-off-by: Lancer <maruixiang6688@gmail.com>
@github-actions github-actions bot added models pipelines size/L PR with diff > 200 LOC labels Apr 9, 2026
@RuixiangMa RuixiangMa marked this pull request as draft April 9, 2026 11:42
Signed-off-by: Lancer <maruixiang6688@gmail.com>
@github-actions github-actions bot added size/L PR with diff > 200 LOC and removed size/L PR with diff > 200 LOC labels Apr 9, 2026
Signed-off-by: Lancer <maruixiang6688@gmail.com>
@github-actions github-actions bot added schedulers size/L PR with diff > 200 LOC and removed size/L PR with diff > 200 LOC labels Apr 9, 2026
Signed-off-by: Lancer <maruixiang6688@gmail.com>
@github-actions github-actions bot added tests size/L PR with diff > 200 LOC and removed size/L PR with diff > 200 LOC labels Apr 9, 2026
@RuixiangMa RuixiangMa marked this pull request as ready for review April 9, 2026 18:20
Copy link
Copy Markdown
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the PR, i left some initial comments

from .autoencoder_kl_wan import (
WanAttentionBlock as AttentionBlock,
)
from .autoencoder_kl_wan import (
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you not import from wan? let's use #Copied from isntead

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will remove the direct imports from Wan and inlined the reused helper blocks in JoyAI with # Copied from comments instead.

self._joyai_force_vae_fp32 = True

@classmethod
def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's going on here? is there any reason we cannot make eit work with DiffusionPipeline.from_pretrained?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The upstream JoyAI checkpoint is not a standard diffusers repo layout, so it cannot be loaded by DiffusionPipeline.from_pretrained directly.

I agree the better long-term solution is to add a one-time conversion/export to a standard diffusers checkpoint and then rely on the base from_pretrained path for the exported directory

prev_sample: torch.FloatTensor


class JoyAIFlowMatchDiscreteScheduler(SchedulerMixin, ConfigMixin):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@claude
can you tell me what's the difference between this and FlowMatchDiscreteSchedule?

Copy link
Copy Markdown
Author

@RuixiangMa RuixiangMa Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I double-checked this. It is close to FlowMatchEulerDiscreteScheduler, but not numerically identical as-is because JoyAI uses its own timestep/sigma shifting behavior.

I will refactorit to subclass FlowMatchEulerDiscreteScheduler and only keep the JoyAI-specific differences in the scheduler implementation.

Signed-off-by: Lancer <maruixiang6688@gmail.com>
Signed-off-by: Lancer <maruixiang6688@gmail.com>
@github-actions github-actions bot added documentation Improvements or additions to documentation size/L PR with diff > 200 LOC and removed size/L PR with diff > 200 LOC labels Apr 10, 2026
@RuixiangMa
Copy link
Copy Markdown
Author

@yiyixuxu Thanks for the comments! PTAL

@Moran232
Copy link
Copy Markdown

Moran232 commented Apr 10, 2026

@RuixiangMa
Thank you very much for your attention and your pull request!

The JoyAI team is the AI team from JD Explore Academy, and this is our first open-source release of a self-developed diffusion model.

After open-sourcing the model on GitHub, we spent one week adding support for the diffusers library. The official name of the model is JoyAI-Image-Edit. In the diffusers pipeline, we would like to name it JoyImageEditPipeline, as we plan to release additional models soon—including JoyImagePipeline, JoyVideo, and others to form a cohesive model family. Therefore, naming this PR JoyAIImagePipeline might not be the most appropriate choice.

I have submitted a new Pull Request that includes the following key updates:

  • The model now fully adopts WanVAE, completely reusing WanVAE's existing code.
  • Removed the dependency on FlashAttention.
  • Fixed discrepancies between the scheduler implementation and the originally open-sourced version.

To facilitate future model updates and open-source releases, could you please migrate your code from this PR to my newly submitted PR? You’re also welcome to directly add any other valuable suggestions or changes to that PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation models pipelines schedulers size/L PR with diff > 200 LOC tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants