[Feat] Adds LongCat-AudioDiT pipeline

[RuixiangMa]

What does this PR do?

Adds LongCat-AudioDiT model support to diffusers.

Although LongCat-AudioDiT can be used for TTS-like generation, it is fundamentally a diffusion-based audio generation model (text conditioning + iterative latent denoising + VAE decoding) rather than a conventional autoregressive TTS model, so i think it fits naturally into diffusers.

Test

import soundfile as sf
import torch
from diffusers import LongCatAudioDiTPipeline

pipeline = LongCatAudioDiTPipeline.from_pretrained(
    "meituan-longcat/LongCat-AudioDiT-1B",
    torch_dtype=torch.float16,
)
pipeline = pipeline.to("cuda")

audio = pipeline(
    prompt="A calm ocean wave ambience with soft wind in the background.",
    audio_end_in_s=5.0,
    num_inference_steps=16,
    guidance_scale=4.0,
    output_type="pt",
).audios

output = audio[0, 0].float().cpu().numpy()
sf.write("longcat.wav", output, pipeline.sample_rate)

Result

longcat.wav

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[dg845]

Similarly, I think we should inline _pixel_shuffle_1d in UpsampleShortcut following #13390 (comment).

[dg845]

Suggested change

	self.time_embed = AudioDiTTimestepEmbedding(dim)
	self.input_embed = AudioDiTEmbedder(latent_dim, dim)
	self.text_embed = AudioDiTEmbedder(dit_text_dim, dim)
	self.rotary_embed = AudioDiTRotaryEmbedding(dim_head, 2048, base=100000.0)
	self.blocks = nn.ModuleList(
	self.time_embed = AudioDiTTimestepEmbedding(dim)
	self.input_embed = AudioDiTEmbedder(latent_dim, dim)
	self.text_embed = AudioDiTEmbedder(dit_text_dim, dim)
	self.rotary_embed = AudioDiTRotaryEmbedding(dim_head, 2048, base=100000.0)
	self.blocks = nn.ModuleList(

See #13390 (comment).

[dg845]

Suggested change

	batch_size = hidden_states.shape[0]
	if timestep.ndim == 0:
	timestep = timestep.repeat(batch_size)
	timestep_embed = self.time_embed(timestep)
	text_mask = encoder_attention_mask.bool()
	encoder_hidden_states = self.text_embed(encoder_hidden_states, text_mask)
	batch_size = hidden_states.shape[0]
	if timestep.ndim == 0:
	timestep = timestep.repeat(batch_size)
	timestep_embed = self.time_embed(timestep)
	text_mask = encoder_attention_mask.bool()
	encoder_hidden_states = self.text_embed(encoder_hidden_states, text_mask)

Can you also refactor forward here so that it is better organized, following #13390 (comment)? See for example the QwenImageTransformer2DModel.forward method:

diffusers/src/diffusers/models/transformers/transformer_qwenimage.py

Line 836 in d7bc233

def forward(

[RuixiangMa]

Reorganized parts of forward incrementally; kept the current structure otherwise to avoid unnecessary behavioral churn.

[dg845]

Thanks for the PR! Left an initial design review :).

[RuixiangMa]

Thanks for the PR! Left an initial design review :).

Thx for the comments, I have made the changes, PTAL.

[dg845]

Can you refactor test_longcat_audio_pipeline_from_pretrained_real_local_weights to be part of a LongCatAudioDiTPipelineSlowTests class? For reference, see the Stable Diffusion 3 slow tests:

diffusers/tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3.py

Lines 227 to 231 in 431066e

	@slow
	@require_big_accelerator
	class StableDiffusion3PipelineSlowTests(unittest.TestCase):
	pipeline_class = StableDiffusion3Pipeline
	repo_id = "stabilityai/stable-diffusion-3-medium-diffusers"

[RuixiangMa]

refactored the real local-weights test into a LongCatAudioDiTPipelineSlowTests class.

[dg845]

Thanks for iterating! I left some follow-up comments.

[RuixiangMa]

Thanks for iterating! I left some follow-up comments.

Thx, PTAL