[LTX-2] Add motion_speed to condition the positional time base apart from the playback frame rate

docs/source/en/api/pipelines/ltx2.md

@@ -915,6 +915,73 @@ print(f"predicted {seconds:.2f}s -> {num_frames} frames")
 
 Converting a 2.5 checkpoint picks the head up automatically with `--full_pipeline`, or on its own with `--duration_head`. Checkpoints predating 2.5 have no such weights, and conversion skips the component rather than failing.
 
+### Motion speed for LTX-2.5
+
+LTX-2 places every video token on a time axis in seconds, derived by default from `frame_rate`, the playback rate. `motion_speed` (default `1.0`) separates the two: the time axis is conditioned on `frame_rate / motion_speed`, while the audio length, the [duration head](#automatic-duration-for-ltx-25) and the returned video still follow `frame_rate`. With `frame_rate=24.0` and `motion_speed=0.2`, the model is told the action was captured at 120 fps and the result plays back at 24 fps, which gives 5x slow motion.
+
+The base model is trained around 24/25/30/60 fps, so speeds far from `1.0` only make sense with an adapter trained for them, such as [`Lightricks/LTX-2.5-22b-LoRA-Slow-Motion-Control`](https://huggingface.co/Lightricks/LTX-2.5-22b-LoRA-Slow-Motion-Control) (gated: accept its license on the Hub first). Without `motion_speed`, that LoRA loads as a plain LoRA and gives none of the motion control. The model card suggests speeds of `1.0`, `0.5`, `0.2`, `0.1`, `0.05` and `0.025`, at LoRA strength `1.0`. Leave speed words ("slow motion", "high fps") out of the prompt, and reuse the same caption across speeds, since `motion_speed` alone drives the effect.
+
+```py
+import torch
+from diffusers import LTX2ImageToVideoPipeline
+from diffusers.pipelines.ltx2.utils import DISTILLED_SIGMA_VALUES
+from diffusers.utils import encode_video, load_image
+
+device = "cuda"  # or "mps", "xpu", "cpu"
+width = 1920  # the LoRA's recommended resolution; the effect is much weaker at 768x512
+height = 1088
+frame_rate = 24.0  # playback rate, always 24 for this LoRA
+model_path = "Lightricks/LTX-2.5-Diffusers"
+
+pipe = LTX2ImageToVideoPipeline.from_pretrained(model_path, dtype=torch.bfloat16)
+pipe.load_lora_weights(
+    "Lightricks/LTX-2.5-22b-LoRA-Slow-Motion-Control",
+    weight_name="ltx-2.5-22b-lora-slow-motion-control-1.0.safetensors",
+    adapter_name="slow_motion",
+)
+pipe.set_adapters("slow_motion", 1.0)
+pipe.enable_sequential_cpu_offload(device=device)
+pipe.vae.enable_tiling()
+
+image = load_image(
+    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
+)
+# No speed wording: the same caption is reused at every speed.
+prompt = (
+    "An astronaut hatches from a fragile egg on the surface of the Moon, the shell cracking apart and lunar dust "
+    "lifting into the air, while the camera pushes in."
+)
+
+for motion_speed in (1.0, 0.2):
+    video, audio = pipe(
+        image=image,
+        prompt=prompt,
+        width=width,
+        height=height,
+        num_frames=121,
+        frame_rate=frame_rate,
+        motion_speed=motion_speed,  # 1.0 = real time, 0.2 = 5x slow motion
+        sigmas=DISTILLED_SIGMA_VALUES,
+        guidance_scale=1.0,
+        audio_guidance_scale=1.0,
+        generator=torch.Generator(device).manual_seed(42),
+        output_type="np",
+        return_dict=False,
+    )
+
+    encode_video(
+        video[0],
+        fps=frame_rate,  # the file plays at the playback rate, not frame_rate / motion_speed
+        audio=audio[0].float().cpu(),
+        audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
+        output_path=f"ltx2_5_i2v_motion_speed_{motion_speed}.mp4",
+    )
+```
+
+Both clips last 121 frames (about 5 seconds at 24 fps) and only the pace of the motion changes. In [two-stage generation](#two-stage-generation-for-ltx-25), pass the same `motion_speed` to **both** stages. Stage 2 re-denoises the video, and it would undo the slow motion if it ran on the real-time axis. The [reference driver](https://huggingface.co/Lightricks/LTX-2.5-22b-LoRA-Slow-Motion-Control/blob/main/distilled_speed_demo.py) does the same by wrapping the stage object both passes share.
+
+`motion_speed` is also an input of the LTX-2 / LTX-2.5 modular blocks, and it is passed to the modular pipeline the same way.
+
 ### LTX-2.5 Modular
 
 LTX-2.5 is also available as a modular pipeline. The default blockset uses the diffusion decoder and predicts the video duration when `num_frames` is omitted. It applies guidance separately to video and audio through the `guider` and `audio_guider` components. See [`LTX2Guidance`] for the available guidance parameters. By default, the modular pipeline will download the prompt enhancer and processor from the [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) repo. Below is a T2V modular example:

src/diffusers/modular_pipelines/ltx2/before_denoise.py

@@ -25,6 +25,7 @@
 # the other shared LTX-2 data/utilities enumerated in `encoders.py`. Imported from the pipelines path here only so
 # the draft is runnable.
 from ...pipelines.ltx2.pipeline_ltx2_ic_lora import LTX2ReferenceCondition
+from ...pipelines.ltx2.utils import conditioning_frame_rate
 from ...schedulers import FlowMatchEulerDiscreteScheduler
 from ...utils import logging
 from ...utils.torch_utils import randn_tensor
@@ -752,6 +753,16 @@ def inputs(self) -> list[InputParam]:
             InputParam(
                 "frame_rate", type_hint=float, default=24.0, description="Frames per second of the generated video."
             ),
+            InputParam(
+                "motion_speed",
+                type_hint=float,
+                default=1.0,
+                description=(
+                    "Speed of the motion relative to real time. The positional embeddings are conditioned on "
+                    "`frame_rate / motion_speed`, so values below 1 slow the motion down, e.g. `0.2` for the "
+                    "Slow-Motion-Control LoRA's 5x slow motion. `frame_rate` stays the playback rate."
+                ),
+            ),
             InputParam("audio_num_frames", type_hint=int, required=True),
             InputParam.template("num_images_per_prompt", name="num_videos_per_prompt"),
             InputParam(
@@ -790,7 +801,12 @@ def __call__(self, components, state: PipelineState) -> tuple[LTX2ModularPipelin
         latent_num_frames = (block_state.num_frames - 1) // components.vae_temporal_compression_ratio + 1
 
         block_state.video_coords = components.transformer.rope.prepare_video_coords(
-            batch_size, latent_num_frames, latent_height, latent_width, device, fps=block_state.frame_rate
+            batch_size,
+            latent_num_frames,
+            latent_height,
+            latent_width,
+            device,
+            fps=conditioning_frame_rate(block_state.frame_rate, block_state.motion_speed),
         )
         block_state.audio_coords = components.transformer.audio_rope.prepare_audio_coords(
             batch_size, block_state.audio_num_frames, device
@@ -863,6 +879,16 @@ def inputs(self) -> list[InputParam]:
             InputParam(
                 "frame_rate", type_hint=float, default=24.0, description="Frames per second of the generated video."
             ),
+            InputParam(
+                "motion_speed",
+                type_hint=float,
+                default=1.0,
+                description=(
+                    "Speed of the motion relative to real time. The positional embeddings are conditioned on "
+                    "`frame_rate / motion_speed`, so values below 1 slow the motion down, e.g. `0.2` for the "
+                    "Slow-Motion-Control LoRA's 5x slow motion. `frame_rate` stays the playback rate."
+                ),
+            ),
             InputParam(
                 "noise_scale",
                 type_hint=float,
@@ -1022,7 +1048,7 @@ def __call__(self, components, state: PipelineState) -> tuple[LTX2ModularPipelin
                 keyframe_latent_width=kf_latent_width,
                 pixel_frame_idx=(latent_idx - 1) * frame_scale_factor + 1,
                 num_pixel_frames=num_pixel_frames,
-                fps=block_state.frame_rate,
+                fps=conditioning_frame_rate(block_state.frame_rate, block_state.motion_speed),
                 patch_size=spatial_patch,
                 patch_size_t=temporal_patch,
                 scale_factors=scale_factors,
@@ -1159,6 +1185,16 @@ def inputs(self) -> list[InputParam]:
             InputParam(
                 "frame_rate", type_hint=float, default=24.0, description="Frames per second of the generated video."
             ),
+            InputParam(
+                "motion_speed",
+                type_hint=float,
+                default=1.0,
+                description=(
+                    "Speed of the motion relative to real time. The positional embeddings are conditioned on "
+                    "`frame_rate / motion_speed`, so values below 1 slow the motion down, e.g. `0.2` for the "
+                    "Slow-Motion-Control LoRA's 5x slow motion. `frame_rate` stays the playback rate."
+                ),
+            ),
             InputParam(
                 "noise_scale",
                 type_hint=float,
@@ -1317,7 +1353,7 @@ def __call__(self, components, state: PipelineState) -> tuple[LTX2ModularPipelin
                 keyframe_latent_width=kf_latent_width,
                 pixel_frame_idx=(latent_idx - 1) * frame_scale_factor + 1,
                 num_pixel_frames=num_pixel_frames,
-                fps=block_state.frame_rate,
+                fps=conditioning_frame_rate(block_state.frame_rate, block_state.motion_speed),
                 patch_size=spatial_patch,
                 patch_size_t=temporal_patch,
                 scale_factors=scale_factors,
@@ -1706,6 +1742,16 @@ def inputs(self) -> list[InputParam]:
             InputParam(
                 "frame_rate", type_hint=float, default=24.0, description="Frames per second of the generated video."
             ),
+            InputParam(
+                "motion_speed",
+                type_hint=float,
+                default=1.0,
+                description=(
+                    "Speed of the motion relative to real time. The positional embeddings are conditioned on "
+                    "`frame_rate / motion_speed`, so values below 1 slow the motion down, e.g. `0.2` for the "
+                    "Slow-Motion-Control LoRA's 5x slow motion. `frame_rate` stays the playback rate."
+                ),
+            ),
             InputParam("audio_num_frames", type_hint=int, required=True),
             InputParam(
                 "appended_coords",
@@ -1750,7 +1796,12 @@ def __call__(self, components, state: PipelineState) -> tuple[LTX2ModularPipelin
         latent_num_frames = (block_state.num_frames - 1) // components.vae_temporal_compression_ratio + 1
 
         video_coords = components.transformer.rope.prepare_video_coords(
-            batch_size, latent_num_frames, latent_height, latent_width, device, fps=block_state.frame_rate
+            batch_size,
+            latent_num_frames,
+            latent_height,
+            latent_width,
+            device,
+            fps=conditioning_frame_rate(block_state.frame_rate, block_state.motion_speed),
         )
         block_state.video_coords = torch.cat([video_coords, block_state.appended_coords.to(video_coords.dtype)], dim=2)
         block_state.audio_coords = components.transformer.audio_rope.prepare_audio_coords(

src/diffusers/modular_pipelines/ltx2/denoise.py

@@ -21,6 +21,7 @@
 from ...configuration_utils import FrozenDict
 from ...guiders import LTX2Guidance
 from ...models import LTX2VideoTransformer3DModel
+from ...pipelines.ltx2.utils import conditioning_frame_rate
 from ...schedulers import FlowMatchEulerDiscreteScheduler
 from ..modular_pipeline import (
     BlockState,
@@ -319,6 +320,16 @@ def inputs(self) -> list[InputParam]:
             InputParam(
                 "frame_rate", type_hint=float, default=24.0, description="Frames per second of the generated video."
             ),
+            InputParam(
+                "motion_speed",
+                type_hint=float,
+                default=1.0,
+                description=(
+                    "Speed of the motion relative to real time. The positional embeddings are conditioned on "
+                    "`frame_rate / motion_speed`, so values below 1 slow the motion down, e.g. `0.2` for the "
+                    "Slow-Motion-Control LoRA's 5x slow motion. `frame_rate` stays the playback rate."
+                ),
+            ),
             InputParam(
                 "use_cross_timestep",
                 type_hint=bool,
@@ -361,7 +372,7 @@ def __call__(
             num_frames=latent_num_frames,
             height=latent_height,
             width=latent_width,
-            fps=block_state.frame_rate,
+            fps=conditioning_frame_rate(block_state.frame_rate, block_state.motion_speed),
             use_cross_timestep=block_state.use_cross_timestep,
             attention_kwargs=block_state.attention_kwargs,
             perturbation_mask=None,

src/diffusers/modular_pipelines/ltx2/encoders.py

@@ -44,6 +44,7 @@
     LTX2_5_I2V_DEFAULT_SYSTEM_PROMPT,
     LTX2_5_T2V_DEFAULT_SYSTEM_PROMPT,
     apply_image_conditioning_crf,
+    conditioning_frame_rate,
     resolve_default_image_crf,
 )
 from ...utils import logging
@@ -1197,6 +1198,16 @@ def inputs(self) -> list[InputParam]:
             InputParam(
                 "frame_rate", type_hint=float, default=24.0, description="Frames per second of the generated video."
             ),
+            InputParam(
+                "motion_speed",
+                type_hint=float,
+                default=1.0,
+                description=(
+                    "Speed of the motion relative to real time. The positional embeddings are conditioned on "
+                    "`frame_rate / motion_speed`, so values below 1 slow the motion down, e.g. `0.2` for the "
+                    "Slow-Motion-Control LoRA's 5x slow motion. `frame_rate` stays the playback rate."
+                ),
+            ),
             InputParam.template("generator"),
         ]
 
@@ -1285,7 +1296,7 @@ def __call__(self, components, state: PipelineState) -> tuple[LTX2ModularPipelin
                 height=ref_latent_height,
                 width=ref_latent_width,
                 device=device,
-                fps=block_state.frame_rate,
+                fps=conditioning_frame_rate(block_state.frame_rate, block_state.motion_speed),
             )
             if downscale_factor != 1:
                 ref_coords[:, 1, :, :] = ref_coords[:, 1, :, :] * downscale_factor