load_video returns mixed-mode frames for GIFs, breaking preprocess_video and AnimateDiff video-to-video #15098

Describe the bug

diffusers.utils.load_video() documents that frames are converted to "RGB" when convert_method=None, but it never converts them. For GIFs, Pillow decodes the first frame as mode P and later frames as RGB (or RGBA if the GIF has transparency), so load_video("x.gif") returns frames with mixed modes, e.g. ['P', 'RGB', 'RGB'].

Passing that list to VideoProcessor.preprocess_video() fails with ValueError: all input arrays must have the same shape, because the P frame becomes a 2D array and the others 3D. The same error is raised by AnimateDiffVideoToVideoPipeline, so the documented example in docs/source/en/api/pipelines/animatediff.md, which calls load_video on a .gif, is affected.

Root cause: the if convert_method is not None: block at loading_utils.py#L144-L145 has no else branch, unlike load_image, which calls image.convert("RGB") in that case.

I have a two-line fix (an else branch that converts every frame to RGB, matching load_image) with a regression test ready, and I'm happy to open a PR.

Reproduction

import os
import tempfile

import PIL.Image

from diffusers.utils import load_video
from diffusers.video_processor import VideoProcessor


# A 3-frame GIF made locally, no downloads.
path = os.path.join(tempfile.mkdtemp(), "video.gif")
frames = [PIL.Image.new("RGB", (64, 64), color) for color in ("red", "green", "blue")]
frames[0].save(path, save_all=True, append_images=frames[1:], duration=100)

video = load_video(path)
print("modes:", [frame.mode for frame in video])

out = VideoProcessor(vae_scale_factor=8).preprocess_video(video, height=64, width=64)
print("preprocess_video:", tuple(out.shape))

Expected: all frames are RGB (as the docstring says) and preprocess_video returns a tensor of shape (1, 3, 3, 64, 64).

Actual: frame modes are ['P', 'RGB', 'RGB'] and preprocess_video raises (see logs).

Logs

modes: ['P', 'RGB', 'RGB']
Traceback (most recent call last):
  File "repro.py", line 18, in <module>
    out = VideoProcessor(vae_scale_factor=8).preprocess_video(video, height=64, width=64)
  File "diffusers\src\diffusers\video_processor.py", line 86, in preprocess_video
    video = torch.stack([self.preprocess(img, height=height, width=width, **kwargs) for img in video], dim=0)
                         ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "diffusers\src\diffusers\image_processor.py", line 695, in preprocess
    image = self.pil_to_numpy(image)  # to np
  File "diffusers\src\diffusers\image_processor.py", line 168, in pil_to_numpy
    images = np.stack(images, axis=0)
  File ".venv\Lib\site-packages\numpy\_core\shape_base.py", line 458, in stack
    raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

System Info

• 🤗 Diffusers version: 0.42.0.dev0 (main at 0c7d56d)
• Platform: Windows-11-10.0.26200-SP0
• Running on Google Colab?: No
• Python version: 3.14.7
• PyTorch version (GPU?): 2.14.1+cpu (False)
• Huggingface_hub version: 2.2.0
• Transformers version: 5.20.2
• Accelerate version: 1.15.0
• PEFT version: not installed
• Safetensors version: 0.8.0
• xFormers version: not installed
• Accelerator: NVIDIA GeForce RTX 4070, 12282 MiB
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No
• Pillow version: 13.0.0
• NumPy version: 2.5.4

Who can help?

@DN6 @a-r-r-o-w