Skip to content

Convert load_video frames to RGB when convert_method is None - #15099

Draft
nettanvirdev wants to merge 1 commit into
huggingface:mainfrom
nettanvirdev:fix-load-video-rgb
Draft

nettanvirdev wants to merge 1 commit into
huggingface:mainfrom
nettanvirdev:fix-load-video-rgb

Conversation

@nettanvirdev

@nettanvirdev nettanvirdev commented Oct 21, 2026 •

Copy link
Copy Markdown

What does this PR do?

load_video() documents that frames are converted to "RGB" when convert_method=None, but it returned them unchanged. For GIFs, Pillow decodes the first frame as P and later frames as RGB/RGBA, so the mixed modes made VideoProcessor.preprocess_video() and AnimateDiffVideoToVideoPipeline fail with ValueError: all input arrays must have the same shape.

This adds the missing else branch so every frame is converted with image.convert("RGB"), the same as load_image. A custom convert_method is still applied as before and skips the conversion.

Fixes #15098

Notes:

  • The conversion happens once after both decode paths. For mp4/imageio frames it is a no-op on the pixels (they are already RGB arrays); I checked a locally written mp4 and the frames are identical to the raw imageio frames.
  • Transparent GIFs (P then RGBA frames) now come back as RGB, matching what load_image returns for the same frames.
  • No "Copied from" counterpart of load_video exists, so nothing else needs updating. No docs change needed; the docstring already describes this behavior.

Tests

Added TestLoadVideo::test_gif_frames_are_rgb in tests/others/test_utils.py.

$ pytest tests/others/test_utils.py -k TestLoadVideo
# on main
E         At index 0 diff: 'P' != 'RGB'
FAILED tests/others/test_utils.py::TestLoadVideo::test_gif_frames_are_rgb
# with this PR
1 passed, 26 deselected

$ pytest tests/others/test_utils.py tests/others/test_video_processor.py
1 failed, 33 passed, 1 skipped

The one failure is TestDeprecate::test_deprecate_stacklevel, which also fails on main on Windows (it asserts a /-separated path against a \-separated filename).

Also ran ruff check / ruff format --check (0.9.10), utils/check_copies.py, utils/check_dummies.py, utils/check_doc_toc.py and doc-builder style --check_only: all clean.

Before submitting

  • Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
    • Did you read the Coding with AI agents guide?
    • Did you run the self-review skill on the diff?
    • Did you share the final self-review notes in the PR description or a comment?
  • Did you read the contributor guideline?
  • Did you read our philosophy doc? (important for complex PRs)
  • Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?
  • Are you the author (or part of the team) of the model/pipeline (only applicable for model/pipeline related PRs)?

Who can review?

@sayakpaul @DN6

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

load_video returns mixed-mode frames for GIFs, breaking preprocess_video and AnimateDiff video-to-video

1 participant