Describe the bug
Trying to profile the ltx2 pipeline encountered an error:
Root Cause: Interaction of Two Factors
When wrapping pipeline component methods (e.g., scheduler.step) using a decorator (annotate), the method is retrieved as a bound method:
|
method = getattr(component, method_name, None) |
|
if method is None: |
|
continue |
|
setattr(component, method_name, annotate(method, label)) |
The decorator then wraps this bound method, capturing its __self__ (the original scheduler instance) inside the closure.
As a result:
- The wrapped step becomes a plain function, not a descriptor.
- The original self is frozen inside the wrapper.
Later, when duplicating the scheduler (a peculiarity of the ltx2 pipeline):
|
audio_scheduler = copy.deepcopy(self.scheduler) |
the wrapped function is copied as-is, including the captured reference to the original scheduler.
As a result: calling audio_scheduler.step(...) actually executes self.scheduler.step(...):
|
latents = self.scheduler.step(noise_pred_video, t, latents, return_dict=False)[0] |
|
# NOTE: for now duplicate scheduler for audio latents in case self.scheduler sets internal state in |
|
# the step method (such as _step_index) |
|
audio_latents = audio_scheduler.step(noise_pred_audio, t, audio_latents, return_dict=False)[0] |
This leads to:
- Shared internal state (_step_index) between supposedly independent schedulers
- Incorrect stepping behavior
- Index out-of-bounds errors during inference
The following shows debugger output from the first loop iteration, right after executing audio_scheduler.step():
The object ID inside both step calls matches the ID of self.scheduler:
Fix Proposal
Decorate the unbound function and rebind it:
for component_name, method_name, label in annotations:
component = getattr(pipe, component_name, None)
if component is None:
continue
component_cls = type(component)
method = getattr(component_cls, method_name, None)
if method is None:
continue
setattr(component, method_name, annotate(method, label).__get__(component, component_cls))
I can open a PR if this solution is suitable.
Reproduction
Run ltx2 profiling for any number of steps:
python examples/profiling/profiling_pipelines.py --pipeline ltx2 --mode eager --num_steps 2
Logs
System Info
Diffusers version: 0.38.0.dev0
- Platform: Linux-6.8.0-84-generic-x86_64-with-glibc2.39
- Running on Google Colab?: No
- Python version: 3.12.3
- PyTorch version (GPU?): 2.10.0+cu128 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 1.9.2
- Transformers version: 5.5.1
- Accelerate version: 1.13.0
- PEFT version: not installed
- Bitsandbytes version: not installed
- Safetensors version: 0.7.0
- xFormers version: not installed
- Accelerator: NVIDIA A100-SXM4-80GB, 81920 MiB
Who can help?
@sayakpaul, @dg845
Describe the bug
Trying to profile the ltx2 pipeline encountered an error:
Root Cause: Interaction of Two Factors
When wrapping pipeline component methods (e.g., scheduler.step) using a decorator (annotate), the method is retrieved as a bound method:
diffusers/examples/profiling/profiling_utils.py
Lines 45 to 48 in 5063aa5
The decorator then wraps this bound method, capturing its
__self__(the original scheduler instance) inside the closure.As a result:
Later, when duplicating the scheduler (a peculiarity of the ltx2 pipeline):
diffusers/src/diffusers/pipelines/ltx2/pipeline_ltx2.py
Line 1172 in 5063aa5
the wrapped function is copied as-is, including the captured reference to the original scheduler.
As a result: calling
audio_scheduler.step(...)actually executesself.scheduler.step(...):diffusers/src/diffusers/pipelines/ltx2/pipeline_ltx2.py
Lines 1402 to 1405 in 5063aa5
This leads to:
The following shows debugger output from the first loop iteration, right after executing
audio_scheduler.step():The object ID inside both step calls matches the ID of
self.scheduler:Fix Proposal
Decorate the unbound function and rebind it:
I can open a PR if this solution is suitable.
Reproduction
Run ltx2 profiling for any number of steps:
python examples/profiling/profiling_pipelines.py --pipeline ltx2 --mode eager --num_steps 2Logs
System Info
Diffusers version: 0.38.0.dev0
Who can help?
@sayakpaul, @dg845