GGUF none-compile support

[zwukong]

GGUF now needs compile in every running, very slow. Can you use the similar codes like comfyui-GGUF to help our low-vram users. GGUF is very important for us 🤗
https://github.com/city96/ComfyUI-GGUF

[DN6]

Hi @zwukong GGUF checkpoints should work with torch compile if you're using torch>=2.7.1. e.g

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel, GGUFQuantizationConfig

model_id = "black-forest-labs/FLUX.1-dev"
ckpt_path = (
    "https://huggingface.co/city96/FLUX.1-dev-gguf/blob/main/flux1-dev-Q2_K.gguf"
)
transformer = FluxTransformer2DModel.from_single_file(
    ckpt_path,
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
)
transformer = torch.compile(transformer, fullgraph=True)

pipe = FluxPipeline.from_pretrained(
    model_id, transformer=transformer, torch_dtype=torch.bfloat16
)
pipe.to("cuda")
images = pipe("A cat holding a sign that says hello").images[0]
images.save("flux-gguf-compile.png")

[zwukong]

Thanks for your reply, what i want is no compile. when gguf runs ,always compile even if you are not using torch.compile

[DN6]

Sorry @zwukong I'm not sure I follow. Are you saying running GGUF without torch compile is slow? Do you have a script or code example I can use to reproduce your issue?

[zwukong]

Your script without compile. run multi times then you will know how slow it is. always compiling. Try comfyui-GGUF,then you will know the difference. How fast comfyui version is

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel, GGUFQuantizationConfig

model_id = "black-forest-labs/FLUX.1-dev"
ckpt_path = (
    "https://huggingface.co/city96/FLUX.1-dev-gguf/blob/main/flux1-dev-Q2_K.gguf"
)
transformer = FluxTransformer2DModel.from_single_file(
    ckpt_path,
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
)

pipe = FluxPipeline.from_pretrained(
    model_id, transformer=transformer, torch_dtype=torch.bfloat16
)
pipe.to("cuda")
images = pipe("A cat holding a sign that says hello").images[0]
images.save("flux-gguf-compile.png")