PEFT Loha/Lokr crashes for SDXL pipeline inference #2783
Description
System Info
The SDXL pipeline inference crashes when apply a Lokr/Loha patch on it. I've tested SD1.5, SANA and Pixart-Sigma and they worked perfectly fine with a Lokr patch.
Traceback (most recent call last):
File "/home/lucas/Documents/YAT/test_sdxl_peft_bug.py", line 32, in <module>
image = pipeline(prompt=prompt,
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lucas/Documents/YAT/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/lucas/Documents/YAT/.venv/lib/python3.12/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 1139, in __call__
add_time_ids = self._get_add_time_ids(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lucas/Documents/YAT/.venv/lib/python3.12/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 742, in _get_add_time_ids
expected_add_embed_dim = self.unet.add_embedding.linear_1.in_features
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lucas/Documents/YAT/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1962, in __getattr__
raise AttributeError(
AttributeError: 'Linear' object has no attribute 'in_features'
Who can help?
No response
Reproduction
from diffusers import StableDiffusionXLPipeline
import torch
from peft import LoHaConfig, get_peft_model
pipeline = StableDiffusionXLPipeline.from_single_file(
"https://huggingface.co/cyberdelia/CyberRealisticXL/blob/main/CyberRealisticXLPlay_V7.0_FP16.safetensors",
torch_dtype=torch.float16
).to("cuda")
lora_target_modules = [
'linear_1',
'linear_2'
'conv1',
'conv2',
'conv',
'to_q',
'to_k',
'to_v',
'to_out.0',
'ff.net.2',
'proj_out',
'conv_shortcut',
'time_emb_proj']
config = LoHaConfig(r=16,
target_modules=lora_target_modules,
alpha=16)
pipeline.unet = get_peft_model(pipeline.unet, config).to(dtype=torch.float16)
prompt = "epic landscape"
image = pipeline(prompt=prompt,
guidance_scale=4.0,
num_inference_steps=20).images[0]
image.save('epic_landscape.png')
Expected behavior
The code should run as expected (the same as with the SD1.5/SANA/Pixart-Sigma pipelines).