XLora for diffusers included flux sd3 and sd models

[Onkarsus13]

No description provided.

[githubnemo]

Hey @Onkarsus13, thanks for your contribution.

Can you please add a little context what this change entails?
Also, there are a lot of changes, is this intentional? It looks like the changes are from older PEFT versions.

[Onkarsus13]

Hey @Onkarsus13, thanks for your contribution.

Can you please add a little context what this change entails? Also, there are a lot of changes, is this intentional? It looks like the changes are from older PEFT versions.

I have updated the current version of peft itself, previously XLoRA is not for the diffusers, we can use XLoRA on SD-Models like SD-XL, SD-3, or FLUX or any video generation model by this PR we can able to integrate X-LORA in it

Below example shows that how we can use x-lora in Diffusers.

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel
from peft import PeftModel
from transformers import AutoConfig
from peft import XLoraConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training

device = torch.device("cuda:0")

transformer = FluxTransformer2DModel.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    cache_dir='/data2/flux_model',
    subfolder='transformer',
    torch_dtype=torch.bfloat16,
    device=device
)

config = XLoraConfig(
    task_type=None,
    hidden_size=3072,
    xlora_depth=2,
    layerwise_scalings=True,
    adapters={
        "adapter_1": "<Adapter 1>",
        "adapter_2": "<Adapter 2>",
    },

)

xlora_model = get_peft_model(transformer, config)

[githubnemo]

Thank you for giving a bit of context. It seems that you based your solution off of PEFT 0.13.2 and a lot has happened since then. Can you apply your changes on the main branch so that the diff is smaller?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[githubnemo]

@Onkarsus13 do you still plan on working on this?

[Onkarsus13]

Yes I am still working on it
But in new peft verision i need to change a lot But still i am trying to give PR by Next week

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[githubnemo]

not stale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[githubnemo]

not stale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.