Add new method: GLoRA

[HsunGong]

Feature request

See: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA

Motivation

See: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA
A new method for PEFT

Your contribution

Could be

[kumar-devesh]

Hi @HsunGong, will you be working on contributing GLoRA? I would like to work on it as well!

[kumar-devesh]

@younesbelkada @BenjaminBossan I would like to work towards adding this to peft.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[githubnemo]

This was adressed in #2568 but wasn't completed. To quote @not-lain:

imo the only good thing to retain from this pr [#2568] is the config file, other scripts are too messy and unusable.

Therefore this is now open again and contributions are welcome. I think it's a good idea to take the config file as suggested and start from there. Some other things from that PR can probably be salvaged as well.

[kumar-devesh]

Hey @githubnemo , I would like to take this up this weekend. Whats the closet to GLoRA peft method that i can use for reference?