In LoRA, should we add scaling factor of alpha/rank?

[Jessen-Li]

$$W_{\text{updated}} = W + AB$$,
Should it be $$W_{\text{updated}} = W + alpha/rank*AB$$ instead?

if so, the following code:

class LoRALayer(torch.nn.Module):
    def __init__(self, in_dim, out_dim, rank, alpha):
        super().__init__()
        self.A = torch.nn.Parameter(torch.empty(in_dim, rank))
        torch.nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))  # similar to standard weight initialization
        self.B = torch.nn.Parameter(torch.zeros(rank, out_dim))
        self.alpha = alpha

    def forward(self, x):
        x = self.alpha * (x @ self.A @ self.B)
        return x

should be changed to:

    def forward(self, x):
        x = self.alpha/rank * (x @ self.A @ self.B)
        return x

?

[rasbt]

Thanks for commenting, coincidentally, it just came up a few days ago here: #821