weight is not tied within `init_empty_weights`

[Ghost-LZW]

System Info

- `Accelerate` version: 1.8.1
- Platform: macOS-14.3.1-arm64-arm-64bit
- `accelerate` bash location: /opt/homebrew/anaconda3/bin/accelerate
- Python version: 3.11.3
- Numpy version: 1.24.3
- PyTorch version: 2.4.1
- PyTorch accelerator: N/A
- System RAM: 32.00 GB

Information

• The official example scripts
• My own modified scripts

Tasks

• One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
• My own task or dataset (give details below)

Reproduction

from accelerate import init_empty_weights
import torch

a = torch.nn.ModuleList([torch.nn.Embedding(12, 12), torch.nn.Embedding(12, 12)])
# tie embedding
a[0].weight = a[1].weight
assert a[0].weight is a[1].weight  # success

with init_empty_weights():
    a = torch.nn.ModuleList([torch.nn.Embedding(12, 12), torch.nn.Embedding(12, 12)])
    # tie embedding
    a[0].weight = a[1].weight
    assert a[0].weight is a[1].weight  # error
"""
Traceback (most recent call last):
  File "test_empty_init.py", line 16, in <module>
    assert a[0].weight is a[1].weight
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
"""

Expected behavior

tie weights success with init_empty_weights.

This issue might come from the register_empty_parameter move the parameter to the meta device without checking whether the parameters are tied.

https://github.com/huggingface/accelerate/blob/main/src/accelerate/big_modeling.py#L129-L135

[Ghost-LZW]

There is an existing issue https://discuss.huggingface.co/t/error-the-model-weights-are-not-tied-please-use-the-tie-weights-method-before-using-the-infer-auto-device-function-even-after-adding-model-tie-weights/46325

[SunMarc]

We indeed lose tied weights link when initializing with init_empty_weights(). This is why users have to call model.tie_weights() afterwards. There were a PR that take care of that, if you want to have a look or finish it ! #2737

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.