fix: random sampling in ForgetRetainDataset #145
Conversation
src/data/unlearn.py
Outdated
| g = torch.Generator() | ||
| rank = torch.distributed.get_rank() if torch.distributed.is_initialized() else 0 | ||
| seed = int(torch.empty((), dtype=torch.int64).random_().item() + rank) | ||
| g.manual_seed(seed) |
There was a problem hiding this comment.
it would be better to use the seed from the experiment config here, rather than
int(torch.empty((), dtype=torch.int64).random_().item() to avoid introducing randomness uncontrolled by the seed.
can you try to see if you can make the experiment's cfg.seed available to this dataset class and then use seed = exp_seed + rank here?
|
Thanks for the feedback! I've updated the PR accordingly. Please let me know if there are any further adjustments required. |
|
Please fix the lint errors! |
There was a problem hiding this comment.
It is not ideal to set the seed at the exact example level. This would mean we select the same retain example index sequences even if we are using a different dataset.
Since the point is that each rank must get a different seed, imo it is better to get the rank in the global seed function: https://github.com/locuslab/open-unlearning/blob/main/src/trainer/utils.py#L8
Let me know if you see any issues.
2 participants
What does this PR do?
Fixes #139
Before submitting