[valid] deadlock workaround

[stas00]

For the full discussion please see: https://huggingface.slack.com/archives/C01NHER1JLS/p1652884593515729
The summary is here https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles.md#2022-05-18-hanging-at-eval-redux

1. We again run into a deadlock on the 2nd valid loss entry
2. This time the srun py-spy tracer worked perfectly so we had all the info needed to analyse the problem
3. @DanielHesslow discovered that the deadlock happens in DataLoader - i.e. pytorch bug Fix possible deadlock in SharedCache inside a forked child proc pytorch/pytorch#25158 - which was supposed to be fixed, but it's not - the race condition is still there - and other recent comments confirm that.
4. So based on Daniel's suggestion we set the valid_dataloaders to use num_workers=0 to work around it - it slows things down by a 5 TFLOPs on training, so we kept the training at num_workers=2

We are currently running already on this code as I hacked directly the local branch we were running from #272

@TevenLeScao, @thomasw21

[thomasw21]

LGTM. I think ideally we'd want to expose that via an command line argument, but the hotfix makes sense right now. Thanks!

[stas00]

Indeed, I will work on the cmd line arg before merging.

[stas00]

Done.

I will rebase the live auto-sync branch and adjust the slurm script to add --valid-num-workers 0 once we merge this.

Thank you!

[stas00]

so I'm merging this, right? not sure if we are going to hear from @TevenLeScao

[TevenLeScao]

Sorry, don't really have the bandwidth to check this, go ahead if it's necessary

[thomasw21]

Looks alright to me! Feel free to merge! Thanks a lot!