[Benchmarks] index

[stas00]

This issue is to document the important transformers benchmarks in one place, so that they are easy to find.

To add a new benchmark entry post it in an Issue (separately or as a comment in an existing issue) and then link from here. If you have edit rights please add a link directly to this post, otherwise please add a note in the comments and I will update this post.

Please do not post actual benchmarks in the comments of this Issue. This is only an index.

Thank you!

Fastest speed combinations

• RTX-3090
• A100

Precision: fp16 vs bf16 vs tf32 vs fp32

• RTX-3090
• A100

Batch size / gradient accumulation steps

• gradient accumulation steps: RTX-3090, A100
• batch size RTX-3090, A100

Gradient checkpointing

• RTX-3090
• A100

Optimizers:

• Adam torch vs. apex vs HF vs adafactor: RTX-3090, A100
• re-run the above a year later with the same list of optimizers, plus BNB's 8bit optimizer and fused torch AdamW PCIe 80GB A100

Network / Interconnects:

• DP/DDP/NVLink