Test BART's memory consumption

[sshleifer]

• this can run on GPU only and be marked @slow
• check how much memory bart is using at __init__
• assert that it doesn't use more than 110% of that.
• check how much memory bart uses on a single forward pass. (optionally test this in fp16).
• assert that it doesn't use more than 110% of that.
• check how much memory bart uses on a single forward and backward pass.
• assert that it doesn't use more than 110% of that.

Bonus:

• add similar asserts for timing!
• let the test run and check memory on CPU (make sure that if pytest is run with -n -8 the test still passes!
• add a test to test_modeling_common.py to make it easy for all models to test this.
• add a test to test_modeling_common_tf.py to make it easy for all TF models to test this.

The benchmarking utilities may help.
It may also help to use torch.cuda.max_memory...

@patrickvonplaten may have further thoughts!

[sshleifer]

@stas00 , this might be up your alley!

[stas00]

Excellent, will work on that.

Thank you, @sshleifer!

[stas00]

I will post my work in progress here, so others could chime in with ideas.

This notebook measures memory consumption and speed of __init__, a single fwd and fwd+bwd pass
https://colab.research.google.com/drive/1n6J3tc8FT4ER1vBCTAtU4__U5Px2mxwI?usp=sharing

The current difficulty is how to establish a baseline for memory and speed, so that it can be validated against in the future - detecting any regressions.

Several issues have been encountered:

1. there is a large discrepancy between results on different GPU cards. Currently tested my own Titan X and Colab's Tesla T4:

The following data is for MODEL_ID = "sshleifer/tinier_bart"

Memory:

func	T4	Titan X
init	0	0
fwd	22MB	62MB
fwd-bwd	22MB	70MB

Currently only GPU-memory was measured and the initial loading of cudnn memory was subtracted. The latter varies hugely - ~600MB on Titan X and ~1GB on T4.

Exec speed:

func	T4	Titan X
init	0.05978s	0.03808s
fwd	0.00375s	0.00273s
fwd-bwd	0.00904s	0.00563s

Speed measurements are an average of 30 (!) runs.

The following data is for MODEL_ID = "tacebook/bart-base"

Memory:

func	T4	Titan X
init	0	0
fwd	596MB	636MB
fwd-bwd	1576MB	1624MB

Exec speed:

func	T4	Titan X
init	3.28328s	2.24100s
fwd	0.01698s	0.00998s
fwd-bwd	0.03551s	0.03039s

2. Inconsistent results on the same card:

just one function of fwd+bwd on Tesla T4:

(memory is the same)

Exec time:

run 1: 0.00904s
run 2: 0.00769s
run 3: 0.00740s
run 4: 0.00768s
run 5: 0.00753s

the notebook was fully restarted for each measurement. Each report is an average of 30 runs.

Randomness is not fixed, but the input data is small and fake so it shouldn't make much of a difference.

So, given that most users have different cards, how can we approach making such a test that would work for everybody (or enough people to warrant this test's usefulness).

What would also help if you could download the notebook and share which measurement numbers you get and your card name. So that we could compare different cards. Though perhaps just one or 2 - the real testing will be on the full models and not the tiny ones. Thank you.

[sshleifer]

Is the variance similar for larger models?

[stas00]

I updated the comment above to add the same data for the full bart-base model.

The difference in memory seems to be "relatively" fixed - i.e. larger model still has about the same diff as the small one between two cards memory-wise.

Speed on the other hand has a much wider variation. Note, I added a comment that speed measurements are already an average of 30 re-runs.

[patrickvonplaten]

Hey @stas00,

Thanks for running the benchmarks. As discussed, I think the way to go here is:

1) GPU Memory:
Slightly adapt the memory measurement to return 3 results:

• Required mem for CUDA/CUDNN init
• Required mem to run function (fwd or bwd)
• Sum of both these functions (this number should be the one that is returned now)
  => so we only need to add a function that measures how much memory is required to load CUDA/CUDNN kernels into GPU.

Then, I think we should report GPU mem requirement for a fixed GPU with fixed CUDA/CUDNN library (at the moment the CUDA/CUDNN version is not reported when running the benchmark, but they should be IMO) and make sure that these numbers are stable. Then we can loosen restrictions on different libraries and see how performance changes I guess and then finally compare on different GPUs.

2. Speed measurements:

My measurements on notebooks regarding speed also always varied a lot, but was very stable on one of our GPUs...not sure why the notebooks were much more unstable. I also think we can reduce 30 to something like 10, which should be fine and maybe always do 2-5 warmup runs (this is currently only done for TPU/torchscript)? What do you think?

Here we should again start with same libraries / same GPU IMO.

[stas00]

Your proposals sound good to me, @patrickvonplaten

Based on my experimentation and your input - it seems that stable speed measurements are the tricky part here.

It looks like this issue (including speed measurements) is now where the discussion is on: #6218 So perhaps let's continue there and return to this issue once there is an updated API and then we can proceed tuning this one.

The complication here is that different cards give different results and we may have to do the painful process of maintaining different baselines for different cards.

[stas00]

Unless someone would like to contribute some new ideas, this issue is stalled at the moment. The data is too divergent between different cards to have a solid test that also does a proper regression measurement.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.