Fix transform benchmarks #1118
Fix transform benchmarks #1118
Conversation
meta-cla
bot
added
the
CLA Signed
| parser.add_argument( | ||
| "--num-threads", | ||
| type=int, | ||
| default=1, | ||
| help="number of threads to use; 0 means FFmpeg decides", | ||
| ) |
There was a problem hiding this comment.
We might want to also call torch.set_num_threads(args.num_threads) when num_threads != 0? In the current conditions of the benchmark, I think torch's resize isn't multi-threaded, so this should have no effect. But there are code paths where it is multithreaded over the batch dimension, depending on the input dtype and the interpolation mode (example: https://github.com/pytorch/pytorch/blame/afb173d9b9440d804b5f77d0c291e53c720d1fcf/aten/src/ATen/native/cpu/UpSampleKernel.cpp#L2024C18-L2024C18).
There was a problem hiding this comment.
If it was as clean as just providing this value to the Torch APIs, I think we should just do it now. But because of the weirdness with 0 (I don't think torch.set_num_threads() has an equivalent for automatic deciding), I think we may want to control it with another flag or do some logic (n_cpus // 2). For those reasons, I'd rather punt on that until we want to get numbers for those scenarios.
2 participants
This PR changes the transform benchmarks to:
VideoDecoderAPI instead of the core API as originally implemented; see Decoder-native transforms benchmark #982. The benchmarks were implemented before the public API existed.The last point is the most important, as the previously reported benchmarks were, unintentionally, using 0 as the number of FFmpeg threads. That meant FFmpeg would decide, which usually uses half of the available cores.
Runtime Performance
I'm going to drop two large batch of results because I think it's useful for posterity. First, I'm using this video:
With threads set to 0, where FFmpeg will use half of my system:
And then with 1 thread:
I'm going to focus on the difference between the largest number of frames sampled with the largest size reduction with 0 and 1 threads:
Importantly, the number of threads changes the relative costs between using decoder transforms versus passing the fully decoded frame to TorchVision. I cannot yet fully explains these results, but possible factors are:
sws_scale()in swscale. We know TorchVision's resize is SIMDized, and libswscale appears to be as well. (See: https://github.com/FFmpeg/FFmpeg/tree/master/libswscale/x86.)format=rgb24before the transforms, including resize. This ensures that the transforms are applied in the output colorspace, but it also forces an extra call tosws_scale(). Not forcing this buys about 7-10% of the performance back. However, in principle, this should be basically the same as just applying the TorchVision resize after a normal decoding.Memory Performance
Here's the good news: decoder transforms are radically more memory efficient. I haven't been able to instrument the benchmarks yet to capture this, but I can observe it through top.
If I hack the benchmark to only run
torchvision_resize(), then the RSS grows to 4.3 GB in cycles. That is, it gets that high, goes down way less than 1 GB, then grows again. I think we're observing the Python garbage collector kicking in.If I hack the benchmark to only run
decoder_resize(), then the RSS stays at 0.4 MB the entire time.