insertedSamplesForDeceleration and removedSamplesForAcceleration of inbound stats confusing

[hamishwillee]

I'm working on the docs for MDN in mdn/content#40495, and the specification isn't very clear (to me):

When playout is slowed down, this counter is increased by the difference between the number of samples received and the number of samples played out. If playout is slowed down by inserting samples, this will be the number of inserted samples.

When is playout slowed down and when is it sped up - i.e. what is the context here.
This might imply that we're choosing to slow the audio that people hear, which seems unlikely.
What I think it means is that the playout of the jitter buffer is slowed down because we're not getting enough samples. To make the audio appear like it isn't being affected we're inserting synthesized samples. Is that right?

But what are recipients supposed to do with this information? Look at the rate of this changing and use that as a measure of link quality or something?

Ditto for removedSamplesForAcceleration - what could cause more samples than we can play out? What are users supposed to do with the information?

There was some attempt to ask these questions in #146 (comment) and #151 (comment))., but I don't think there were answers.

Note, the current guess without any "what is it useful for" infomraiton is:

[alvestrand]

The purpose of these counters is basically to figure out the number of cases where the jitter buffer is too small.
This can be used (when slicing on connection type, geography, type of user and other conditions) to figure out whether the jitter buffer configuration needs to be changed for certain demographics.

Very many stats are useless in the individual call, but of great use when aggregated over all the calls in a service.

[alvestrand]

Hmm... I think I got the wrong numbers; this is not about jitter buffer overflow.
These numbers are useful when the sender's clock and the receiver's clock are not running at the same speed; if, for instance, the sender sends 8001 samples per second and the receiver plays them out at 7999 samples per second, we need to delete two samples from the input stream every second in order to ensure that the playout doesn't drift away from "real time".

[fippo]

Hmm... I think I got the wrong numbers;

My interpretation has been that neteq speeds up or removes to fill gaps and these stats tell you how much. https://webrtchacks.com/how-webrtcs-neteq-jitter-buffer-provides-smooth-audio/ explains a lot of it.

[hamishwillee]

Thanks @alvestrand @fippo . I'm not seeing a lot of certainty, and it is concerning that there is no way to infer this information from the spec.

That article is awesome and explains a lot. My interpretation of this statistic is the same as ^^^. The jitter buffer attempts to keep the playout rate constant given a varying packet arrival delay, packet loss, errored packets and so on. It does varying things to tune the buffer buffer size, and the packetization length to allow this.

One of the tricks it does is attempt to target a particular delay that keeps the input and output flow pretty even. It then matches the current playout delay to the target delay (see the better way). If there is no "next packet" it creates a concealment sample. If there is a packet but the current delay is too early or too late it may slow down or speed up the playout of sample. The inference is that this is done by inserting samples or removing them from the buffer, but might be done in some other way (?)

The original statement s you made @alvestrand still seems true-ish "The purpose of these counters is basically to figure out the number of cases where the jitter buffer is too small.". Also differing clocks on sender and receiver might presumably lead to effectively a persistently greater drain than inflow (and visa versa).

Would it be more true to say:

The insertedSamplesForDeceleration property of the RTCInboundRtpStreamStats dictionary accumulates the difference between the number of samples received and the number of samples played out of the jitter buffer while audio playout is slowed down.

The WebRTC jitter buffer sets a target playout level where the inflow and outflow of the jitter buffer are approximately equal.
If the jitter buffer empties too quickly the audio sample that is next in line to be output may be "ahead of schedule".
In this case the engine may slow down the playout of that sample by inserting additional audio samples.
This value indicates the accumulated number of such additional samples.

The value might be correlated with totalSamplesReceived to get a relative measure of deceleration.
You might also track the changes in insertedSamplesForDeceleration over the duration of the connection.
Consistently high and increasing values suggests that playout is consistently running too fast relative to the incoming audio stream.
This might lead to an accumulation of samples in the jitter buffer, increasing latency.

Any advice?

[vr000m]

One of the practical outcomes is that the user will report that the audio sounds warbled or like donald duck (sped up). If you have logs of timeslices where the inserted or removed counters bump up, gives you a notion on when that issue happened.

The totals at the end of the call also give you a notion of how many samples or seconds were impacted. If the device polls and logs the changes at some intervals, say every 10s, then you have a notion if this happened intermittently or every few seconds in the call. And you could correlate a particular metric in a time slice with other metrics in that timeslice (e.g., packet loss, network change, cpu, etc).

[hamishwillee]

Thank you very much all. It would be wonderful if someone who knows what they doing wrote a using the WebRTC stats guide that covered this kind of detail :-)

The purpose part is answered. I am still a bit unclear what this means:

If playout is slowed down by inserting samples, this will be the number of inserted samples.

The implication is that we don't always slow down by inserting samples.
If that is the case then how do we do it, and what then does the count mean?

[fippo]

It would be wonderful if someone who knows what they doing wrote a using the WebRTC stats guide that covered this kind of detail

We got you covered: https://bloggeek.me/getstats/