Explicit frame drop signal from VideoEncoder

[Djuffin]

Currently the spec doesn't specify any way for VideoEncoder to signal that a frame has been dropped due to realtime mode constraints.
I think it would be useful to have such a signal.
Possible options:

1. the output callback should be called with an empty encoded chunk
2. the output callback should be called with a new flag set in EncodedVideoChunkMetadata

This came up after WPT test change that accounts for frame drops.

[Djuffin]

On the other hand, 2 existing implementations drop frames silently, so using the regular output callback as is will be a breaking change. If we ever do this we'll need a VideoEncoderConfig option to turn it on.
Let's wait and see if it actually needed by people building RTC apps.

[Djuffin]

At least Meet is interested in drop frame notifications.

[Djuffin]

Currently when a real time encoder drops a frame clients don’t get any signal from VideoEncoder. If they really want to know that a frame was dropped they need to look at the timestamps of the encoded video chunks.

This issue can be addressed by a new flag in the encoder config that would cause the encoder to call EncodedVideoChunkOutputCallback for dropped frames with an empty data buffer in the encoded chunk.

partial dictionary VideoEncoderConfig {
	boolean notifyAboutFrameDrops = false;
}

[aboba]

@Djuffin When encoding with H.264, am seeing the following messages refer to bEnableFrameSkip. Is there any way to set this to a value other than 0?

[OpenH264] this = 0x0x13c05ca6e70, Warning:bEnableFrameSkip = 0,bitrate can't be controlled for RC_QUALITY_MODE,RC_BITRATE_MODE and RC_TIMESTAMP_MODE without enabling skip frame.
[OpenH264] this = 0x0x13c05ca11a0, Warning:bEnableFrameSkip = 0,bitrate can't be controlled for RC_QUALITY_MODE,RC_BITRATE_MODE and RC_TIMESTAMP_MODE without enabling skip frame.
[OpenH264] this = 0x0x13c05ca11a0, Warning:Actual input framerate 0.008349 is different from framerate in setting 30.000000, suggest to use other rate control modes
[OpenH264] this = 0x0x13c05ca11a0, Warning:Actual input framerate 23.112173 is different from framerate in setting 30.000000, suggest to use other rate control modes

[Djuffin]

@aboba We have a task for it, but it never got any attention https://issues.chromium.org/issues/40811827
IMO this OpenH264 log message is exaggerating the problem.

[taste1981]

hi @Djuffin any progress on this? Now Chromium turns on frame dropping in some of its VEAs. With this application may not receive output and wait there forever, before proceed to next frame?

[Djuffin]

no progress yet

[taste1981]

Currently when a real time encoder drops a frame clients don’t get any signal from VideoEncoder. If they really want to know that a frame was dropped they need to look at the timestamps of the encoded video chunks.

This issue can be addressed by a new flag in the encoder config that would cause the encoder to call EncodedVideoChunkOutputCallback for dropped frames with an empty data buffer in the encoded chunk.

partial dictionary VideoEncoderConfig {
boolean notifyAboutFrameDrops = false;
}

@dalecurtis Are we going to standarize this?