CODEWITHSUNDEEP
ffmpegwebrtcvideo-streamingh.264
Lucas
Lucas

Reputation: 313

h264 via WebRTC latency issue

I am trying to send a video stream encoded with h264 (hardware accelerated with nvidia encoder) via WebRTC for low latency display on a browser.

More precisely, I have a thread that encodes an opengl framebuffer at a fixed frame rate, the resulting AVPacket's data (I encode using ffmpeg's C api) is then forwarded via WebRTC to the client (using aiortc)

The problem is that I observe significant delays, that seem to depend on the frame rate I use. For example, running it locally, I get around 160ms delay when running at 30fps, and around 30ms when encoding at 90fps.

The delay here is the measured time to encode + transmit + decode, and I have the strong impression that the issue happen when presenting the video frame, like the browser is not immediately presenting the frame... (encoding is fast, I would expect the transmission to be also rather fast on a local setup, and decoding seems to be fine as well, as reported by the RTP stats in the browser).

I tried to play with RTP timestamps, but that did not change anything, the only variable that seems to impact the latency is the encoding thread 'frequency'.

Any idea on what could be creating this latency? Am I missing a parameter?

Also, here are the codec options I use: (they do not influence the latency that much from what I experimented)

profile = high
preset = llhq  # low latency, high quality
tune = zerolatency
zerolatency = 1
g = 2 * FRAME_PER_SECOND  # key frame every 2s
strict-gop = 1

UPDATE

I have the impression that the jitter buffer on Chrome's side is kind of preventing the rtp packets to be decoded immediately, is that possible?

UPDATE 2

UPDATE 3

After further investigations, I came to the conclusion that it was not possible to get a lower latency by going through the standard webrtc for video streams, as there is little to no control on the video buffering, which I believe to be responsible of the observed latency.

On a side note, I tried to check how google stadia is doing it, as they seem to use WebRTC as well, but they use some in-house frameworks... (plus Chrome is the only supported browser)

Upvotes: 4

Views: 2403

Answers (2)

Artem
Artem

Reputation: 103

BUsing RTP playout-delay header extension slightly reduced the latency.

Setting playoudDelayHint in browser also seemed to help a bitlockquote

I have the same problem (but for some reason, my similar question was closed by moderators), please explain more on these points. I tried adding these lines to the SDP on the server side, but it didn’t give any result:

a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=playout-delay:0 1

I also tried setting playoudDelayHint on the client side (JS), but there was no difference either. Perhaps I am doing something wrong. In JS, I tried this:

pc.ontrack = function(event) {
  var el = document.createElement(event.track.kind)
  el.srcObject = event.streams[0]
  el.autoplay = true
  el.controls = true
  el.preload = 'none'
  el.buffered = false
  if(event.track.kind === "video") {
      event.receiver.playoutDelayHint = 0;
   }

  document.getElementById('remoteVideos').appendChild(el)
}

Upvotes: 0

ploover
ploover

Reputation: 11

If you use NvFBC with NvENC you shouldn't use ffmpeg to encode the frames, store your framebuffer as an opengl texture and feed that to your encoder, you're currently copying your framebuffer data from your VRAM to system memory, back to VRAM and then back again. There is an example in the Nvidia Capture zip file that's on their website.

I'm currently dealing with a very similar issue, but the reason the jitter buffer piles up is possibly because of timestamping issues, not the protocol itself.

Upvotes: 1

Related Questions