ffmpeg zero latency webcam streaming

Question

I'm trying to use ffmpeg to stream a webcam with as close to zero latency as possible.

My test environment is streaming from localhost to localhost, on a macOS machine.

I've tried every parameter switch that I can understand - and many suggestions found both here and elsewhere. But, the latency is persistently just about one second. And to be honest, none of the options that I've tried appear to make any difference at all.

I've read that it should be possible to get the latency down to a couple of hundred milliseconds - or even as low as a single video frame (in these ideal test conditions). Can anyone help me with the magic ffmpeg settings?

Here's my base ffmpeg call. Note that I've deliberately not included here any of the exotic options that might help me. I want to start fresh with this question:

ffmpeg -f avfoundation -i "1:none" -vcodec libx264 -preset ultrafast -tune zerolatency -x264-params "" -f mpegts udp://127.0.0.1:9090

I monitor the stream on the same machine using ffplay, with this:

ffplay udp://127.0.0.1:9090

My belief is that ffmpeg (and X264) are somehow buffering the input stream from the webcam during the encoding process. I know why this can be necessary. But, for the purposes of this requirement, I'm prepared to sacrifice everything in the pursuit of low latency. Quality and stability are less important.

I acknowledge that there might be latency in my ffplay monitoring. I'm not so sure though - because I have tested with other monitors, with pretty much the same results. However, I'd be happy to receive advice regarding low latency monitoring.

I'm assuming I'm looking for options that will encode with all i-frames, no lookahead, and no buffering. Am I right? And if I am, how do I achieve that?

Thanks!

---

Output from ffmpeg request, as per Gyan:

ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[avfoundation @ 0x7ffbc2008a00] Selected pixel format (yuv420p) is not supported by the input device.
[avfoundation @ 0x7ffbc2008a00] Supported pixel formats:
[avfoundation @ 0x7ffbc2008a00]   uyvy422
[avfoundation @ 0x7ffbc2008a00]   yuyv422
[avfoundation @ 0x7ffbc2008a00]   nv12
[avfoundation @ 0x7ffbc2008a00]   0rgb
[avfoundation @ 0x7ffbc2008a00]   bgr0
[avfoundation @ 0x7ffbc2008a00] Overriding selected pixel format to use uyvy422 instead.
Input #0, avfoundation, from '1:none':
  Duration: N/A, start: 51270.672067, bitrate: N/A
    Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 1280x720, 17 tbr, 1000k tbn, 1000k tbc
At least one output file must be specified```


---------------------

Second output, as per second request from Gyan:

ffmpeg -f avfoundation -analyzeduration 200k -probesize 6M -pixel_format uyvy422 -i "1:none" -vcodec libx264 -preset ultrafast -tune zerolatency -x264-params "" -f mpegts udp://127.0.0.1:9090
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, avfoundation, from '1:none':
  Duration: N/A, start: 319.565533, bitrate: N/A
    Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 1280x720, 15 fps, 15 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x7feda5093200] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x7feda5093200] profile High 4:2:2, level 3.1, 4:2:2 8-bit
Output #0, mpegts, to 'udp://127.0.0.1:9090':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (libx264), yuv422p, 1280x720, q=-1--1, 15 fps, 90k tbn, 15 tbc
    Metadata:
      encoder         : Lavc58.54.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame= 1209 fps= 14 q=22.0 Lsize=   65771kB time=00:01:24.13 bitrate=6404.1kbits/s speed=   1x    
video:60873kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.047672%
[libx264 @ 0x7feda5093200] frame I:5     Avg QP:17.80  size:142658
[libx264 @ 0x7feda5093200] frame P:1204  Avg QP:19.99  size: 51180
[libx264 @ 0x7feda5093200] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x7feda5093200] mb P  I16..4:  5.9%  0.0%  0.0%  P16..4: 71.8%  0.0%  0.0%  0.0%  0.0%    skip:22.3%
[libx264 @ 0x7feda5093200] coded y,uvDC,uvAC intra: 17.4% 87.8% 49.8% inter: 29.4% 74.5% 38.3%
[libx264 @ 0x7feda5093200] i16 v,h,dc,p: 57% 20% 11% 12%
[libx264 @ 0x7feda5093200] i8c dc,h,v,p: 43% 18% 27% 12%
[libx264 @ 0x7feda5093200] kb/s:6186.95
Exiting normally, received signal 2.
slartibartfast:~ simon$ ffmpeg -f avfoundation -analyzeduration 200k -probesize 6M -pixel_format uyvy422 -i "1:none" 
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, avfoundation, from '1:none':
  Duration: N/A, start: 421.525767, bitrate: N/A
    Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 1280x720, 14.25 tbr, 1000k tbn, 1000k tbc
At least one output file must be specified

Answer 1

Have you tried -profile:v baseline?

This setting will output only I and P frames and get rid of B frames, which needs bidirectional reference to decode, also, I think ffplay is not the cause of latency, you may try connect ffmpeg and ffplay using pipe.

Answer 2

Not sure whether this concerns you as your stream seems to have lower resolution than the ones I am working on. However the latency being measured in seconds (very high) you could be the victim of the same issue I am having.

I am developing an application that shows the camera feed of the USB-connected 4K camera using FFmpeg-libraries. I have major latency issues on macOS when handling the video stream with 4K resolution and 30 fps. With 20fps the latency gets reduced noticeable.

So far my research on the issue points to AVfoundation instead of FFmpeg/x264. The frames returned by the low level FFmpeg-function (av_read_frame) are already decoded which implies the decoding of the raw data happens outside FFmpeg. When running the application I can see that CPU-consuming VTDecoderXPCService-process is spawned, so most likely the decoding (and latency inducing buffering) is here. When running the same code in Windows (using dshow) the frames returned by that function are still being encoded and the decoding happens inside FFmpeg-process.

So the key to the solution lies in the AVfoundation and possibly how FFmpeg communicates with it. Haven't yet found a satisfying solution for this but next going to investigate the IPC-mechanism between FFmpeg process and AVfoundation to see if the latency is caused by hitting some internal boundaries (macOS has notoriously low default values for System V settings).

Will keep you posted if I find something!