CODEWITHSUNDEEP
videoffmpeggstreamer
Florian Echtler
Florian Echtler

Reputation: 2513

Writing multi-stream MP4 file with Gstreamer

My Gstreamer 1.20 application is writing two H.264 video streams and one Opus audio stream to a MP4 file. The relevant parts of the pipeline looks as follows:

Video encoder (this part is present twice):

queue max-size-time=100000000 leaky=downstream ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux.

Audio encoder:

queue max-size-time=100000000 leaky=downstream ! opusenc bitrate-type=vbr inband-fec=true ! queue ! opusparse ! mux.

Muxer and filesink:

mp4mux name=mux fragment-duration=1000 latency=100000000 ! filesink sync=true location=out.mp4

The result is an MP4 file which I can play just fine in Totem on Ubuntu, and even switch between video streams using "Switch angles".

However, none of the following work:

In all cases, I get some messages about "Invalid NAL unit size" and/or "non-monotonous DTS". So what would I need to do to my output pipeline to get a "compliant" MP4 file?

Note: I can "fix" the resulting file by using GStreamer with qtdemux to split it into two raw H.264 files and a WAV file, and then use ffmpeg to re-encode it into MP4 with an MP3 audio track. But I'd rather have a workable MP4 file right away.

Upvotes: 1

Views: 1256

Answers (1)

SeB
SeB

Reputation: 1626

Your pipeline may lack video information from caps. You may check with:

gst-launch-1.0 \
 videotestsrc ! video/x-raw,format=NV12,width=640,height=480,framerate=30/1 ! queue ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 insert-vui=1 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux. \
 videotestsrc pattern=ball ! video/x-raw,format=NV12,width=640,height=480,framerate=30/1 ! queue ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 insert-vui=1 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux. \
 audiotestsrc ! audioresample ! voaacenc ! queue ! mux. \
 qtmux name=mux fragment-duration=1000 ! queue ! filesink sync=true location=out.mp4

# Or try mkv for opus:
gst-launch-1.0 \
 videotestsrc ! queue ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux. \
 videotestsrc pattern=ball ! queue ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux. \
 audiotestsrc ! audioconvert ! opusenc ! queue ! mux.  \
 matroskamux name=mux ! queue ! filesink sync=true location=out.mkv

Once recorded, you should be able to see the streams:

# stream 0
ffplay -autoexit out.mp4

# stream 1
ffplay -autoexit out.mp4 -vst v:1

Upvotes: 1

Related Questions