CODEWITHSUNDEEP
c++audiovideoffmpegmuxer
PookyFan
PookyFan

Reputation: 840

Glitchy audio or broken video in fragmented MP4

I'm working on small C++ library for muxing audio and video. This is basically a facade for FFMPEG functions and structures. The code is here with minimal reproduction testing code here and as of now it seems like it's working fine... almost.

For the record - my MP4 file is so-called "fragmented MP4", with headers moved at the beginning of the file in a way that would allow to stream that file (ie. play it in a browser while it's being buffered). That's what these movflags I'm setting in Mp4Muxer::writeHeader() are for.

While testing this library with raw H264 video stream and MP3 file (video is <1 min long, MP3 - a few minutes long), I observed that:

[h264 @ 0x7f90a40ae2c0] Invalid NAL unit size (2162119 > 76779).0
[h264 @ 0x7f90a40ae2c0] Error splitting the input into NAL units.
[mp3float @ 0x7f90a4009540] Header missing  515KB sq=    0B f=0/0
[h264 @ 0x7f90a40cb0c0] Invalid NAL unit size (-860010620 > 17931).
[h264 @ 0x7f90a40cb0c0] Error splitting the input into NAL units.
[h264 @ 0x7f90a42bf440] Invalid NAL unit size (-168012642 > 8000).
[h264 @ 0x7f90a42bf440] Error splitting the input into NAL units.
[h264 @ 0x7f90a42fa780] Invalid NAL unit size (-1843711407 > 5683).
[ and it repeats...]

[mp3float @ 0x7f744c01b640] overread, skip -6 enddists: -1 -1=0/0

Not limitting muxed audio (at all or enough) relative to muxed video also results in following messages in my muxing application:

[mp4 @ 0x55d0c6c21940] Delay between the first packet and last packet in the muxing queue is 10004898 > 10000000: forcing output

For now, the fix is quite ugly and I don't even understand why it works, but before writting MP4 header I manually set a limit for frames buffered by muxer, like so:

formatCtxt->max_interleave_delta = 10000000LL * 10LL;

This way the muxer can store more packets of one stream that's way "ahead" of the other (maximum difference between DTS of the packets at the beginning and at the end of queue is set to 10x larger than default; it also gets rid of information log mentioned above). Obviously, I'd like to resolve it more properly, without hacking things like that.

I was trying various things, including manual skipping of ID3 tags in MP3 file (but seems like FFMPEG handles them just fine and it didn't change anything). I was also experimenting with FLAC in MP4 instead of MP3. and while I know it's rather experimental thing, I encountered very similar problems with glitching audio (no problem with video being frozen when lots of audio data gets muxed, though). It also seems that problem with glitching audio or frozen video varies in scale depending on how large are input data chunks that I feed muxer with. For now, honestly, I'm out of ideas.

Upvotes: 0

Views: 403

Answers (1)

PookyFan
PookyFan

Reputation: 840

I'm answering my own question, maybe someone will learn from my mistakes. Btw the code on the main branch changed significantly, but the faulty part can still be highlighted.

It looks like the problematic code was this fragment in Mp4Muxer::muxMediaData() method:

for(auto packet = mediaCtxt.getNextFrame();
    packet.size > 0 && checkLimit(audioAheadOfVideoInCommonTimebase, timeAheadInCommonTimebaseLimit);
    packet = mediaCtxt.getNextFrame(), ++packetsMuxedCnt)

In this loop, I first fetch media packet, and then check if stream is limited due to being too much ahead of the other stream. If it is, the media packet is lost. And since audio was almost always way ahead of the video, it was glitching out because of dropped samples.

Upvotes: 0

Related Questions