Muxing H264 packets into a MPEGTS container using libav*

Question

I'm writing a C++ program where I need to encode packets in h264 format and mux them to a MPEG TS container. For the encoding part, I based my code on the encode_video example (https://ffmpeg.org/doxygen/trunk/encode_video_8c-example.html#a9) provided in FFMPEG documentation, and it seems to work fine. In particular, I generate a std::vector of packets which I sequentially write to an output .ts file for debug. Such .ts file plays fine with SMPlayer, and a ffproba command gives

>> ffprobe  -print_format json -show_format -show_streams out.ts
Input #0, h264, from 'out.ts':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 640x480 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 1200k tbn, 50 tbc
    "streams": [
        {
            "index": 0,
            "codec_name": "h264",
            "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
            "profile": "Main",
            "codec_type": "video",
            "codec_time_base": "1/50",
            "codec_tag_string": "[0][0][0][0]",
            "codec_tag": "0x0000",
            "width": 640,
            "height": 480,
            "coded_width": 640,
            "coded_height": 480,
            "has_b_frames": 1,
            "sample_aspect_ratio": "1:1",
            "display_aspect_ratio": "4:3",
            "pix_fmt": "yuv420p",
            "level": 30,
            "chroma_location": "left",
            "field_order": "progressive",
            "refs": 1,
            "is_avc": "false",
            "nal_length_size": "0",
            "r_frame_rate": "25/1",
            "avg_frame_rate": "25/1",
            "time_base": "1/1200000",
            "bits_per_raw_sample": "8",
            "disposition": {
                "default": 0,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0
            }
        }
    ],
    "format": {
        "filename": "out.ts",
        "nb_streams": 1,
        "nb_programs": 0,
        "format_name": "h264",
        "format_long_name": "raw H.264 video",
        "size": "435443",
        "probe_score": 51
    }
}

The dts and pts timestamps are also set. However, if I try to mux them in MPEG TS format, using as a base the example mux.c (https://ffmpeg.org/doxygen/trunk/mux_8c-example.html), it doesn't work. A shortened version of my muxing code is as follows: (the variables ending with "_" are class fields)

int MyProcessing::Mux(const std::string outputFilename) {
    AVFormatContext *muxingContest;
    avformat_alloc_output_context2(&muxingContest, NULL, NULL, m_output.c_str());

    auto outFormat = muxingContest->oformat;
    outFormat->video_codec = AV_CODEC_ID_H264;

    AVStream *outStream;
    const AVCodec *codec;

    Mux_AddStream(&outStream, muxingContest, &codec, outFormat->video_codec);

    AVDictionary *opt = nullptr;
    Mux_OpenVideo(muxingContest, codec, outStream, opt);
 
    if (!(muxingContest->flags & AVFMT_NOFILE)) {
      avio_open(&muxingContest->pb, m_output.c_str(), AVIO_FLAG_WRITE);
    }
    avformat_write_header(muxingContest, &opt);

    auto muxOk = true;
    size_t countMuxedFrames = 0;
    while ((muxOk) && (countMuxedFrames < packets_.size())) {
        muxOk = !MuxPacket(muxingContest, outStream, packets_[countMuxedFrames], &opt);
        countMuxedFrames++;
    }

    av_write_trailer(muxingContest);
    if (!(muxCodecContextPtr_->flags & AVFMT_NOFILE)) avio_closep(&muxingContest->pb);
 
    return 0;
}


int MyProcessing::Mux_AddStream(AVStream **stream, AVFormatContext *format, const AVCodec **codec, enum AVCodecID codecId) {
    *codec = avcodec_find_encoder(codecId);
    muxPacketTmpPtr_ = av_packet_alloc();
    *stream = avformat_new_stream(format, *codec);
    (*stream)->time_base = (AVRational){ 1, STREAM_FRAME_RATE };
    (*stream)->id = format->nb_streams-1;
    (*stream)->index = 0;
    muxCodecContextPtr_ = avcodec_alloc_context3(*codec);
    Mux_FillCodecContext(*muxCodecContextPtr_, codecId, **stream);
    if (format->oformat->flags & AVFMT_GLOBALHEADER)
        muxCodecContextPtr_->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
    return 0;
}


void MyProcessing::Mux_FillCodecContext(AVCodecContext &cc, enum AVCodecID codecId, AVStream &stream) {
    cc.codec_id = codecId;
    cc.bit_rate = 400000;
    cc.width    = outputWidth_;
    cc.height   = outputHeight_;
    cc.time_base  = stream.time_base;
    cc.gop_size = 10;
    cc.max_b_frames = 1;
    cc.gop_size      = 12;
    cc.pix_fmt       = AV_PIX_FMT_YUV420P;
    if (cc.codec_id == AV_CODEC_ID_MPEG2VIDEO) cc.max_b_frames = 2;
    if (cc.codec_id == AV_CODEC_ID_MPEG1VIDEO)  cc.mb_decision = 2;
    av_opt_set(&cc, "preset", "slow", 0);
    av_opt_set(&cc, "tune", "zerolatency", 0);
}


int MyProcessing::Mux_OpenVideo(AVFormatContext *format, const AVCodec *codec, AVStream *stream, AVDictionary *opt_arg) {
    AVDictionary *opt = nullptr;
    av_dict_copy(&opt, opt_arg, 0);
    muxCodecContextPtr_->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
    avcodec_open2(muxCodecContextPtr_, codec, &opt);
    av_dict_free(&opt);
    avcodec_parameters_from_context(stream->codecpar, muxCodecContextPtr_);
    return 0;
}

int MyProcessing::MuxPacket(AVFormatContext *format, AVStream *stream, AVPacket &pkt, AVDictionary **opt) {
    AVBitStreamFilterContext *bsf = av_bitstream_filter_init("h264_mp4toannexb");
    AVPacket filteredPkt = pkt;
    auto filterResult = av_bitstream_filter_filter(bsf, format->streams[stream->index]->codec, NULL,
                                         &filteredPkt.data, &filteredPkt.size,
                                         pkt.data, pkt.size,
                                         pkt.flags & AV_PKT_FLAG_KEY);

    if (filterResult < 0) return filterResult;
    else {
        filteredPkt.buf = av_buffer_create(filteredPkt.data, filteredPkt.size,
                                       av_buffer_default_free, NULL, 0);
    }
    av_bitstream_filter_close(bsf);
    filteredPkt.stream_index = stream->index;
    filteredPkt.dts = filteredPkt.pts;
    filteredPkt.duration = ((double)stream->time_base.num / (double)stream->time_base.den) / STREAM_FRAME_RATE;
    av_packet_rescale_ts(&filteredPkt, muxCodecContextPtr_->time_base, stream->time_base); // rescale output packet timestamp values from codec to stream timebase
    auto writePktResult = av_write_frame(format, &filteredPkt);
   // auto writePktResult = av_interleaved_write_frame(format, &filteredPkt);
    return 0;
}

The console error is

[mpegts @ 0x55555736edc0] H.264 bitstream malformed, no startcode found, use the video bitstream filter 'h264_mp4toannexb' to fix it ('-bsf:v h264_mp4toannexb' option with ffmpeg)

It Is telling me to apply the h264_mp4toannexb filter. As you see from the code, I've put the filtering accordingly, but the error message persists (unless I'm applying the filter in a wrong way).

In the last lines of method MuxPacket(), if I uncomment the line with av_interleaved_write_frame() and comment the previous one, I get the same error, as well as a seg fault. Inspecting with GDB, the call stack for the seg fault is as follows:

#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
#1  0x00007ffff67c7cb6 in av_packet_copy_props () at /lib/x86_64-linux-gnu/libavcodec.so.58
#2  0x00007ffff67c8447 in av_packet_ref () at /lib/x86_64-linux-gnu/libavcodec.so.58
#3  0x00007ffff7e2fa13 in  () at /lib/x86_64-linux-gnu/libavformat.so.58
#4  0x00007ffff7e2fb11 in  () at /lib/x86_64-linux-gnu/libavformat.so.58
#5  0x00007ffff7e30575 in av_interleaved_write_frame () at /lib/x86_64-linux-gnu/libavformat.so.58

I tried to look at solutions online, but they are either old or they don't work. Some of the things I tried and didn't work:

1. Putting the line

muxCodecContextPtr_->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;

in Mux() after the call to avformat_alloc_output_context2.

2. Setting

packet.flags |= AV_PKT_FLAG_KEY;

before the call to av_write_frame / av_interleaved_write_frame.

3. Trying to write by hand to the file the starting code as described here Need to convert h264 stream from annex-b format to AVCC format.

4. Playing with parameters in Mux_FillCodecContext().