MediaMuxer writing H264 stream to mpg file

Question

MediaMuxer has been driving me mad for two days & nights now :-(

The situation: I receive a H264 encoded 1280x720 video stream via UDP. The h264 stream contains NALU 1 - slice and NALU 5 - keyframe (5 is always preceded by NALU 7 - SPS and NALU 8 - PPS). This stream appears to be stable 30fps with at least one NALU 5 keyframe per second. Bitrate is variable but less than 4Mbps. MediaCodec sucessfully decodes the stream and renders it in a surface view so that part works well.

But now I need to save the H.264 into a local mpg file. I set up a MediaMuxer with all MediaFormat information that I have, and feed it with the sample data from the stream. Each sample contains one frame (NALU 1 or 5), and the first data sent to MediaMuxer is a keyframe (NALU 5). The presentation time is calculated based on framenumber and framerate. All involved methods are called from the same thread.

But the mpg file is never created. As you can see in the output below the data in the ByteBuffer does start with NALU headers followed by varying size of data. MediaMuxer seems to "see" frames in the data as it counts the frames. So what is wrong here?

Minimum API is 21, and I have tested with a Samsung Galaxy S4 running stock Android 5 and a couple of devices running Lineageos Oreo and Nougat.

Here is the code to setup the MediaMuxer:

void setupMuxer(File f) throws IOException {
    if (DEBUG) Log.d(TAG, "Setup Muxer: " + f.getAbsolutePath() +" can write: " + f.canWrite());
    MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, decoderWidth, decoderHeight);
    format.setInteger(MediaFormat.KEY_BIT_RATE, 4000000);
    format.setInteger(MediaFormat.KEY_FRAME_RATE, 29);
    format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
    format.setByteBuffer("csd-0", ByteBuffer.wrap(sps)); // sps and pps have been retrieved from the stream's NAL 7/8
    format.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
    format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);

    muxer = new MediaMuxer(f.getPath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
    videoTrack = muxer.addTrack(format);
    muxer.start();
}

This method is called for each (complete) NALU 1 and NALU 5:

 void muxFrame(ByteBuffer buf, int frame) {
    MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
    bufferInfo.offset = buf.arrayOffset();
    bufferInfo.size = buf.position() - bufferInfo.offset;
    bufferInfo.flags = (buf.get(4) & 0x1f) == 5 ? MediaCodec.BUFFER_FLAG_KEY_FRAME : 0;
    bufferInfo.presentationTimeUs = computePresentationTime(frame);


    if (DEBUG)
        Log.d(TAG, "muxFrame frame: " + frame + " size: " + bufferInfo.size + " NAL: " + (buf.get(4) & 0x1f) + " Flags: " + bufferInfo.flags + " PTS: " + bufferInfo.presentationTimeUs + " content: " + BitByteUtil.toByteString(buf.array(), buf.arrayOffset(), 8));

    try {
        muxer.writeSampleData(videoTrack, buf, bufferInfo);
    } catch (Exception e) {
        Log.w(TAG, "muxer failed", e);
    } finally {
    }
}

private static long computePresentationTime(int frameIndex) {
    return 42 + frameIndex * 1000000 / FRAME_RATE;
}

Here is my output if MediaMuxer is stopped after it has consumed 100 NALUs.

05.651 D/VideoDecoderView: Setup Muxer: /storage/emulated/0/Pictures/test.mpg can write: true
05.656 I/MPEG4Writer: limits: 4294967295/0 bytes/us, bit rate: -1 bps and the estimated moov size 3317 bytes
06.263 D/VideoDecoderView: muxFrame frame: 2 size: 7257 NAL: 5 Flags: 1 PTS: 66708 content: 0:000 1:000 2:000 3:001 4:101 5:184 6:000 7:015 
06.264 I/MPEG4Writer: setStartTimestampUs: 66708
06.264 I/MPEG4Writer: Earliest track starting time: 66708
06.308 D/VideoDecoderView: muxFrame frame: 3 size: 8998 NAL: 1 Flags: 0 PTS: 100042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:034 7:020 
06.342 D/VideoDecoderView: muxFrame frame: 4 size: 13664 NAL: 1 Flags: 0 PTS: 133375 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:066 7:020 
06.375 D/VideoDecoderView: muxFrame frame: 5 size: 13674 NAL: 1 Flags: 0 PTS: 166708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:098 7:020 
06.409 D/VideoDecoderView: muxFrame frame: 6 size: 13772 NAL: 1 Flags: 0 PTS: 200042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:130 7:020 
06.483 D/VideoDecoderView: muxFrame frame: 7 size: 13707 NAL: 1 Flags: 0 PTS: 233375 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:162 7:020 
06.520 D/VideoDecoderView: muxFrame frame: 8 size: 13778 NAL: 1 Flags: 0 PTS: 266708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:194 7:020 
06.555 D/VideoDecoderView: muxFrame frame: 9 size: 13743 NAL: 1 Flags: 0 PTS: 300042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:226 7:020 
06.575 D/VideoDecoderView: muxFrame frame: 10 size: 7338 NAL: 5 Flags: 1 PTS: 333375 content: 0:000 1:000 2:000 3:001 4:101 5:184 6:000 7:015 
06.593 D/VideoDecoderView: muxFrame frame: 11 size: 9059 NAL: 1 Flags: 0 PTS: 366708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:034 7:020 
06.618 D/VideoDecoderView: muxFrame frame: 12 size: 13587 NAL: 1 Flags: 0 PTS: 400042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:066 7:020 
06.644 D/VideoDecoderView: muxFrame frame: 13 size: 13650 NAL: 1 Flags: 0 PTS: 433375 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:098 7:020 
06.671 D/VideoDecoderView: muxFrame frame: 14 size: 13797 NAL: 1 Flags: 0 PTS: 466708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:130 7:020 
.... [snip]
09.620 D/VideoDecoderView: muxFrame frame: 97 size: 7212 NAL: 5 Flags: 1 PTS: 3233375 content: 0:000 1:000 2:000 3:001 4:101 5:184 6:000 7:015 
09.661 D/VideoDecoderView: muxFrame frame: 98 size: 8814 NAL: 1 Flags: 0 PTS: 3266708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:034 7:020 
09.692 D/VideoDecoderView: muxFrame frame: 99 size: 13566 NAL: 1 Flags: 0 PTS: 3300042 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:066 7:020 
09.737 D/VideoDecoderView: muxFrame frame: 100 size: 13733 NAL: 1 Flags: 0 PTS: 3333375 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:098 7:020 
09.771 D/VideoDecoderView: muxFrame frame: 101 size: 13771 NAL: 1 Flags: 0 PTS: 3366708 content: 0:000 1:000 2:000 3:001 4:065 5:224 6:130 7:020 
09.775 D/MPEG4Writer: Video track stopping. Stop source
09.775 I/MPEG4Writer: Received total/0-length     (100/1) buffers and encoded 100 frames. - Video
09.775 D/MPEG4Writer: Video track source stopping
09.775 D/MPEG4Writer: Video track source stopped
09.775 D/MPEG4Writer: Video track stopped. Stop source
09.775 D/MPEG4Writer: Stopping writer thread
09.776 D/MPEG4Writer: 0 chunks are written in the last batch
09.779 D/MPEG4Writer: Writer thread stopped
09.780 I/MPEG4Writer: Ajust the moov start time from 66708 us -> 66708 us
09.780 D/MPEG4Writer: Video track stopping. Stop source

Answer 1

@greeble31: You are right. The first log entry clearly states "Pictures" and not "Videos". I spent hours looking at this problem without noticing a simple cut&paste mistake in my preferences keys. How stupid is that!!?!

Note to myself: Coding two days & nights in a row is not heroic but just plain stupid.