Encoding mp4 video to support seeking

Question

My Android app creates an MP4 using the MediaCodec and the MediaMuxer. I use the MediaPlayer to play back the video. While the video plays, it isn't possible to seek to any location in the video using Android's MediaPlayer. More specifically, the seekTo function will not work. Using other apps to play the video and seek is somewhat sketchy. Some apps seem to work while others do not.

I have swapped my mp4 with a video that I recorded on the camera as well as various videos I've found on the Internet and none of them have the problem seeking. The fact that the stock camera app can generate an MP4 and let you seek clearly indicates a problem in the way the codec is being setup. This leads me to believe that the problem is most likely in the format settings used to create the video. I have tried modifying a number of settings without any success including the profile (used both baseline and main), the profile level, the I-frame interval (GOP) as well as the bitrate and video size. I also made sure the presentation time for each frame matched the frame rate exactly. Here is the info I am getting for both the video that doesn't support seek and one that does (the camera video). Is there anything in these settings that could be causing the problem?

A short test file can be downloaded here. The seek works if you play this in QuickTime or VLC:

https://drive.google.com/file/d/15QiDPYdPd_tVQTkqXuP0v2L7eKoMbWQo/view?usp=sharing

Video that doesn't support seek:

**General**
Complete name : test.mp4
Format :    MPEG-4
Format profile :    Base Media
File size : 9.46 MiB
Duration :  19 s 488 ms
Overall bit rate :  4 071 kb/s
Encoded date :  UTC 2021-07-05 12:32:22
Tagged date :   UTC 2021-07-05 12:32:22
com_android_version :   11
    
**Video**
ID :    2
Format :    AVC
Format/Info :   Advanced Video Codec
Format profile :    Baseline
Format level :  5
Format settings, CABAC :    No
Format settings, Reference frames : 1 frame
Format settings, GOP :  M=1, N=30
Codec ID :  avc1
Duration :  17 s 900 ms
Source duration :   17 s 900 ms
Bit rate :  4 289 kb/s
Width : 1 440 pixels
Height :    2 614 pixels
Display aspect ratio :  0.551
Frame rate mode :   Constant
Frame rate :    30.000 FPS
Standard :  NTSC
Color space :   YUV
Chroma subsampling :    4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) :    0.038
Stream size :   9.15 MiB (97%)
Source stream size :    9.15 MiB (97%)
Title : VideoHandle
Language :  English
Encoded date :  UTC 2021-07-05 12:32:22
Tagged date :   UTC 2021-07-05 12:32:22
Color range :   Limited
Color primaries :   BT.601 PAL
Transfer characteristics :  BT.709
transfer_characteristics_Original : BT.601
Matrix coefficients :   BT.601
mdhd_Duration : 17900
Codec configuration box :   avcC
    
**Audio**
ID :    1
Format :    AAC LC
Format/Info :   Advanced Audio Codec
Codec ID :  mp4a-40-2
Duration :  19 s 488 ms
Bit rate mode : Constant
Bit rate :  128 kb/s
Channel(s) :    1 channel
Channel layout :    C
Sampling rate : 32.0 kHz
Frame rate :    31.250 FPS (1024 SPF)
Compression mode :  Lossy
Stream size :   305 KiB (3%)
Title : SoundHandle
Language :  English
Encoded date :  UTC 2021-07-05 12:32:22
Tagged date :   UTC 2021-07-05 12:32:22

Video that supports seek:

**General**
Complete name : camera-1.mp4
Format :    MPEG-4
Format profile :    Base Media
File size : 44.4 MiB
Duration :  21 s 823 ms
Overall bit rate :  17.1 Mb/s
Encoded date :  UTC 2021-07-05 06:19:27
Tagged date :   UTC 2021-07-05 06:19:27
com_android_version :   11

**Video**
ID :    2
Format :    AVC
Format/Info :   Advanced Video Codec
Format profile :    High
Format level :  4
Format settings, CABAC :    Yes
Format settings, Reference frames : 1 frame
Format settings, GOP :  M=1, N=30
Codec ID :  avc1
Duration :  21 s 823 ms
Bit rate :  17.0 Mb/s
Width : 1 920 pixels
Height :    1 080 pixels
Display aspect ratio :  16:9
Frame rate mode :   Variable
Frame rate :    30.000 FPS
Minimum frame rate :    29.890 FPS
Maximum frame rate :    30.120 FPS
Standard :  NTSC
Color space :   YUV
Chroma subsampling :    4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) :    0.273
Stream size :   44.2 MiB (99%)
Title : VideoHandle
Language :  English
Encoded date :  UTC 2021-07-05 06:19:27
Tagged date :   UTC 2021-07-05 06:19:27
Color range :   Full
Color primaries :   BT.601 PAL
colour_primaries_Original : BT.601 NTSC
Transfer characteristics :  BT.709
transfer_characteristics_Original : BT.601
Matrix coefficients :   BT.601
mdhd_Duration : 21823
Codec configuration box :   avcC

**Audio**
ID :    1
Format :    AAC LC
Format/Info :   Advanced Audio Codec
Codec ID :  mp4a-40-2
Duration :  21 s 819 ms
Source duration :   21 s 717 ms
Bit rate mode : Constant
Bit rate :  96.0 kb/s
Channel(s) :    1 channel
Channel layout :    C
Sampling rate : 48.0 kHz
Frame rate :    46.875 FPS (1024 SPF)
Compression mode :  Lossy
Stream size :   255 KiB (1%)
Source stream size :    255 KiB (1%)
Title : SoundHandle
Language :  English
Encoded date :  UTC 2021-07-05 06:19:27
Tagged date :   UTC 2021-07-05 06:19:27
mdhd_Duration : 21819

Answer 1

Markus has already provided root-cause of the issue.

Instead of modifying flag in BufferInfo, you can use below API for setting I-frame interval to your media-format.

KEY_I_FRAME_INTERVAL - encoder-only, time-interval between key frames.

More details: https://developer.android.com/reference/android/media/MediaFormat

Answer 2

Thanks to Markus to pointing me in the right direction. When using the MediaMuxer from Android, you need to set the BufferInfo.flags to MediaCodec.BUFFER_FLAG_KEY_FRAME on those frames that you want to have as sync frames. I was setting this but only on the first frame.

It wasn't clear to me how I-Frames in the MediaCodec were related to the sync frames in the muxer. They are in fact the same thing. When you encode a video using MediaCodec, you can indicate which frames are to act as the I-Frames. This however is not required. In fact, in my app, it isn't possible to set it because the encoder receives it's bitmap images from a hardware canvas and there is no possibility to set BufferInfo.flags in the MediaCodec for this. Even if you could set it, the MediaCodec doesn't do anything with it other than pass it along to the encoded output. It's up to you to pass it on further to the MediaMuxer when you combine the encoded video with encoded audio. So if you want your video to have seek support, make sure that those frames that you are feeding into the muxer have the BufferInfo.flags set to MediaCodec.BUFFER_FLAG_KEY_FRAME

If your generated mp4 lacks the atom object stss that Markus indicated, some video players, like the MediaPlayer will not be able provide any seek functionality. More advanced players though like ExoPlayer, QuickTime and VLC are capabable of seeking without the key frames but this will generally be slower.

Answer 3

A MP4 player needs to know the location of the sync samples (I-frames or IDR-frames). The sync sample location is usually signaled with then Sync Sample Box 'stss' located in moov->trak->mdia->minf->stbl->stss.

In your sample file the 'stss' box is missing.