CODEWITHSUNDEEP
c#androidxamarinandroid-mediacodecmediaextractor
programmer
programmer

Reputation: 1

Android MediaExtractor/MediaCodec results in mostly Black frames, Outputs only some frames, and they're out of order

Having a few issues extracting/decoding YUV frames from a video file recorded using MediaRecorder MP4/H264 1280x720 @ 240fps. The video plays fine, so it's not the video. The resultant frames are mostly black with some squares at the start (see sample image: i420 Frame). I have confirmed that Formatyuv420planar is supported; but, as noted in code, the default has the same issue. The saved .yuv frame files are the expected size of 1280 * 720 * 1.5 = 1382400 bytes.

Another issue I'm seeing is that the Input Buffers are in order, but the Output Buffers are out of order. And the third issue is that only some of the Input frames result in Outputs. Many are skipped. I understand that there are I-Frames and partial frames, but it's outputting more than just I-Frames. See the Output section below which shows a small excerpt from the start of the video extraction. The Sync frames (I-Frames) appear about every second... but there's only one shown in this short excerpt.

At this point I have only tested in Debug mode from Visual Studio USB connected to a Galaxy S9, I'm not sure if Debug is causing issues or not.

The code is below (Xamarin / C#). Note this is just figuring it out code, I'll likely make this async after I figure it out.

MediaExtractor me; MediaFormat mf; MediaCodec mc; Java.Nio.ByteBuffer bb; MediaCodec.BufferInfo bi;

me = new MediaExtractor();
me.SetDataSource(fileName);
me.SelectTrack(0); // There's only a video track
mf = me.GetTrackFormat(0); // There's only a video track
sFormat = mf.GetString(MediaFormat.KeyMime); // video/avc
mc = MediaCodec.CreateDecoderByType(sFormat);
mf = MediaFormat.CreateVideoFormat(sFormat, 1280, 720);

mf.SetInteger(MediaFormat.KeyColorFormat, (int)MediaCodecCapabilities.Formatyuv420planar); // I420  
// Commenting the previous line results in Formatyuv420semiplanar (NV12) format
// But it outputs the same images minus most of the color
mc.Configure(mf, null, null, MediaCodecConfigFlags.None);
mc.Start();

bEOS = false; // End of Stream
while (true) {
    if (!bEOS) {
        Index = mc.DequeueInputBuffer(-1); if (Index < 0) break;
        bb = mc.GetInputBuffer(Index);
        nData = me.ReadSampleData(bb, 0);
        if (nData > 0) {
            time = me.SampleTime;
Console.WriteLine(String.Format(" IN:   Time:{0}   Type:{1}", time, me.SampleFlags.ToString()));
            mc.QueueInputBuffer(Index, 0, nData, time, MediaCodecBufferFlags.None);
            me.Advance();
        } else {
            // End of Stream
            mc.QueueInputBuffer(Index, 0, 0, 0L, MediaCodecBufferFlags.EndOfStream);
            bEOS = true;
        }
    }

    Index = mc.DequeueOutputBuffer(bi, 0);
    if (Index >= 0) {
        if ((bi.Flags & MediaCodecBufferFlags.EndOfStream) != 0) break; // End of Stream
Console.WriteLine(String.Format("OUT:   Time:{0}", bi.PresentationTimeUs));
        bb = mc.GetOutputBuffer(Index);

        // Save yuv frame to file
        Java.Nio.Channels.FileChannel fout = new Java.IO.FileOutputStream(fileName + "_###.yuv").Channel;
        fout.Write(bb); fout.Close();

        mc.ReleaseOutputBuffer(Index, false);
    }
}

mc.Stop();
mc.Release();

Output...

In:   Time:0   Type:Sync
In:   Time:4166   Type:None
In:   Time:8333   Type:None
In:   Time:12500   Type:None
In:   Time:17000   Type:None
In:   Time:21166   Type:None
In:   Time:25333   Type:None
In:   Time:29500   Type:None
In:   Time:33988   Type:None
In:   Time:38155   Type:None
Out:   Time:0   Type:None
In:   Time:42322   Type:None
In:   Time:46488   Type:None
In:   Time:50988   Type:None
In:   Time:55155   Type:None
In:   Time:59322   Type:None
In:   Time:63488   Type:None
In:   Time:67988   Type:None
In:   Time:72155   Type:None
In:   Time:76322   Type:None
In:   Time:80488   Type:None
In:   Time:84977   Type:None
In:   Time:89144   Type:None
In:   Time:93311   Type:None
In:   Time:97477   Type:None
In:   Time:101977   Type:None
In:   Time:106144   Type:None
In:   Time:110311   Type:None
In:   Time:114477   Type:None
In:   Time:118966   Type:None
In:   Time:123133   Type:None
In:   Time:127300   Type:None
In:   Time:131466   Type:None
In:   Time:135966   Type:None
In:   Time:140133   Type:None
In:   Time:144300   Type:None
In:   Time:148466   Type:None
In:   Time:152966   Type:None
In:   Time:157133   Type:None
In:   Time:161300   Type:None
In:   Time:165466   Type:None
In:   Time:169955   Type:None
In:   Time:174122   Type:None
In:   Time:178288   Type:None
In:   Time:182455   Type:None
In:   Time:186955   Type:None
In:   Time:191122   Type:None
In:   Time:195288   Type:None
In:   Time:199455   Type:None
In:   Time:203955   Type:None
In:   Time:208122   Type:None
In:   Time:212288   Type:None
In:   Time:216455   Type:None
In:   Time:220944   Type:None
In:   Time:225111   Type:None
In:   Time:229277   Type:None
In:   Time:233444   Type:None
In:   Time:237944   Type:None
In:   Time:242111   Type:None
In:   Time:246277   Type:None
Out:   Time:157133   Type:None
In:   Time:250444   Type:None
In:   Time:254944   Type:None
In:   Time:259111   Type:None
Out:   Time:89144   Type:None
In:   Time:263266   Type:None
In:   Time:267444   Type:None
In:   Time:271933   Type:None
In:   Time:276100   Type:None
In:   Time:280266   Type:None
In:   Time:284433   Type:None
In:   Time:288933   Type:None
In:   Time:293100   Type:None
In:   Time:297266   Type:None
Out:   Time:80488   Type:None
In:   Time:301433   Type:None
Out:   Time:178288   Type:None
In:   Time:305922   Type:None
Out:   Time:284433   Type:None
In:   Time:310088   Type:None
Out:   Time:280266   Type:None
In:   Time:314255   Type:None
In:   Time:318422   Type:None
In:   Time:322922   Type:None
In:   Time:327088   Type:None
In:   Time:331255   Type:None
In:   Time:335422   Type:None
Out:   Time:288933   Type:None
In:   Time:339922   Type:None
Out:   Time:144300   Type:None
In:   Time:344088   Type:None
Out:   Time:314255   Type:None
...

Upvotes: 0

Views: 526

Answers (1)

programmer
programmer

Reputation: 1

In case anyone stumbles upon this. This was a bit confusing because of oodles of examples out there that ignore this, didn't explain it very well, suggested it was not needed, or were ancient.

Because this was video/avc format, I needed to include SPS/SPP config data, which I pulled from the MediaFormat provided by MediaExtractor. Here's the two lines of code needed to be added (Insert indented lines in code above where shown).

mf = me.GetTrackFormat(0); // There's only a video track

  csd0 = mf.GetByteBuffer("csd-0"); csd1 = mf.GetByteBuffer("csd-1");

mf = MediaFormat.CreateVideoFormat(sFormat, RecordSize.Width, RecordSize.Height);

  mf.SetByteBuffer("csd-0", csd0); mf.SetByteBuffer("csd-1", csd1);

Upvotes: 0

Related Questions