Mediacodec Decoding h264 OutputBuffer Index Returns -1

Question

I am decoding an h264 video stream from a drone on Android using Mediacodec. The drone sends a byte array each time that contains NAL units. The drone document indicates that the IDR-Frame is not included in the returned byte array. So I managed to download the I-Frame file from their website and fed it to the decoder before any other access unit. However, the index returned from mCodec.dequeueOutputBuffer is always -1.

UPDATE 04/01/2016: Now I correctly feed the decoder with SPS/PPS data and IDR frame before any other NAL units. I observed that the dequeueOutputBuffer returned -3 and then a few positive numbers. After that it returned -1 all the time. So my guess is that there is something wrong with how I handle the access units in the video stream?

inputBuffer.put(SPS_PPS,0,SPS_PPS.length);
mCodec.queueInputBuffer(inIndex, 0, SPS_PPS.length, presentationTime, BUFFER_FLAG_CODEC_CONFIG);

Some assumptions I've made:

1. I feed each input buffer of Mediacodec with a complete access unit starting with 00 00 00 01 09

2. I decode to a surface from a predefined GLSurface in xml layout, using GLSurface.getHolder().getSurface(). I'm not sure if this is the right way to do.

My Problems:

1. outputbuffer index returned is always -1, and no video on the screen

2. The byte array returned from the drone always has its latter half to be all 0x00. I don't know if these 0s should all be included in the NAL units as well

3. The byte array contains NAL units and I have to split them by myself. But these nal units have only 0x25, 0x27, 0x28, 0x06 and 0x09 types. I only know that 0x09 indicates a AUD

I really hope anyone could provide even the slightest piece of advice since this thing is driving me crazy these days.

My code snippets:

First of all my main activity class implements the surfaceHolder callback.

Setting up the surface holder:

private DjiGLSurfaceView mDjiGLSurfaceView;
mDjiGLSurfaceView = (DjiGLSurfaceView)findViewById(R.id.DjiSurfaceView_);
mDjiGLSurfaceView.getHolder().addCallback(this);

Read the I-frame from a .264 file to store in byte array -> iframe:

BufferedInputStream buf = new BufferedInputStream(is);
buf.read(iframe, 0, iframe.length);

My asynchronous processing part:

The DJIReceivedVideoDataCallBack is called when the byte array containing video data is received.

public void surfaceCreated(SurfaceHolder holder){
    Log.e(TAG, "Surface Created!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
    format.setString("KEY_MIME", videoFormat);
    try
    {
        mCodec = MediaCodec.createDecoderByType(videoFormat);
        mCodec.configure(format, mDjiGLSurfaceView.getHolder().getSurface(), null, 0 );
        mCodec.start();

        mReceivedVideoDataCallBack = new DJIReceivedVideoDataCallBack(){
            private int packetLength = 0;
            private ByteBuffer accessUnitBuffer = ByteBuffer.allocate(50000);
            int inIndex;
            long presentationTime = 0;

            @Override
            public void onResult(byte[] videoBuffer, int size){
                ArrayList<byte []> NAL_Units = splitNALunits(videoBuffer,size);
                //Send the I-Frame first
                if (!seq_start && iframe_ready){
                    inIndex = mCodec.dequeueInputBuffer(0);
                    if (inIndex >= 0) {
                        ByteBuffer inputBuffer = mCodec.getInputBuffer(inIndex);
                        inputBuffer.put(iframe,0,iframe.length);
                        mCodec.queueInputBuffer(inIndex, 0, iframe.length, presentationTime, 0);
                        presentationTime += 100;
                        seq_start = true;
                        handler.sendMessage(handler.obtainMessage(SHOWTOAST, "I-Frame queued!!!"));
                    }
                }
                for( int i=0; i< NAL_Units.size(); i++ ) {
                    if( NAL_Units.get(i)[4] == 0x09 ) {
                        // Send off the current buffer of data (Access Unit)
                        inIndex = mCodec.dequeueInputBuffer(0);
                        if (inIndex >= 0) {
                            ByteBuffer inputBuffer = mCodec.getInputBuffer(inIndex);
                            if (packetLength > 0)
                                inputBuffer.put(accessUnitBuffer.array(), 0, packetLength);
                            mCodec.queueInputBuffer(inIndex, 0, packetLength, presentationTime, 0);
                            presentationTime += 100;
                            packetLength = 0;
                            accessUnitBuffer.clear();
                            accessUnitBuffer.rewind();
                        }
                    }
                    accessUnitBuffer.put(NAL_Units.get(i));
                    packetLength += NAL_Units.get(i).length;
                }
                MediaCodec.BufferInfo bufferinfo = new MediaCodec.BufferInfo();
                int outputBufferIndex = mCodec.dequeueOutputBuffer(bufferinfo, 0);
                handler.sendMessage(handler.obtainMessage(SHOWTOAST, "outputBufferIndex = "+ outputBufferIndex));
                while (outputBufferIndex >= 0){
                    mCodec.releaseOutputBuffer(outputBufferIndex, true);
                    outputBufferIndex = mCodec.dequeueOutputBuffer(bufferinfo, 0);
                }
            }
        };
        DJIDrone.getDjiCamera().setReceivedVideoDataCallBack(mReceivedVideoDataCallBack);
    }
    catch(IOException e)
    {
        e.printStackTrace();
    }
}

Answer 1

Try using MediaExtractor. If none of supported data sources (see setDataSource method) fits your needs, consider creating custom data source by implementing MediaDataSource

Also, please, note, that some codecs may need you to feed several input buffers before they produce single output buffer.

Answer 2

In media codec.configure() ,if you don't set surface ( set null), it will get the result of decoded and you need draw the image by yourself.

You can search the keyword: "media codec opengl" . you will find the sample.

PS. The format of output is uncertain. (Maybe RGBA)