Improper decoding of raw H264 video stream using android MediaCodec

Question

I am building a screen sharing app and trying to receive raw H264 video stream from a socket and displaying it using surfaceview by decoding using MediaCodec class. However I am successfully able to receive data and display on surface. But the problem where I am stuck from two days is the video is very laggy,jittery and having green bands and patches. You can see the issue in the youtube video link. If I save this video to sdcard and play it using MxPlayer then it is playing fine.Also I tried with Gstreamer and everything is fine. This is my class calling from Activity when surfaceview is created.

//New Edited Code
public class Server {
static final int socketServerPORT = 53515;
MainActivity activity;
ServerSocket serverSocket;

public Server(MainActivity activity, Surface surface) {
    Log.e("constructor()", "called");
    this.activity = activity;
    Thread socketServerThread = new Thread(new SocketServerThread(surface));
    socketServerThread.start();
}

private static MediaCodecInfo selectCodec(String mimeType) {
    int numCodecs = MediaCodecList.getCodecCount();
    for (int i = 0; i < numCodecs; i++) {
        MediaCodecInfo codecInfo = MediaCodecList.getCodecInfoAt(i);

        if (codecInfo.isEncoder()) {
            continue;
        }

        String[] types = codecInfo.getSupportedTypes();
        for (int j = 0; j < types.length; j++) {
            if (types[j].equalsIgnoreCase(mimeType)) {
                Log.e("codecinfo", codecInfo.getName());
                return codecInfo;
            }
        }
    }
    return null;
}

private class SocketServerThread extends Thread {
    InputStream is;
    Socket socket;
    private MediaCodec codec;
    private Surface surface;

    public SocketServerThread(Surface surface) {
        this.surface = surface;
    }

    @Override
    public void run() {
        Log.e("socketthread", "called");
        try {
            selectCodec("video/avc");
            codec = MediaCodec.createByCodecName(selectCodec("video/avc").getName());

            MediaFormat format = MediaFormat.createVideoFormat("video/avc", 800, 480);
            //  format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
            format.setInteger(MediaFormat.KEY_BIT_RATE, 1024000);
            format.setInteger(MediaFormat.KEY_FRAME_RATE, 25);
            codec.configure(format, surface, null, 0);
            codec.start();
            serverSocket = new ServerSocket(socketServerPORT);
            while (true) {
                socket = serverSocket.accept();
                Log.e("connection", "accepted");
                is = socket.getInputStream();
                if (is != null) {
                    //          File file = new File(Environment.getExternalStorageDirectory() + "/stream.mp4");
                    //              OutputStream output = new FileOutputStream(file);
                    byte[] buff = new byte[4 * 1024]; // or other buffer size
                    int read;
                    while ((read = is.read(buff)) != -1) {
                        //              output.write(buff, 0, read);
                        if (buff.length == 1)
                            continue;

                        int inIndex = codec.dequeueInputBuffer(10000);
                        if (inIndex >= 0) {
                            ByteBuffer inputBuffer = codec.getInputBuffer(inIndex);
                            inputBuffer.clear();
                            inputBuffer.put(buff);

                            codec.queueInputBuffer(inIndex, 0, buff.length, 16, 0);
                        }

                        MediaCodec.BufferInfo buffInfo = new MediaCodec.BufferInfo();
                        int outIndex = codec.dequeueOutputBuffer(buffInfo, 10000);

                        switch (outIndex) {
                            case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
                                break;
                            case MediaCodec.INFO_TRY_AGAIN_LATER:
                                break;
                            case -3:
                                break;
                            default:
                                codec.releaseOutputBuffer(outIndex, true);
                        }
                    }
                }
            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (codec != null) {
                codec.release();
            }
            if (socket != null) {
                try {
                    socket.close();
                    is.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
  }
}

I tried the way you mentioned to create MediaCodec as

MediaCodec.createByCodecName(selectCodec("video/avc").getName());

   private static MediaCodecInfo selectCodec(String mimeType) {
    int numCodecs = MediaCodecList.getCodecCount();
    for (int i = 0; i < numCodecs; i++) {
        MediaCodecInfo codecInfo = MediaCodecList.getCodecInfoAt(i);

        if (!codecInfo.isEncoder()) {
            continue;
        }

        String[] types = codecInfo.getSupportedTypes();
        for (int j = 0; j < types.length; j++) {
            if (types[j].equalsIgnoreCase(mimeType)) {
                Log.e("codecinfo",codecInfo.getName());
                return codecInfo;
            }
        }
    }
    return null;
}

but it gives me error

I/OMXClient: MuxOMX ctor
I/MediaCodec: [OMX.qcom.video.encoder.avc] setting surface generation   to 28345345
W/ACodec: [OMX.qcom.video.encoder.avc] Failed to set standard   component role 'video_decoder.avc'.
E/ACodec: [OMX.qcom.video.encoder.avc] configureCodec returning error -1010
E/ACodec: signalError(omxError 0x80001001, internalError -1010)
E/MediaCodec: Codec reported err 0xfffffc0e, actionCode 0, while in state 3
E/MediaCodec: configure failed with err 0xfffffc0e, resetting...
I/OMXClient: MuxOMX ctor
E/AndroidRuntime: FATAL EXCEPTION: Thread-5
Process: com.androidsrc.server, PID: 27681
android.media.MediaCodec$CodecException: Error 0xfffffc0e
at android.media.MediaCodec.configure(MediaCodec.java:1884)
at com.androidsrc.server.Server$SocketServerThread.run(Server.java:70)
at java.lang.Thread.run(Thread.java:761)

Also I tried to create Mediacodec earlier by this method

codec =  MediaCodec.createDecoderByType("video/avc");

but the video quality remains same as in the video shared on youtube.

Answer 1

Try to reduce timeout on dequeue (codec.dequeueInputBuffer(10000);), unless you think it does take so much time to decode.

Also setup following flags:

mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, CodecCapabilities.COLOR_FormatSurface); mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);

You mentioned that you changed code to select by type but original post had errors when decoder was created. Are you able to fix those errors?

You do need to use hardware decoder to ensure that you can play without significant performance cost. New video definitely is worst than first attempt, which means you had decoder selection issue. You can use ffplay with -f h264 option and confirm that format is correct.

Lastly, look into video dimensions and you can read back buffer from decoder (by passing false in releaseOutputBuffer) and write it to disk to confirm your dimensions are correct. Raw video to final video will have different dimension. Read width, height from codecFormat to confirm it is able to decode correctly.

Answer 2

I'd rather comment this but still missing the required points :D

I think you (or i'm not aware that it' works otherwise) are missing some important bits. The video you provided shows signs of missing information on how to interpret things.

• Are you sending and using SPS and PPS? I'm not sure if the MediaFormat you create contains all the needed information.

• MediaCodec requires entire frames to be fed to it. Are you sure that you receive a full frame using this sort of logic?

• Some Decoders expect the frames without a NAL header. Are you sending frames without NAL headers?

• I don't see you transfer any presentation time

I suggest you Look at RTP/RTSP protocol on how to transfer media. To further help you i suggest you provide the producer logic as well

Answer 3

OMX.google.h264.decoder is a software codec with a limited functionality (at least on some devices and api levels).

Try to use createDecoderByType instead of createDecoderByName or to choose another codec as in the example here: https://developer.android.com/reference/android/media/MediaCodecInfo.html (modify it to choose decoder other than OMX.google.h264.decoder)