What is the best way to draw YUV data to Surface in Android?

Question

I need to play video in a specific way. That's why I need to "pre-decode" some frames and keep it and show it later. I use MediaCodec to decode the video frames. The output buffers are YUV(420) format and the question is, what is the best way to show them later on screen?

1. Is it better to duplicate output of codec.GetOutputBuffer() or keeping code.GetOutputImage()?
2. Is there a hardware accelerated way to convert from YUV to RGB available as Java function?
3. How to draw the data to surface efficiently?
4. Is it necessary to make the conversion at all? Isn't it possible to draw the YUV data directly (in any form like ByteBuffer or Image or whatever) to Surface with a newer (>=8.0) Android version?
5. The mp4 codec cannot be instructed to produce RGB output instead of YUV, right?

I have browsed through multitude of similar (or same) questions, but mostly they were old and I just saw some manual YUV to RGB conversion which seems a too slow way to me. I was hoping that things got little better with newer Android versions ;)

Thank you for any ideas.

Answer 1

If you are using a SurfaceView or a TextureView then you don't need to convert from YUV to RGB. The Hardware Composer supports YCbCr formats out of the box.

First, instead of getting an output ByteBuffer from the MediaCodec, get an output Image. An Image contains all the information required for being able to draw the frame (pixel format, color planes, etc).

Second, use an ImageWriter:

ImageWriter.newInstance(surfaceHolder.surface, 1)
imageWriter.queueInputImage(image)

Important! If you are using a SurfaceView, then you should set the format and buffer size before drawing:

holder.setFormat(ImageFormat.YUV_420_888)
holder.setFixedSize(imageWidth, imageHeight)

Answer 2

Example of shaders to convert from rgb to yuv and convert it back to rgb. To make it simple i convert 2 adjacent rgba pixels convert them to yuv and save it into 1 pixels in the format of (y1y2uv). It's not a working example but I hope it can get you into the right direction.

// From rgb to yuv
companion object {
    val FRAGMENT_SHADER = "" +
        "precision highp float;\n" +
        "varying vec2 textureCoordinate;\n" +
        " \n" +
        "uniform sampler2D inputImageTexture;\n" +
        "uniform float step;\n" +
        "const vec3 multiplier = vec3(0.299, 0.587, 0.114);\n" +
        " \n" +
        "void main()\n" +
        "{\n" +
        "     vec4 first= texture2D(inputImageTexture, textureCoordinate);\n" +
        "     float y1 = clamp(dot(multiplier, first.rgb), 0.0, 1.0);\n" +
        "     vec2 secondCoord = textureCoordinate;\n" +
        "     secondCoord.x = secondCoord.x + step;\n" +
        "     vec4 second= texture2D(inputImageTexture, secondCoord);\n" +
        "     float y2 = clamp(dot(multiplier, second.rgb), 0.0, 1.0);\n" +
        "     vec3 rgbAve = mix(first.rgb, second.rgb, 0.5);\n" +
        "     float yAve = 0.5 * (y1 + y2);\n" +
        "     float u = clamp((rgbAve.b - yAve) * 0.565 + 0.5, 0.0, 1.0);\n" +
        "     float v = clamp((rgbAve.r - yAve) * 0.713 + 0.5, 0.0, 1.0);\n" +
        "     gl_FragColor.r = y1;\n" +
        "     gl_FragColor.g = y2;\n" +
        "     gl_FragColor.b = u;\n" +
        "     gl_FragColor.a = v;\n" +
        "}"
}

// From yuv to rgb
companion object {
    val FRAGMENT_SHADER = "" +
        "precision highp float;\n" +
        "varying vec2 textureCoordinate;\n" +
        " \n" +
        "uniform sampler2D inputImageTexture;\n" +
        "uniform float step;\n" +
        " \n" +
        "void main()\n" +
        "{\n" +
        "     vec2 secondCoord = textureCoordinate;\n" +
        "     float correction = secondCoord.x - 2.0 * step * floor(secondCoord.x / (2.0 * step));\n" +
        "     correction = clamp((correction - (step/2.0))/ (step/2.0), 0.0, 1.0);\n" +
        "     secondCoord.x = secondCoord.x - (correction * step);\n" +
        "     vec4 yuv = texture2D(inputImageTexture, secondCoord);\n" +
        "     yuv.r = (1.0 - correction) * yuv.r + correction * yuv.g;\n" +
        "     yuv.g = yuv.b - 0.5;\n" +
        "     yuv.b = yuv.a - 0.5;\n" +
        "     gl_FragColor.r = clamp(yuv.r + 1.403 * yuv.b, 0.0, 1.0);\n" +
        "     gl_FragColor.g = clamp(yuv.r - 0.344 * yuv.g - 0.714 * yuv.b, 0.0, 1.0);\n" +
        "     gl_FragColor.b = clamp(yuv.r + 1.770 * yuv.g, 0.0, 1.0);\n" +
        "     gl_FragColor.a = 1.0;\n" +
        "}"
}