glReadPixels is slow

Question

I am creating a custom camera preview using the GLSurfaceView, using OpenGl to render the frames given to me by the camera. I have the camera fully implemented and working how I would expect the camera to work with no fps loss and correct aspect ratios etc. But then the issue came when I needed to capture frames coming from the camera feed, my first thought was to use glReadPixles()

Using GLES20.glReadPixels() I find that some devices experience fps loss, it was mainly the devices with higher screen resolution this makes sense because glReadPixels needs to read more pixels with the higher resolution.

I did some digging and found others had similar issues with glReadPixels, and many suggested using a PBO, well using two of them acting as a double buffer which would allow me to read pixel data without blocking/stalling the current rendering process. I fully understand the concept of double buffering, I'm fairly new to OpenGL and need some guidance on how to get a double buffered PBO working.

I have found a few solutions to the PBO double buffering but I can never find a complete solution to fully understand how it interacts with GLES.

My implementation of the GLSurfaceView.Renderer.onDrawFrame()

    // mBuffer and mBitmap are declared and allocated outside of the onDrawFrame Method

    // Buffer is used to store pixel data from glReadPixels
    mBuffer.rewind();


    GLES20.glUseProgram(hProgram);
    if (tex_matrix != null)
    {
        GLES20.glUniformMatrix4fv(muTexMatrixLoc, 1, false, tex_matrix, 0);
    }
    GLES20.glUniformMatrix4fv(muMVPMatrixLoc, 1, false, mMvpMatrix, 0);

    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, tex_id);
    GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, GLConstants.VERTEX_NUM);
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);

    // Read pixels from the current GLES context
    GLES10.glReadPixels(0, 0, width, height, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, mBuffer);

    // Copy the Pixels from the buffer
    mBitmap.copyPixelsFromBuffer(mBuffer);

    GLES20.glUseProgram(0);

Answer 1

After a fair amount of research and digging I have found a solution to glReadPixels and how to use a PBO to buffer images/frames for later processing.

So the first thing we need to do is expose an additional function in GLES2. In your app module add a new directory called cpp, then create a new c file called GlesHelper (Or what ever you want to call it)

And paste the following code:

#include <jni.h>
#include <GLES2/gl2.h>
JNIEXPORT void JNICALL


// Change
Java_com_your_full_package_name_helper_GlesHelper_glReadPixels(JNIEnv *env, jobject instance, jint x,
                                                        jint y, jint width, jint height,
                                                        jint format, jint type) {
    // TODO
    glReadPixels(x, y, width, height, format, type, 0);
}

Then we you'll need to add a CMakeFile to the root your project. Right click, new file, enter CMakeLists.txt

And paste the following code

cmake_minimum_required(VERSION 3.4.1)

add_library( # Specifies the name of the library.
             native-lib

             # Sets the library as a shared library.
             SHARED

             # Provides a relative path to your source file(s).
             src/main//cpp//GlesHelper.c )

target_link_libraries( # Specifies the target library.
        native-lib

        # Links the target library to the log library
        # included in the NDK.
        ${log-lib}
        GLESv2)

Now open up your app/modules build.gradle file

Paste this in the android.defaultConfig section of the Gradle file

externalNativeBuild {
    // Encapsulates your CMake build configurations.
    cmake {
        // Provides a relative path to your CMake build script.
        cppFlags "-std=c++11 -fexceptions"
        arguments "-DANDROID_STL=c++_shared"
    }
}

Then paste this in the android section of the Gradle file

externalNativeBuild {
// Encapsulates your CMake build configurations.
    cmake {

        // Provides a relative path to your CMake build script.
        path "CMakeLists.txt"
    }
}

So that's all the MakeFile and c stuff all setup lets move onto some java

Create a new file in your project that matches the package in the c file i.e com_your_full_package_name_helper = com.your.full.package.name.helper

Ensure these match correctly, the same with the class name and function name.

So your class should look like this

package com.your.full.package.name.helper;

public class GlesHelper
{
    public static native void glReadPixels(int x, int y, int width, int height, int format, int type);
}

Because we have added native code to the project we need to use the System.loadLibrary("native-lib") to load in our new method.

Before we start the next bit add these member variables to your Renderer

/**
 * The PBO Ids, increase the allocate amount for more PBO's
 * The more PBO's the smoother the frame rate (to an extent)
 * Side affect of having more PBO's the frames you get from the PBO's will lag behind by the amount of pbo's
 */
private IntBuffer mPboIds = IntBuffer.allocate(2);;

/**
 * The current PBO Index
 */
private int mCurrentPboIndex = 0;

/**
 * The next PBO Index
 */
private int mNextPboIndex = 1;

So now we need to initialise our PBO's this is pretty simple

    // Generate the buffers for the pbo's
    GLES30.glGenBuffers(mPboIds.capacity(), mPboIds);

    // Loop for how many pbo's we have
    for (int i = 0; i < mPboIds.capacity(); i++)
    {
        // Bind the Pixel_Pack_Buffer to the current pbo id
        GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, mPboIds.get(i));

        // Buffer empty data, capacity is the width * height * 4
        GLES30.glBufferData(GLES30.GL_PIXEL_PACK_BUFFER, capacity, null, GLES30.GL_STATIC_READ);
    }

    // Reset the current buffer so we can draw properly
    GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, 0);

Then before we begin drawing call this method, this will read pixel data into the pbo, swap buffers and give you access to the pixel data.

/**
 * Reads the pixels from the PBO and swaps the buffers
 */
private void readPixelsFromPBO()
{
    // Bind the current buffer
    GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, mPboIds.get(mCurrentPboIndex));

    // Read pixels into the bound buffer
    GlesHelper.glReadPixels(0, 0, mViewWidth, mViewHeight, GLES20.GL_RGBA, GLES30.GL_UNSIGNED_BYTE);

    // Bind the next buffer
    GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, mPboIds.get(mNextPboIndex));

    // Map to buffer to a byte buffer, this is our pixel data
    ByteBuffer pixelsBuffer = (ByteBuffer) GLES30.glMapBufferRange(GLES30.GL_PIXEL_PACK_BUFFER, 0, mViewWidth * mViewHeight * 4, GLES30.GL_MAP_READ_BIT);

    if(mSkipFirstFrame)
    {
        // Skip the first frame as the PBO's have nothing in them until the second render cycle
    }
    // Set skip first frame to true so we can begin frame processing
    mSkipFirstFrame = true;

    // Swap the buffer index
    mCurrentPboIndex = (mCurrentPboIndex + 1) % mPboIds.capacity();
    mNextPboIndex = (mNextPboIndex + 1) % mPboIds.capacity();

    // Unmap the buffers
    GLES30.glUnmapBuffer(GLES30.GL_PIXEL_PACK_BUFFER);
    GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, GLES20.GL_NONE);
    GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, GLES20.GL_NONE);
}

So going back to my initial question our Redner/onDrawMethod would look something like this.

    // Use the OpenGL Program for rendering
    GLES20.glUseProgram(mProgram);

    // If the Texture Matrix is not null
    if (textureMatrix != null)
    {
        // Apply the Matrix
        GLES20.glUniformMatrix4fv(mTexMatrixLoc, 1, false, textureMatrix, 0);
    }


    // Apply the Matrix
    GLES20.glUniformMatrix4fv(mMVPMatrixLoc, 1, false, mMvpMatrix, 0);

    // Bind the Texture
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textureID);

    // Draw the texture
    GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, GLConstants.VERTEX_NUM);

    // Unbind the Texture
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);

    // Read from PBO
    readPixelsFromPBO()

I hope this helps someone who's having a similar issue with the performance with glReadPixels or at least struggling to implement PBO's