YUV_420_888 interpretation on Samsung Galaxy S7 (Camera2)

Question

I wrote a conversion from YUV_420_888 to Bitmap, considering the following logic (as I understand it):

To summarize the approach: the kernel’s coordinates x and y are congruent both with the x and y of the non-padded part of the Y-Plane (2d-allocation) and the x and y of the output-Bitmap. The U- and V-Planes, however, have a different structure than the Y-Plane, because they use 1 byte for coverage of 4 pixels, and, in addition, may have a PixelStride that is more than one, in addition they might also have a padding that can be different from that of the Y-Plane. Therefore, in order to access the U’s and V’s efficiently by the kernel I put them into 1-d allocations and created an index “uvIndex” that gives the position of the corresponding U- and V within that 1-d allocation, for given (x,y) coordinates in the (non-padded) Y-plane (and, so, the output Bitmap).

In order to keep the rs-Kernel lean, I excluded the padding area in the yPlane by capping the x-range via LaunchOptions (this reflects the RowStride of the y-plane which thus can be ignored WITHIN the kernel). So we just need to consider the uvPixelStride and uvRowStride within the uvIndex, i.e. the index used in order to access to the u- and v-values.

This is my code:

Renderscript Kernel, named yuv420888.rs

#pragma version(1)
#pragma rs java_package_name(com.xxxyyy.testcamera2);
#pragma rs_fp_relaxed

int32_t width;
int32_t height;

uint picWidth, uvPixelStride, uvRowStride;
rs_allocation ypsIn, uIn, vIn;

// The LaunchOptions ensure that the Kernel does not enter the padding  zone of Y,
// so yRowStride can be ignored WITHIN the Kernel.
uchar4 __attribute__((kernel)) doConvert(uint32_t x, uint32_t y) {
    // index for accessing the uIn's and vIn's
    uint uvIndex= uvPixelStride * (x/2) + uvRowStride*(y/2);

    // get the y,u,v values
    uchar yps= rsGetElementAt_uchar(ypsIn, x, y);
    uchar u= rsGetElementAt_uchar(uIn, uvIndex);
    uchar v= rsGetElementAt_uchar(vIn, uvIndex);

    // calc argb
    int4 argb;
    argb.r = yps + v * 1436 / 1024 - 179;
    argb.g = yps -u * 46549 / 131072 + 44 -v * 93604 / 131072 + 91;
    argb.b = yps +u * 1814 / 1024 - 227;
    argb.a = 255;

    uchar4 out = convert_uchar4(clamp(argb, 0, 255));
    return out;
}

Java side:

private Bitmap YUV_420_888_toRGB(Image image, int width, int height){
    // Get the three image planes
    Image.Plane[] planes = image.getPlanes();
    ByteBuffer buffer = planes[0].getBuffer();
    byte[] y = new byte[buffer.remaining()];
    buffer.get(y);

    buffer = planes[1].getBuffer();
    byte[] u = new byte[buffer.remaining()];
    buffer.get(u);

    buffer = planes[2].getBuffer();
    byte[] v = new byte[buffer.remaining()];
    buffer.get(v);

    // get the relevant RowStrides and PixelStrides
    // (we know from documentation that PixelStride is 1 for y)
    int yRowStride= planes[0].getRowStride();
    int uvRowStride= planes[1].getRowStride();  // we know from   documentation that RowStride is the same for u and v.
    int uvPixelStride= planes[1].getPixelStride();  // we know from   documentation that PixelStride is the same for u and v.


    // rs creation just for demo. Create rs just once in onCreate and use it again.
    RenderScript rs = RenderScript.create(this);
    //RenderScript rs = MainActivity.rs;
    ScriptC_yuv420888 mYuv420=new ScriptC_yuv420888 (rs);

    // Y,U,V are defined as global allocations, the out-Allocation is the Bitmap.
    // Note also that uAlloc and vAlloc are 1-dimensional while yAlloc is 2-dimensional.
    Type.Builder typeUcharY = new Type.Builder(rs, Element.U8(rs));

    //using safe height
    typeUcharY.setX(yRowStride).setY(y.length / yRowStride);

    Allocation yAlloc = Allocation.createTyped(rs, typeUcharY.create());
    yAlloc.copyFrom(y);
    mYuv420.set_ypsIn(yAlloc);

    Type.Builder typeUcharUV = new Type.Builder(rs, Element.U8(rs));
    // note that the size of the u's and v's are as follows:
    //      (  (width/2)*PixelStride + padding  ) * (height/2)
    // =    (RowStride                          ) * (height/2)
    // but I noted that on the S7 it is 1 less...
    typeUcharUV.setX(u.length);
    Allocation uAlloc = Allocation.createTyped(rs, typeUcharUV.create());
    uAlloc.copyFrom(u);
    mYuv420.set_uIn(uAlloc);

    Allocation vAlloc = Allocation.createTyped(rs, typeUcharUV.create());
    vAlloc.copyFrom(v);
    mYuv420.set_vIn(vAlloc);

    // handover parameters
    mYuv420.set_picWidth(width);
    mYuv420.set_uvRowStride (uvRowStride);
    mYuv420.set_uvPixelStride (uvPixelStride);

    Bitmap outBitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
    Allocation outAlloc = Allocation.createFromBitmap(rs, outBitmap, Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT);

    Script.LaunchOptions lo = new Script.LaunchOptions();
    lo.setX(0, width);  // by this we ignore the y’s padding zone, i.e. the right side of x between width and yRowStride
    //using safe height
    lo.setY(0, y.length / yRowStride);

    mYuv420.forEach_doConvert(outAlloc,lo);
    outAlloc.copyTo(outBitmap);

    return outBitmap;
}

Testing on Nexus 7 (API 22) this returns nice color Bitmaps. This device, however, has trivial pixelstrides (=1) and no padding (i.e. rowstride=width). Testing on the brandnew Samsung S7 (API 23) I get pictures whose colors are not correct - except of the green ones. But the Picture does not show a general bias towards green, it just seems that non-green colors are not reproduced correctly. Note, that the S7 applies an u/v pixelstride of 2, and no padding.

Since the most crucial code line is within the rs-code the Access of the u/v planes uint uvIndex= (...) I think, there could be the problem, probably with incorrect consideration of pixelstrides here. Does anyone see the solution? Thanks.

UPDATE: I checked everything, and I am pretty sure that the code regarding the access of y,u,v is correct. So the problem must be with the u and v values themselves. Non green colors have a purple tilt, and looking at the u,v values they seem to be in a rather narrow range of about 110-150. Is it really possible that we need to cope with device specific YUV -> RBG conversions...?! Did I miss anything?

UPDATE 2: have corrected above code, it works now, thanks to Eddy's Feedback (see edit history to find wrong code).

Answer 1

An FYI in case someone else gets this as I was also getting "android.support.v8.renderscript.RSIllegalArgumentException: Array too small for allocation type" when trying out the code. In my case it turns out that the when allocating the buffer for Y i had to rewind the buffer because it was being left at the wrong end and wasn't copying the data. By doing buffer.rewind(); before allocation the new bytes array makes it work fine now.

Answer 2

Posting full solution to convert YUV->BGR (can be adopted for other formats too) and also rotate image to upright using renderscript. Allocation is used as input and byte array is used as output. It was tested on Android 8+ including Samsung devices too.

Java

/**
 * Renderscript-based process to convert YUV_420_888 to BGR_888 and rotation to upright.
 */
public class ImageProcessor {

    protected final String TAG = this.getClass().getSimpleName();

    private Allocation mInputAllocation;
    private Allocation mOutAllocLand;
    private Allocation mOutAllocPort;

    private Handler mProcessingHandler;
    private ScriptC_yuv_bgr mConvertScript;
    private byte[] frameBGR;

    public ProcessingTask mTask;
    private ImageListener listener;
    private Supplier<Integer> rotation;

    public ImageProcessor(RenderScript rs, Size dimensions, ImageListener listener, Supplier<Integer> rotation) {

        this.listener = listener;
        this.rotation = rotation;
        int w = dimensions.getWidth();
        int h = dimensions.getHeight();

        Type.Builder yuvTypeBuilder = new Type.Builder(rs, Element.YUV(rs));
        yuvTypeBuilder.setX(w);
        yuvTypeBuilder.setY(h);
        yuvTypeBuilder.setYuvFormat(ImageFormat.YUV_420_888);
        mInputAllocation = Allocation.createTyped(rs, yuvTypeBuilder.create(),
                Allocation.USAGE_IO_INPUT | Allocation.USAGE_SCRIPT);

        //keep 2 allocations to handle different image rotations
        mOutAllocLand = createOutBGRAlloc(rs, w, h);
        mOutAllocPort = createOutBGRAlloc(rs, h, w);

        frameBGR = new byte[w*h*3];

        HandlerThread processingThread = new HandlerThread(this.getClass().getSimpleName());
        processingThread.start();
        mProcessingHandler = new Handler(processingThread.getLooper());

        mConvertScript = new ScriptC_yuv_bgr(rs);
        mConvertScript.set_inWidth(w);
        mConvertScript.set_inHeight(h);

        mTask = new ProcessingTask(mInputAllocation);
    }

    private Allocation createOutBGRAlloc(RenderScript rs, int width, int height) {
        //Stored as Vec4, it's impossible to store as Vec3, buffer size will be for Vec4 anyway
        //using RGB_888 as alternative for BGR_888, can be just U8_3 type
        Type.Builder rgbTypeBuilderPort = new Type.Builder(rs, Element.RGB_888(rs));
        rgbTypeBuilderPort.setX(width);
        rgbTypeBuilderPort.setY(height);
        Allocation allocation = Allocation.createTyped(
            rs, rgbTypeBuilderPort.create(), Allocation.USAGE_SCRIPT
        );
        //Use auto-padding to be able to copy to x*h*3 bytes array
        allocation.setAutoPadding(true);

        return allocation;
    }

    public Surface getInputSurface() {
        return mInputAllocation.getSurface();
    }

    /**
     * Simple class to keep track of incoming frame count,
     * and to process the newest one in the processing thread
     */
    class ProcessingTask implements Runnable, Allocation.OnBufferAvailableListener {

        private int mPendingFrames = 0;

        private Allocation mInputAllocation;

        public ProcessingTask(Allocation input) {
            mInputAllocation = input;
            mInputAllocation.setOnBufferAvailableListener(this);
        }

        @Override
        public void onBufferAvailable(Allocation a) {
            synchronized(this) {
                mPendingFrames++;
                mProcessingHandler.post(this);
            }
        }

        @Override
        public void run() {
            // Find out how many frames have arrived
            int pendingFrames;
            synchronized(this) {
                pendingFrames = mPendingFrames;
                mPendingFrames = 0;

                // Discard extra messages in case processing is slower than frame rate
                mProcessingHandler.removeCallbacks(this);
            }

            // Get to newest input
            for (int i = 0; i < pendingFrames; i++) {
                mInputAllocation.ioReceive();
            }

            int rot = rotation.get();

            mConvertScript.set_currentYUVFrame(mInputAllocation);
            mConvertScript.set_rotation(rot);

            Allocation allocOut = rot==90 || rot== 270 ? mOutAllocPort : mOutAllocLand;

            // Run processing
            // ain allocation isn't really used, global frame param is used to get data from
            mConvertScript.forEach_yuv_bgr(allocOut);

            //Save to byte array, BGR 24bit
            allocOut.copyTo(frameBGR);

            int w = allocOut.getType().getX();
            int h = allocOut.getType().getY();

            if (listener != null) {
                listener.onImageAvailable(frameBGR, w, h);
            }
        }
    }

    public interface ImageListener {

        /**
         * Called when there is available image, image is in upright position.
         *
         * @param bgr BGR 24bit bytes
         * @param width image width
         * @param height image height
         */
        void onImageAvailable(byte[] bgr, int width, int height);
    }
}

RS

#pragma version(1)
#pragma rs java_package_name(com.affectiva.camera)
#pragma rs_fp_relaxed

//Script convers YUV to BGR(uchar3)

//current YUV frame to read pixels from
rs_allocation currentYUVFrame;

//input image rotation: 0,90,180,270 clockwise
uint32_t rotation;
uint32_t inWidth;
uint32_t inHeight;

//method returns uchar3  BGR which will be set to x,y in output allocation
uchar3 __attribute__((kernel)) yuv_bgr(uint32_t x, uint32_t y) {

    // Read in pixel values from latest frame - YUV color space

    uchar3 inPixel;
    uint32_t xRot = x;
    uint32_t yRot = y;

    //Do not rotate if 0
    if (rotation==90) {
      //rotate 270 clockwise
      xRot = y;
      yRot = inHeight - 1 - x;
    } else if (rotation==180) {
      xRot = inWidth - 1 - x;
      yRot = inHeight - 1 - y;
    } else if (rotation==270) {
      //rotate 90 clockwise
      xRot = inWidth - 1 - y;
      yRot = x;
    }

    inPixel.r = rsGetElementAtYuv_uchar_Y(currentYUVFrame, xRot, yRot);
    inPixel.g = rsGetElementAtYuv_uchar_U(currentYUVFrame, xRot, yRot);
    inPixel.b = rsGetElementAtYuv_uchar_V(currentYUVFrame, xRot, yRot);

    // Convert YUV to RGB, JFIF transform with fixed-point math
    // R = Y + 1.402 * (V - 128)
    // G = Y - 0.34414 * (U - 128) - 0.71414 * (V - 128)
    // B = Y + 1.772 * (U - 128)

    int3 bgr;
    //get red pixel and assing to b
    bgr.b = inPixel.r +
            inPixel.b * 1436 / 1024 - 179;
    bgr.g = inPixel.r -
            inPixel.g * 46549 / 131072 + 44 -
            inPixel.b * 93604 / 131072 + 91;
    //get blue pixel and assign to red
    bgr.r = inPixel.r +
            inPixel.g * 1814 / 1024 - 227;

    // Write out
    return convert_uchar3(clamp(bgr, 0, 255));
}

Answer 3

Re: RSIllegalArgumentException

In my case this was the case that buffer.remaining() was not multiple of stride: The length of last line was less than stride (i.e. only up to where actual data was.)

Answer 4

Furthermore for anyone else getting

android.support.v8.renderscript.RSIllegalArgumentException: Array too small for allocation type

I fixed it by changing yAlloc.copyFrom(y); to yAlloc.copy1DRangeFrom(0, y.length, y);

Answer 5

For people who encounter error

android.support.v8.renderscript.RSIllegalArgumentException: Array too small for allocation type

use buffer.capacity() instead of buffer.remaining()

and if you already made some operations on the image, you'll need to call rewind() method on the buffer.

Answer 6

This code requires the use of the RenderScript compatibility library (android.support.v8.renderscript.*).

In order to get the compatibility library to work with Android API 23, I updated to gradle-plugin 2.1.0 and Build-Tools 23.0.3 as per Miao Wang's answer at How to create Renderscript scripts on Android Studio, and make them run?

If you follow his answer and get an error "Gradle version 2.10 is required" appears, do NOT change

classpath 'com.android.tools.build:gradle:2.1.0'

Instead, update the distributionUrl field of the Project\gradle\wrapper\gradle-wrapper.properties file to

distributionUrl=https\://services.gradle.org/distributions/gradle-2.10-all.zip

and change File > Settings > Builds,Execution,Deployment > Build Tools > Gradle >Gradle to Use default gradle wrapper as per "Gradle Version 2.10 is required." Error.

Answer 7

On a Samsung Galaxy Tab 5 (Tablet), android version 5.1.1 (22), with alleged YUV_420_888 format, the following renderscript math works well and produces correct colors:

uchar yValue    = rsGetElementAt_uchar(gCurrentFrame, x + y * yRowStride);
uchar vValue    = rsGetElementAt_uchar(gCurrentFrame, ( (x/2) + (y/4) * yRowStride ) + (xSize * ySize) );
uchar uValue    = rsGetElementAt_uchar(gCurrentFrame, ( (x/2) + (y/4) * yRowStride ) + (xSize * ySize) + (xSize * ySize) / 4);

I do not understand why the horizontal value (i.e., y) is scaled by a factor of four instead of two, but it works well. I also needed to avoid use of rsGetElementAtYuv_uchar_Y|U|V. I believe the associated allocation stride value is set to zero instead of something proper. Use of rsGetElementAt_uchar() is a reasonable work-around.

On a Samsung Galaxy S5 (Smart Phone), android version 5.0 (21), with alleged YUV_420_888 format, I cannot recover the u and v values, they come through as all zeros. This results in a green looking image. Luminous is OK, but image is vertically flipped.

Answer 8

Look at

floor((float) uvPixelStride*(x)/2)

which calculates your U,V row offset (uv_row_offset) from the Y x-coordinate.

if uvPixelStride = 2, then as x increases:

x = 0, uv_row_offset = 0
x = 1, uv_row_offset = 1
x = 2, uv_row_offset = 2
x = 3, uv_row_offset = 3

and this is incorrect. There's no valid U/V pixel value at uv_row_offset = 1 or 3, since uvPixelStride = 2.

You want

uvPixelStride * floor(x/2)

(assuming you don't trust yourself to remember the critical round-down behavior of integer divide, if you do then):

uvPixelStride * (x/2)

should be enough

With that, your mapping becomes:

x = 0, uv_row_offset = 0
x = 1, uv_row_offset = 0
x = 2, uv_row_offset = 2
x = 3, uv_row_offset = 2

See if that fixes the color errors. In practice, the incorrect addressing here would mean every other color sample would be from the wrong color plane, since it's likely that the underlying YUV data is semiplanar (so the U plane starts at V plane + 1 byte, with the two planes interleaved)