How to resize CVPixelBufferRef 420f with preserving aspect ratio?

Question

I am trying to resize a pixel buffer with the kCVPixelFormatType_420YpCbCr8BiPlanarFullRange (420f) pixel format to another size with preserving aspect ratio and adding black bars (if needed).

I am using the vImageScale_Planar8 and vImageRotate90_Planar8 functions from the Accelerate framework to scale and rotate the Y plane, and the vImageScale_CbCr8 and vImageRotate90_Planar16U functions to scale and rotate the CbCr plane.

Here is the code:

+ (CVPixelBufferRef)resizeProportionallyBuffer:(CVPixelBufferRef)srcPixelBuffer withOrientation:(CGImagePropertyOrientation)orientation toSize:(CGSize)dstSize {
    OSType pixelFormat = CVPixelBufferGetPixelFormatType(srcPixelBuffer);
    
    CVPixelBufferLockFlags srcFlags = kCVPixelBufferLock_ReadOnly;
    CVPixelBufferLockBaseAddress(srcPixelBuffer, srcFlags);
    
    void *srcLuminanceData = CVPixelBufferGetBaseAddressOfPlane(srcPixelBuffer, 0);
    size_t srcLuminanceWidth = CVPixelBufferGetWidthOfPlane(srcPixelBuffer, 0);//750
    size_t srcLuminanceHeight = CVPixelBufferGetHeightOfPlane(srcPixelBuffer, 0);//1334
    size_t srcLuminanceBytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(srcPixelBuffer, 0);//768
    
    void *srcCbCrData = CVPixelBufferGetBaseAddressOfPlane(srcPixelBuffer, 1);
    int srcCbCrWidth = (int)CVPixelBufferGetWidthOfPlane(srcPixelBuffer, 1);//375
    int srcCbCrHeight = (int)CVPixelBufferGetHeightOfPlane(srcPixelBuffer, 1);//667
    size_t srcCbCrBytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(srcPixelBuffer, 1);//768
    
    vImage_Buffer srcLuminanceBuffer = {
        .data = srcLuminanceData,
        .width = srcLuminanceWidth,
        .height = srcLuminanceHeight,
        .rowBytes = srcLuminanceBytesPerRow
    };
    
    vImage_Buffer srcCbCrBuffer = {
        .data = srcCbCrData,
        .width = srcCbCrWidth,
        .height = srcCbCrHeight,
        .rowBytes = srcCbCrBytesPerRow
    };
    
    size_t srcWidth = srcLuminanceWidth;
    size_t srcHeight = srcLuminanceHeight;
    int dstWidth = (int)dstSize.width;
    int dstHeight = (int)dstSize.height;
    uint8_t rotationConstant = 0;
    size_t scaledWidth = srcLuminanceWidth;
    size_t scaledHeight = srcLuminanceHeight;
    if (orientation == kCGImagePropertyOrientationUp || orientation == kCGImagePropertyOrientationDown) {
        BOOL srcIsWider = dstHeight * srcWidth > dstWidth * srcHeight;
        scaledWidth = srcIsWider ? dstWidth : (size_t)round((CGFloat)(dstHeight * srcWidth) / (CGFloat)srcHeight);
        scaledHeight = srcIsWider ? (size_t)round((CGFloat)(dstWidth * srcHeight) / (CGFloat)srcWidth) : dstHeight;
        rotationConstant = (orientation == kCGImagePropertyOrientationUp) ? kRotate0DegreesClockwise : kRotate180DegreesClockwise;
    } else if (orientation == kCGImagePropertyOrientationLeft || orientation == kCGImagePropertyOrientationRight) {
        BOOL srcIsWider = dstHeight * srcHeight > dstWidth * srcWidth;
        scaledHeight = srcIsWider ? dstWidth: (size_t)round((CGFloat)(srcHeight * dstHeight) / (CGFloat)srcWidth);
        scaledWidth = srcIsWider ? (size_t)round((CGFloat)(srcWidth * dstWidth) / (CGFloat)srcHeight) : dstHeight;
        rotationConstant = (orientation == kCGImagePropertyOrientationLeft) ? kRotate90DegreesClockwise : kRotate90DegreesCounterClockwise;
    }
    
    Pixel_8 backColor = 0;
    
    CVPixelBufferRef dstPixelBuffer = NULL;
    CVReturn result = CVPixelBufferCreate(kCFAllocatorDefault,
                                          dstWidth,
                                          dstHeight,
                                          pixelFormat,
                                          nil,
                                          &dstPixelBuffer);
    CVPixelBufferLockBaseAddress(dstPixelBuffer, 0);
    
    void *dstLuminanceData = CVPixelBufferGetBaseAddressOfPlane(dstPixelBuffer, 0);
    int dstLuminanceWidth = (int)CVPixelBufferGetWidthOfPlane(dstPixelBuffer, 0);//1280
    int dstLuminanceHeight = (int)CVPixelBufferGetHeightOfPlane(dstPixelBuffer, 0);//720
    size_t dstLuminanceBytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(dstPixelBuffer, 0);//1280
    
    void *dstCbCrData = CVPixelBufferGetBaseAddressOfPlane(dstPixelBuffer, 1);
    int dstCbCrWidth = (int)CVPixelBufferGetWidthOfPlane(dstPixelBuffer, 1);//640
    int dstCbCrHeight = (int)CVPixelBufferGetHeightOfPlane(dstPixelBuffer, 1);//360
    size_t dstCbCrBytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(dstPixelBuffer, 1);//1280
    
    vImage_Buffer dstLuminanceBuffer = {
        .data = dstLuminanceData,
        .width = dstLuminanceWidth,
        .height = dstLuminanceHeight,
        .rowBytes = dstLuminanceBytesPerRow
    };
    
    vImage_Buffer dstCbCrBuffer = {
        .data = dstCbCrData,
        .width = dstCbCrWidth,
        .height = dstCbCrHeight,
        .rowBytes = dstCbCrBytesPerRow
    };
    
    void *dstLuminanceIntermediateBufferData = malloc(scaledWidth*scaledHeight);
    vImage_Buffer dstLuminanceIntermediateBuffer = {
        .data = dstLuminanceIntermediateBufferData,
        .width = scaledWidth,
        .height = scaledHeight,
        .rowBytes = scaledWidth
    };
    vImage_Error error = vImageScale_Planar8(&srcLuminanceBuffer, &dstLuminanceIntermediateBuffer, nil, kvImageNoFlags);
    error = vImageRotate90_Planar8(&dstLuminanceIntermediateBuffer, &dstLuminanceBuffer, rotationConstant, backColor, kvImageBackgroundColorFill);
    free(dstLuminanceIntermediateBufferData);

    void *dstCbCrIntermediateBufferData = malloc(scaledWidth*scaledHeight);
    vImage_Buffer dstCbCrIntermediateBuffer = {
        .data = dstCbCrIntermediateBufferData,
        .width = scaledWidth/2,
        .height = scaledHeight/2,
        .rowBytes = scaledWidth
    };
    error = vImageScale_CbCr8(&srcCbCrBuffer, &dstCbCrIntermediateBuffer, nil, kvImageNoFlags);
    error = vImageRotate90_Planar16U(&dstCbCrIntermediateBuffer, &dstCbCrBuffer, rotationConstant, 127, kvImageBackgroundColorFill);
    free(dstCbCrIntermediateBufferData);
    
    CVPixelBufferUnlockBaseAddress(srcPixelBuffer, srcFlags);
    CVPixelBufferUnlockBaseAddress(dstPixelBuffer, 0);
    
    return dstPixelBuffer;
}

The code works, but when an image has the kCGImagePropertyOrientationLeft or kCGImagePropertyOrientationRight orientation, green bars are added.

How can I fix this problem?

Answer 1

Images in both coregraphics and corevideo can have an orientation tag. This means that the image may be rotated or flipped in the storage container relative to how it is supposed to be drawn. VImage however doesn’t have any such concept. As far as it is concerned, pixels is pixels. Where things go wrong is when the logical height and width don’t match the physical height and width. VImage is concerned only with the physical height and width, whereas these other APIs will likely feed you the logical height and width, as it would appear on screen so you can position it correctly. (They regard the storage format as an Implementation detail, while VImage does not.) When the rotation is left or right, I would suggest trying swapping the height and width when populating the vImage_Buffer structure so that it matches the storage orientation.

If the stripe is really tiny for a resampling filter, like 1-3 pixels wide, then I’d probably also look at whether I’d correctly described the size of the image. Resampling filters will look at at least the 6 nearest src pixels to the center of the new pixel. If some of those are garbage, particularly off the right or bottom of the image, then you get garbage in / garbage out. Vimage is very careful not to touch nonexistent or logically nonexistent pixels, but if the data is incorrectly described in the vimage_buffer struct then it may be licensed to use that data and then you’ll get stripes at edges and/or a crash.

Note that vimage scale is not appropriate for tiled images. It will need to see the surrounding area outside the tile to work correctly, which is likely absent. This will cause stitching edges visible in the result at tile boundaries. To solve this problem, you’d need to fall back on the lower level shear routines. Alas vImage doesn’t provide a way to know how many extra scanlines / columns will be needed, so this is hard for downsampling.