CODEWITHSUNDEEP
ioscoremlcore-videocvpixelbuffer
Nikita Ermolenko
Nikita Ermolenko

Reputation: 2259

Fit CVPixelBuffer into a square image with resizing and aspect ratio saving

I have an image that should be preprocessed before passing to CoreML to 640x640 square with resizing and saving aspect ratio (check the image). I found a lot of helpful links about resizing using vImageScale_*, but haven't found anything similar to adding coloured paddings to the resized image.

I know that Vision has the scaleFit option, but the final output is a bit different, so I'm trying to make image centered.

enter image description here

Upvotes: 2

Views: 724

Answers (2)

Ian Ollmann
Ian Ollmann

Reputation: 1592

The colored padding is just going to be something like vImageBufferFill_ARGB8888, and the middle bits will be vImageScale_ARGB8888. Since the scale operation works with integer heights, widths and origin, you won't need to worry about antialiasing the content at the seams between regions.

Note that this is also relatively easy to do with CGBitmapContextRef and CGContextDrawImage(CGContextRef, CGRect, CGImageRef), though CG just uses a Lanczos2 resampler. You will have less details to micromanage with image orientation and colorspace conversion, and it allows for fractional pixel placement.

You can also get fractional pixel placement with vImage, but will need to use AffineWarp or the shear functions. For ML, I'd probably not attempt to do too much image manipulation in case the ringing has some odd effect. A simple low pass filter + bilinear interpolation might be something to try. This will be a bit blurrier to the human eye but might avoid feeding artifacts to the training workload. Presumably there is ample literature about ML image preprocessing to guide you.

Upvotes: 0

Flex Monkey
Flex Monkey

Reputation: 3633

The vImageScale_* functions scale to fit the destination, so the aspect ratio will change if the source and destination are different ratios.

vImage provides affine transform operations (with support for a background colour!). Take a look at https://developer.apple.com/documentation/accelerate/applying_geometric_transforms_to_images for more information. The final example, Apply a Complex Affine Transform to a vImage Buffer, does exactly what you need (you just need to remove the rotate step).

Upvotes: 2

Related Questions