How do you extend the space (bounds) of a CIImage without stretching the original?

Question

I'm applying several filters on an already cropped image, and I'd like a flipped duplicate of it next to the original. This would make it twice as wide.

Problem: How do you extend the bounds so both can fit? .cropped(to:CGRect) will stretch whatever original content was there. The reason there is existing content is because I'm trying to use applyingFilter as much as possible to save on processing. It's also why I'm cropping the original un-mirrored image.

Below is my CIImage "alphaMaskBlend2" with a compositing filter, and a transform applied to the same image that flips it and adjusts its position. sourceCore.extent is the size I want the final image.

    alphaMaskBlend2 = alphaMaskBlend2?.applyingFilter("CISourceAtopCompositing",
                                                      parameters: [kCIInputImageKey: (alphaMaskBlend2?.transformed(by: scaledImageTransform))!,
                                                                   kCIInputBackgroundImageKey: alphaMaskBlend2!]).cropped(to: sourceCore.extent)

I've played around with the position of the transform in LLDB. I found with this filter being cropped, the left most image becomes stretched. If I use clamped to the same extent, and then I re-crop the image to the same extent again, the image is no longer distorted, but the bounds of the image is only half the width that it should be.

The only way I could achieve this, is compositing against a background image (sourceCore) that would be the size of the two images combined, and then compositing the other image:

    alphaMaskBlend2 = alphaMaskBlend2?.applyingFilter("CISourceAtopCompositing",
                                                      parameters: [kCIInputImageKey: alphaMaskBlend2!,
                                                                   kCIInputBackgroundImageKey: sourceCore])

    alphaMaskBlend2 = alphaMaskBlend2?.applyingFilter("CISourceAtopCompositing",
                                                      parameters: [kCIInputImageKey: (alphaMaskBlend2?.cropped(to: cropRect).transformed(by: scaledImageTransform))!,
                                                                   kCIInputBackgroundImageKey: alphaMaskBlend2!])

Problem is, that this is more expensive than necessary. I even tested it with benchmarking. It would make a lot more sense if I could do this with one composite.

Answer 1

While I can "flip" a CIImage I couldn't find a way to use an existing CIFilter to "stitch" it along side the original. However, with some basic knowledge of writing your own CIKernel, you can. A simple project of achieving this is here.

This project contains a sample image, and using CoreImage and a GLKView it:

• flips the image by transposing the Y "bottom/top" coordinates for CIPerspectiveCorrection
• creates a new "palette" image using CIConstantColor and then crops it using CICrop to be twice the width of the original
• uses a very simple CIKernel (registered as "Stitch" to actually stitch it together

Here's the code to flip:

    // use CIPerspectiveCorrection to "flip" on the Y axis

    let minX:CGFloat = 0
    let maxY:CGFloat = 0
    let maxX = originalImage?.extent.width
    let minY = originalImage?.extent.height

    let flipFilter = CIFilter(name: "CIPerspectiveCorrection")
    flipFilter?.setValue(CIVector(x: minX, y: maxY), forKey: "inputTopLeft")
    flipFilter?.setValue(CIVector(x: maxX!, y: maxY), forKey: "inputTopRight")
    flipFilter?.setValue(CIVector(x: minX, y: minY!), forKey: "inputBottomLeft")
    flipFilter?.setValue(CIVector(x: maxX!, y: minY!), forKey: "inputBottomRight")
    flipFilter?.setValue(originalImage, forKey: "inputImage")
    flippedImage = flipFilter?.outputImage

Here's the code to create the palette:

    let paletteFilter = CIFilter(name: "CIConstantColorGenerator")
    paletteFilter?.setValue(CIColor(red: 0.7, green: 0.4, blue: 0.4), forKey: "inputColor")
    paletteImage = paletteFilter?.outputImage
    let cropFilter = CIFilter(name: "CICrop")
    cropFilter?.setValue(paletteImage, forKey: "inputImage")
    cropFilter?.setValue(CIVector(x: 0, y: 0, z: (originalImage?.extent.width)! * 2, w: (originalImage?.extent.height)!), forKey: "inputRectangle")
    paletteImage = cropFilter?.outputImage

Here's the code to register and use the custom CIFilter:

    // register and use stitch filer

    StitchedFilters.registerFilters()
    let stitchFilter = CIFilter(name: "Stitch")
    stitchFilter?.setValue(originalImage?.extent.width, forKey: "inputThreshold")
    stitchFilter?.setValue(paletteImage, forKey: "inputPalette")
    stitchFilter?.setValue(originalImage, forKey: "inputOriginal")
    stitchFilter?.setValue(flippedImage, forKey: "inputFlipped")
    finalImage = stitchFilter?.outputImage

All of this code (long with layout constraints) in the demo project is in viewDidLoad, so please, place it where it belongs!

Here's the code to (a) create a CIFilter subclass called Stitch and (b) register it so you can use it like any other filter:

func openKernelFile(_ name:String) -> String {
    let filePath = Bundle.main.path(forResource: name, ofType: ".cikernel")
    do {
        return try String(contentsOfFile: filePath!)
    }
    catch let error as NSError {
        return error.description
    }
}

let CategoryStitched = "Stitch"

class StitchedFilters: NSObject, CIFilterConstructor {
    static func registerFilters() {
        CIFilter.registerName(
            "Stitch",
            constructor: StitchedFilters(),
            classAttributes: [
                kCIAttributeFilterCategories: [CategoryStitched]
            ])
    }
    func filter(withName name: String) -> CIFilter? {
        switch name {
        case "Stitch":
            return Stitch()
        default:
            return nil
        }
    }
}

class Stitch:CIFilter {

    let kernel = CIKernel(source: openKernelFile("Stitch"))
    var inputThreshold:Float  = 0
    var inputPalette: CIImage!
    var inputOriginal: CIImage!
    var inputFlipped: CIImage!

    override var attributes: [String : Any] {
        return [
            kCIAttributeFilterDisplayName: "Stitch",

            "inputThreshold": [kCIAttributeIdentity: 0,
                               kCIAttributeClass: "NSNumber",
                               kCIAttributeDisplayName: "Threshold",
                               kCIAttributeDefault: 0.5,
                               kCIAttributeMin: 0,
                               kCIAttributeSliderMin: 0,
                               kCIAttributeSliderMax: 1,
                               kCIAttributeType: kCIAttributeTypeScalar],

            "inputPalette": [kCIAttributeIdentity: 0,
                             kCIAttributeClass: "CIImage",
                             kCIAttributeDisplayName: "Palette",
                             kCIAttributeType: kCIAttributeTypeImage],

            "inputOriginal": [kCIAttributeIdentity: 0,
                              kCIAttributeClass: "CIImage",
                              kCIAttributeDisplayName: "Original",
                              kCIAttributeType: kCIAttributeTypeImage],

            "inputFlipped": [kCIAttributeIdentity: 0,
                             kCIAttributeClass: "CIImage",
                             kCIAttributeDisplayName: "Flipped",
                             kCIAttributeType: kCIAttributeTypeImage]
        ]
    }
    override init() {
        super.init()
    }
    override func setValue(_ value: Any?, forKey key: String) {
        switch key {
        case "inputThreshold":
            inputThreshold = value as! Float
        case "inputPalette":
            inputPalette = value as! CIImage
        case "inputOriginal":
            inputOriginal = value as! CIImage
        case "inputFlipped":
            inputFlipped = value as! CIImage
        default:
            break
        }
    }
    @available(*, unavailable) required init?(coder aDecoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
    override var  outputImage: CIImage {
        return kernel!.apply(
            extent: inputPalette.extent,
            roiCallback: {(index, rect) in return rect},
            arguments: [
                inputThreshold as Any,
                inputPalette as Any,
                inputOriginal as Any,
                inputFlipped as Any
            ])!
    }
}

Finally, the CIKernel code:

kernel vec4 stitch(float threshold, sampler palette, sampler original, sampler flipped) {
    vec2 coord = destCoord();
    if (coord.x < threshold) {
        return sample(original, samplerCoord(original));
    } else {
        vec2 flippedCoord = coord - vec2(threshold, 0.0);
        vec2 flippedCoordinate = samplerTransform(flipped, flippedCoord);
        return sample(flipped, flippedCoordinate);
    }
}

Now, someone else may have something more elegant - maybe even using an existing CIFilter - but this works well. It only uses the GPU, so performance-wise, can be used in "real time". I added unneeded code (registering the filter, using a dictionary to define attributes) to make it more of a teaching exercise for those new to creating CIKernels that anyone with knowledge of using CIFilters can consume. If you focus on the kernel code, you'll recognize how similar to C it looks.

Last, a caveat. I am only stitching the (Y-axis) flipped image to the right of the original. You'll need to adjust things if you want something else.