Downsize very large image to MTLTexture

Question

I've followed this example from Apple to downsize a very large image I'm downloading from a remote location. I've rewritten the code in Swift. It apparently works, but when I call MTKTextureLoader.newTexture, the application crashes with a EXC_BAD_ACCESS in _loadCGImage. There's no other hint, but I suspect the image data has been release already or something...

Any hints why would it crash without any proper error message?

This is the top level code,

// this is an extension of MTKTextureLoader
// [...]
if let uiImage = UIImage(contentsOfFile: cachedFileURL.path) {
    let maxDim : CGFloat = 8192
    if uiImage.size.width > maxDim || uiImage.size.height > maxDim {
        let scale = uiImage.size.width > maxDim ? maxDim / uiImage.size.width : maxDim / uiImage.size.height
        if let cgImage = MTKTextureLoader.downsize(image: uiImage, scale: scale) {
            return self.newTexture(with: cgImage, options: options, completionHandler: completionHandler)
        } else {
            anError = TextureError.CouldNotDownsample
        }
    } else {
        return self.newTexture(withContentsOf: cachedFileURL, options: options, completionHandler: completionHandler)
    }
}

And this is the downsize method,

private static func downsize(image: UIImage, scale: CGFloat) -> CGImage? {
    let destResolution = CGSize(width: Int(image.size.width * scale), height: Int(image.size.height * scale))
    let kSourceImageTileSizeMB : CGFloat = 40.0 // The tile size will be (x)MB of uncompressed image data
    let pixelsPerMB = 262144
    let tileTotalPixels = kSourceImageTileSizeMB * CGFloat(pixelsPerMB)
    let destSeemOverlap : CGFloat = 2.0 // the numbers of pixels to overlap the seems where tiles meet.
    let colorSpace = CGColorSpaceCreateDeviceRGB()
    let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedLast.rawValue)
    guard let destContext = CGContext(data: nil, width: Int(destResolution.width), height: Int(destResolution.height), bitsPerComponent: 8, bytesPerRow: 0, space: colorSpace, bitmapInfo: bitmapInfo.rawValue) else {
        NSLog("failed to create the output bitmap context!")
        return nil
    }
    var sourceTile = CGRect()
    sourceTile.size.width = image.size.width
    sourceTile.size.height = floor( tileTotalPixels / sourceTile.size.width )
    print("source tile size: \(sourceTile.size)")
    sourceTile.origin.x = 0.0
    // the output tile is the same proportions as the input tile, but
    // scaled to image scale.
    var destTile = CGRect()
    destTile.size.width = destResolution.width
    destTile.size.height = sourceTile.size.height * scale
    destTile.origin.x = 0.0
    print("dest tile size: \(destTile.size)")
    // the source seem overlap is proportionate to the destination seem overlap.
    // this is the amount of pixels to overlap each tile as we assemble the output image.
    let sourceSeemOverlap : CGFloat = floor( ( destSeemOverlap / destResolution.height ) * image.size.height )
    print("dest seem overlap: \(destSeemOverlap), source seem overlap: \(sourceSeemOverlap)")
    // calculate the number of read/write operations required to assemble the
    // output image.
    var iterations = Int( image.size.height / sourceTile.size.height )
    // if tile height doesn't divide the image height evenly, add another iteration
    // to account for the remaining pixels.
    let remainder = Int(image.size.height) % Int(sourceTile.size.height)
    if remainder > 0 {
        iterations += 1
    }
    // add seem overlaps to the tiles, but save the original tile height for y coordinate calculations.
    let sourceTileHeightMinusOverlap = sourceTile.size.height
    sourceTile.size.height += sourceSeemOverlap
    destTile.size.height += destSeemOverlap
    print("beginning downsize. iterations: \(iterations), tile height: \(sourceTile.size.height), remainder height: \(remainder)")
    for y in 0.. 0 {
                    var dify = destTile.size.height
                    destTile.size.height = CGFloat( sourceTileImage.height ) * scale
                    dify -= destTile.size.height
                    destTile.origin.y += dify
                }
                // read and write a tile sized portion of pixels from the input image to the output image.
                destContext.draw(sourceTileImage, in: destTile, byTiling: false)
            }
            // !!! In the original LargeImageDownsizing code, it released the source image here !!!
            // [image release];
            // !!! I assume I don't need to do that in Swift?? !!!
            /* while CGImageCreateWithImageInRect lazily loads just the image data defined by the argument rect,
             that data is finally decoded from disk to mem when CGContextDrawImage is called. sourceTileImageRef
             maintains internally a reference to the original image, and that original image both, houses and
             caches that portion of decoded mem. Thus the following call to release the source image. */
            // http://en.swifter.tips/autoreleasepool/
            // drain will be called
            // to free all objects that were sent -autorelease within the scope of this loop.
        }
        // !!! Original code reallocated the image here !!!
        // we reallocate the source image after the pool is drained since UIImage -imageNamed
        // returns us an autoreleased object.
    }
    print("downsize complete.")
    // create a CGImage from the offscreen image context
    return destContext.makeImage()
}

Edit:

It crashes every time I try to initialize the MTLTexture with a CGImage, even if the image is small and fits in memory. So it seems unrelated to the resizing... This code also crashes,

func newTexture(with uiImage: UIImage, options: [String : NSObject]? = nil, completionHandler: @escaping MTKTextureLoaderCallback) {
    if let cgImage = uiImage.cgImage {
        return self.newTexture(with: cgImage, options: options, completionHandler: completionHandler)
    } else {
        completionHandler(nil, TextureError.CouldNotBeCreated)
    }
}

at ImageIO copyImageBlockSetWithOptions.

Edit2: Workaround based on warrenm's answer: make the call to newTexture(with: cgImage) sync instead of async. For instance, the function above becomes,

func newTexture(with uiImage: UIImage, options: [String : NSObject]? = nil, completionHandler: MTKTextureLoaderCallback) {
    if let cgImage = uiImage.cgImage {
        let tex = try? self.newTexture(with: cgImage, options: options)
        completionHandler(tex, tex == nil ? TextureError.CouldNotBeCreated : nil)
    } else {
        completionHandler(nil, TextureError.CouldNotGetCGImage)
    }
}

(Note that I've removed the @escaping since now I actually call the sync method.)

The downsize code was correct. It works now with this workaround :)

warrenm · Accepted Answer

This appears to be an issue with the lifetime of the CGImage used to create the texture, and it may be a bug in MetalKit.

Essentially, the CGImage returned by the context is only guaranteed to live until the end of the scope of the autorelease pool in which it is created. When you call the asynchronous newTexture method, MTKTextureLoader moves the work of creating the texture onto a background thread, and operates on the CGImage outside of the scope of its enclosing autorelease pool, after its backing store has already been deallocated.

You can work around this by either capturing the image in the completion handler (which will cause ARC to extend its lifetime), or using the corresponding synchronous texture creation method, newTexture(with:options:), which will remain in the relevant scope until the texture is fully initialized.

Downsize very large image to MTLTexture

Answers (1)

Related Questions