CODEWITHSUNDEEP
iosswiftavfoundationcore-imagemetal
Breno Rodrigues
Breno Rodrigues

Reputation: 115

Converting MTLTexture to CVPixelBuffer

I am currently working on live filters using Metal. After defining my CIImage I render the image to a MTLTexture.

Below is my rendering code. context is a CIContext backed by Metal; targetTexture is the alias to the texture attached to the currentDrawable property of my MTKView instance:

context?.render(drawImage, to: targetTexture, commandBuffer: commandBuffer, bounds: targetRect, colorSpace: colorSpace)

It renders correctly as I can see the image being displayed on the metal view.

The problem is that after rendering the image (and displaying it), I want to extract the CVPixelBuffer and save it to disk using the class AVAssetWriter.

Another alternative would be to have two rendering steps, one rendering to the texture and another rendering to a CVPixelBuffer. (But it isn't clear how to create such buffer, or the impact that two rendering steps would have in the framerate)

Any help will be appreciated, Thanks!

Upvotes: 7

Views: 6304

Answers (3)

NoemiDev
NoemiDev

Reputation: 1

The top answer won't work for the textures not created using metal buffer. In this case texture.buffer is nil. In my case I had the flow CVPixelBuffer -> MTLTexture -> Process the texture -> CVPixelBuffer.

A working snippet I found here:

Not sure about the performance though and if there's any better way to do it.

func makePixelBuffer(from texture: MTLTexture) -> CVPixelBuffer? {
    var pixelBuffer: CVPixelBuffer?

    CVPixelBufferCreate(
        kCFAllocatorDefault,
        texture.width,
        texture.height,
        kCVPixelFormatType_32BGRA,
        nil,
        &pixelBuffer
    )

    guard let pixelBuffer else {
        return nil
    }

    CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))

    guard let pixelBufferBytes = CVPixelBufferGetBaseAddress(pixelBuffer) else {
        return nil
    }
    let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)
    let region = MTLRegionMake2D(0, 0, texture.width, texture.height)

    texture.getBytes(pixelBufferBytes, bytesPerRow: bytesPerRow, from: region, mipmapLevel: 0)
    CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))

    return pixelBuffer
}

Upvotes: 0

Jonathan Wang
Jonathan Wang

Reputation: 1

+ (void)getPixelBufferFromBGRAMTLTexture:(id<MTLTexture>)texture result:(void(^)(CVPixelBufferRef pixelBuffer))block {
    
    CVPixelBufferRef pxbuffer = NULL;
    NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
                             nil];
    
    size_t imageByteCount = texture.width * texture.height * 4;
    void *imageBytes = malloc(imageByteCount);
    NSUInteger bytesPerRow = texture.width * 4;
    MTLRegion region = MTLRegionMake2D(0, 0, texture.width, texture.height);
    [texture getBytes:imageBytes bytesPerRow:bytesPerRow fromRegion:region mipmapLevel:0];
    
    CVPixelBufferCreateWithBytes(kCFAllocatorDefault,texture.width,texture.height,kCVPixelFormatType_32BGRA,imageBytes,bytesPerRow,NULL,NULL,(__bridge CFDictionaryRef)options,&pxbuffer);
    
    if (block) {
        block(pxbuffer);
    }
    CVPixelBufferRelease(pxbuffer);
    free(imageBytes);
}

Upvotes: -1

Abs
Abs

Reputation: 41

You can try to copy the raw data from the MTLTexture like this :

var outPixelbuffer: CVPixelBuffer?
if let datas = targetTexture.texture.buffer?.contents() {
    CVPixelBufferCreateWithBytes(kCFAllocatorDefault, targetTexture.width, 
    targetTexture.height, kCVPixelFormatType_64RGBAHalf, datas, 
    targetTexture.texture.bufferBytesPerRow, nil, nil, nil, &outPixelbuffer);
}

Upvotes: 4

Related Questions