Pulling data from a CMSampleBuffer in order to create a deep copy

Question

I am trying to create a copy of a CMSampleBuffer as returned by captureOutput in a AVCaptureVideoDataOutputSampleBufferDelegate.

Since the CMSampleBuffers come from a preallocated pool of (15) buffers, if I attach a reference to them they cannot be recollected. This causes all remaining frames to be dropped.

To maintain optimal performance, some sample buffers directly reference pools of memory that may need to be reused by the device system and other capture inputs. This is frequently the case for uncompressed device native capture where memory blocks are copied as little as possible. If multiple sample buffers reference such pools of memory for too long, inputs will no longer be able to copy new samples into memory and those samples will be dropped.

If your application is causing samples to be dropped by retaining the provided CMSampleBufferRef objects for too long, but it needs access to the sample data for a long period of time, consider copying the data into a new buffer and then releasing the sample buffer (if it was previously retained) so that the memory it references can be reused.

Obviously I must copy the CMSampleBuffer but CMSampleBufferCreateCopy() will only create a shallow copy. Thus I conclude that I must use CMSampleBufferCreate(). I filled in the 12! parameters that the constructor needs but ran into the problem that my CMSampleBuffers do not contain a blockBuffer (not entirely sure what that is but it seems important).

This question has been asked several times but not answered.

Deep Copy of CMImageBuffer or CVImageBuffer and Create a copy of CMSampleBuffer in Swift 2.0

One possible answer is "I finally figured out how to use this to create a deep clone. All the copy methods reused the data in the heap which kept would lock the AVCaptureSession. So I had to pull the data out into a NSMutableData object and then created a new sample buffer." credit to Rob on SO. However, I do not know how to do this correcly.

If you are interested, this is the output of print(sampleBuffer). There is no mention of blockBuffer, aka CMSampleBufferGetDataBuffer returns nil. There is a imageBuffer, but creating a "copy" using CMSampleBufferCreateForImageBuffer does not seem to free the CMSampleBuffer either.

---

EDIT: Since this question has been posted I have been trying even more ways of copying the memory.

I did the same thing that user Kametrixom tried. This is my attempt at the same idea, to first copy the CVPixelBuffer then use CMSampleBufferCreateForImageBuffer to create the final sample buffer. However this results in one of two error:

• A EXC_BAD_ACCESS on the memcpy instruction. AKA a segfault from trying to access outside of the application's memory.
• Or, the memory will copy successfully but the CMSampleBufferCreateReadyWithImageBuffer() will fail with result code -12743 which "Indicates that the format of the given media does not match the given format description. For example, a format description paired with a CVImageBuffer that fails CMVideoFormatDescriptionMatchesImageBuffer."

You can see that both Kametrixom and I did use CMSampleBufferGetFormatDescription(sampleBuffer) to try to copy the source buffer's format description. Thus, I'm not sure why the format of the given media does not match the given format description.

Answer 1

I thank @northern-captain for providing me the intuition to handle sample buffer in autoreleasepool { }

I extend the solution provided by https://stackoverflow.com/users/536664/northern-captain

In his solution, I had faced error kCMSampleBufferError_InvalidMediaFormat in line,

guard let bufOut = copiedSampleBuffer else { fatalError("CMSampleBuffer copy: CreateReady \(status)") }

It denotes, copiedPixelBuffer's format Description does not match with format Description of let format = CMSampleBufferGetFormatDescription(self)!.

So, I created PixelBuffer with

    var _copy : CVPixelBuffer?
CVPixelBufferCreate(
        nil,
        CVPixelBufferGetWidth(self),
        CVPixelBufferGetHeight(self),
        CVPixelBufferGetPixelFormatType(self),
        CVPixelBufferCopyCreationAttributes(self),
        &_copy)

Which, resolved the issue.

FurtherMore, Apple Doc says,

Because CVImageBuffers hold visual data, the format description provided is a CMVideoFormatDescription. The format description must be consistent with the attributes and formatting information attached to the CVImageBuffer. The width, height, and codecType must match (for CVPixelBuffers the codec type is given by CVPixelBufferGetPixelFormatType(pixelBuffer); for other CVImageBuffers, the codecType must be 0). The format description extensions must match the image buffer attachments for all the keys in the list returned by CMVideoFormatDescriptionGetExtensionKeysCommonWithImageBuffers (if absent in either they must be absent in both).

So, I added attachements as per the document,

         //Getting and Setting Attachment one by one .
    let commonKeys = NSSet(array: CMVideoFormatDescriptionGetExtensionKeysCommonWithImageBuffers() as! [Any])

    
    let propagatedAttachments = NSDictionary(dictionary: CVBufferCopyAttachments(self, .shouldPropagate)!)
    propagatedAttachments.enumerateKeysAndObjects { key, obj, stop in
        if commonKeys.contains(key) {
            CVBufferSetAttachment(copy, key as! CFString, obj as AnyObject, .shouldPropagate)
        }
    }

    let nonPropagatedAttachments = NSDictionary(dictionary: CVBufferCopyAttachments(self, .shouldNotPropagate)!)
    nonPropagatedAttachments.enumerateKeysAndObjects { key, obj, stop in
        if commonKeys.contains(key) {
            CVBufferSetAttachment(copy, key as! CFString, obj as AnyObject, .shouldNotPropagate)
        }
    }

Entire code,

extension CMSampleBuffer {
    
    func deepCopy() -> CMSampleBuffer {
        guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else { fatalError("CMSampleBuffer copy: get image buffer")  }

        //print("Original Pixel Buffer: \(pixelBuffer)")
        let copiedPixelBuffer = pixelBuffer.deepCopy()
        //print("Copied Pixel Buffer: \(copiedPixelBuffer)")

        let formatDesc = CMSampleBufferGetFormatDescription(self)!
        //print("Format Description of CMSampleBuffer: \(formatDesc)")
        //let pixelFormatDesc = try! CMVideoFormatDescription(imageBuffer: copiedPixelBuffer)
        //print("Pixel Format Description of CMSampleBuffer: \(pixelFormatDesc)")
        
        var timing = CMSampleTimingInfo()
        CMSampleBufferGetSampleTimingInfo(self, at: 0, timingInfoOut: &timing)

        var copiedSampleBuffer : CMSampleBuffer?

        let status = CMSampleBufferCreateReadyWithImageBuffer(allocator: nil,
                imageBuffer: copiedPixelBuffer,
                formatDescription: formatDesc,
                sampleTiming: &timing,
                sampleBufferOut: &copiedSampleBuffer)
        guard let bufOut = copiedSampleBuffer else { fatalError("CMSampleBuffer copy: CreateReady \(status)") }
        return bufOut 
    }
}

extension CVPixelBuffer {

    func deepCopy() -> CVPixelBuffer {
        precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

        
        var _copy : CVPixelBuffer?
        CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            CVPixelBufferCopyCreationAttributes(self),
            &_copy)
        

        guard let copy = _copy else {
            fatalError("Failed to create a copy of CVPixelBuffer")
        }
        //        CVBufferPropagateAttachments(self, copy)

        CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags.readOnly)
        CVPixelBufferLockBaseAddress(copy, [])

        for plane in 0..<CVPixelBufferGetPlaneCount(self) {
            let dest = CVPixelBufferGetBaseAddressOfPlane(copy, plane)
            let source = CVPixelBufferGetBaseAddressOfPlane(self, plane)
            let height = CVPixelBufferGetHeightOfPlane(self, plane)
            let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(self, plane)

            memcpy(dest, source, height * bytesPerRow)
        }

        CVPixelBufferUnlockBaseAddress(copy, [])
        CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags.readOnly)
        
        /*
         //Getting and Setting all Attachments at once.
        if let attachments = CVBufferCopyAttachments(self, .shouldPropagate) as? [String: Any]{
            CVBufferSetAttachments(copy, attachments as CFDictionary, .shouldPropagate)
        }
        if let attachments = CVBufferCopyAttachments(self, .shouldNotPropagate) as? [String: Any]{
            CVBufferSetAttachments(copy, attachments as CFDictionary, .shouldNotPropagate)
        }
        */
        
         //Getting and Setting Attachment one by one .
        let commonKeys = NSSet(array: CMVideoFormatDescriptionGetExtensionKeysCommonWithImageBuffers() as! [Any])

        
        let propagatedAttachments = NSDictionary(dictionary: CVBufferCopyAttachments(self, .shouldPropagate)!)
        propagatedAttachments.enumerateKeysAndObjects { key, obj, stop in
            if commonKeys.contains(key) {
                CVBufferSetAttachment(copy, key as! CFString, obj as AnyObject, .shouldPropagate)
            }
        }

        let nonPropagatedAttachments = NSDictionary(dictionary: CVBufferCopyAttachments(self, .shouldNotPropagate)!)
        nonPropagatedAttachments.enumerateKeysAndObjects { key, obj, stop in
            if commonKeys.contains(key) {
                CVBufferSetAttachment(copy, key as! CFString, obj as AnyObject, .shouldNotPropagate)
            }
        }
        return copy
    }
    
}

Answer 2

Spent some time trying to pull this together. I needed a function that should be able to create a CMSampleBuffer deep copy with CVPixelBuffer inside, not just the pixel buffer copy.

Here is what I came up with and it works for me in iOS 15, Swift 5:

extension CVPixelBuffer {
func copy() -> CVPixelBuffer {
    precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

    var _copy : CVPixelBuffer?
    CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            nil,
            &_copy)
    guard let copy = _copy else { fatalError() }

    CVBufferPropagateAttachments(self, copy)


    CVPixelBufferLockBaseAddress(self, .readOnly)
    CVPixelBufferLockBaseAddress(copy, CVPixelBufferLockFlags())

    for plane in 0 ..< CVPixelBufferGetPlaneCount(self) {
        let dest = CVPixelBufferGetBaseAddressOfPlane(copy, plane)
        let source = CVPixelBufferGetBaseAddressOfPlane(self, plane)
        let height = CVPixelBufferGetHeightOfPlane(self, plane)
        let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(self, plane)

        memcpy(dest, source, height * bytesPerRow)
    }

    CVPixelBufferUnlockBaseAddress(copy, CVPixelBufferLockFlags())
    CVPixelBufferUnlockBaseAddress(self, .readOnly)

    return copy
  }
}

Sample buffer extension

extension CMSampleBuffer {
func copy() -> CMSampleBuffer {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else { fatalError("CMSampleBuffer copy: get image buffer")  }

    let copiedPixelBuffer = pixelBuffer.copy()
    let format = CMSampleBufferGetFormatDescription(self)!
    var timing = CMSampleTimingInfo()
    CMSampleBufferGetSampleTimingInfo(self, at: 0, timingInfoOut: &timing)

    var copiedSampleBuffer : CMSampleBuffer?

    let status = CMSampleBufferCreateReadyWithImageBuffer(allocator: nil,
            imageBuffer: copiedPixelBuffer,
            formatDescription: format,
            sampleTiming: &timing,
            sampleBufferOut: &copiedSampleBuffer)
    guard let bufOut = copiedSampleBuffer else { fatalError("CMSampleBuffer copy: CreateReady \(status)") }
    return bufOut
  }
}

And a call to copy:

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    let bufferCopy = sampleBuffer.copy()
    // free to use bufferCopy as it won't stop video recording
    // don't forget to put it into autoreleasepool { } if used in non-main thread
}

Answer 3

I spent a good couple hours trying to get this to work. It turns out both the attachments from the original CVPixelBuffer and the IOSurface options found in PixelBufferMetalCompatibilityKey are necessary.

(FYI, VideoToolbox's PixelTransferSession is macOS-only, sadly.)

Here's what I ended up with. I've left in a few lines at the end that allow you to verify the memcpy by comparing the average colours of both the original and copied CVPixelBuffers. It does slow things down so it should be removed once you're confident your copy() is working as expected. The CIImage.averageColour extension is adapted from this code.

extension CVPixelBuffer {
    func copy() -> CVPixelBuffer {
        precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

        let ioSurfaceProps = [
            "IOSurfaceOpenGLESFBOCompatibility": true as CFBoolean,
            "IOSurfaceOpenGLESTextureCompatibility": true as CFBoolean,
            "IOSurfaceCoreAnimationCompatibility": true as CFBoolean
        ] as CFDictionary

        let options = [
            String(kCVPixelBufferMetalCompatibilityKey): true as CFBoolean,
            String(kCVPixelBufferIOSurfacePropertiesKey): ioSurfaceProps
        ] as CFDictionary

        var _copy : CVPixelBuffer?
        CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            options,
            &_copy)

        guard let copy = _copy else { fatalError() }

        CVBufferPropagateAttachments(self as CVBuffer, copy as CVBuffer)

        CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags.readOnly)
        CVPixelBufferLockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))

        let copyBaseAddress = CVPixelBufferGetBaseAddress(copy)
        let currBaseAddress = CVPixelBufferGetBaseAddress(self)

        memcpy(copyBaseAddress, currBaseAddress, CVPixelBufferGetDataSize(self))

        CVPixelBufferUnlockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))
        CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags.readOnly)

        // let's make sure they have the same average color
//        let originalImage = CIImage(cvPixelBuffer: self)
//        let copiedImage = CIImage(cvPixelBuffer: copy)
//
//        let averageColorOriginal = originalImage.averageColour()
//        let averageColorCopy = copiedImage.averageColour()
//
//        assert(averageColorCopy == averageColorOriginal)
//        debugPrint("average frame color: \(averageColorCopy)")

        return copy
    }
}

Answer 4

I believe that with VideoToolbox.framework, you can use VTPixelTransferSession to copy pixel buffers. In fact it's the only thing that this class do.

Reference: https://developer.apple.com/documentation/videotoolbox/vtpixeltransfersession-7cg

Answer 5

This is the Swift 3 solution to the top rated answer.

extension CVPixelBuffer {
func copy() -> CVPixelBuffer {
    precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

    var _copy : CVPixelBuffer?
    CVPixelBufferCreate(
        kCFAllocatorDefault,
        CVPixelBufferGetWidth(self),
        CVPixelBufferGetHeight(self),
        CVPixelBufferGetPixelFormatType(self),
        nil,
        &_copy)

    guard let copy = _copy else { fatalError() }

    CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags.readOnly)
    CVPixelBufferLockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))


    let copyBaseAddress = CVPixelBufferGetBaseAddress(copy)
    let currBaseAddress = CVPixelBufferGetBaseAddress(self)

    memcpy(copyBaseAddress, currBaseAddress, CVPixelBufferGetDataSize(self))

    CVPixelBufferUnlockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))
    CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags.readOnly)


    return copy
}
}

Answer 6

Alright, I think I finally got it. I created a helper extension to make a full copy of a CVPixelBuffer:

extension CVPixelBuffer {
    func copy() -> CVPixelBuffer {
        precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

        var _copy : CVPixelBuffer?
        CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            CVBufferGetAttachments(self, kCVAttachmentMode_ShouldPropagate)?.takeUnretainedValue(),
            &_copy)

        guard let copy = _copy else { fatalError() }

        CVPixelBufferLockBaseAddress(self, kCVPixelBufferLock_ReadOnly)
        CVPixelBufferLockBaseAddress(copy, 0)

        for plane in 0..<CVPixelBufferGetPlaneCount(self) {
            let dest = CVPixelBufferGetBaseAddressOfPlane(copy, plane)
            let source = CVPixelBufferGetBaseAddressOfPlane(self, plane)
            let height = CVPixelBufferGetHeightOfPlane(self, plane)
            let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(self, plane)

            memcpy(dest, source, height * bytesPerRow)
        }

        CVPixelBufferUnlockBaseAddress(copy, 0)
        CVPixelBufferUnlockBaseAddress(self, kCVPixelBufferLock_ReadOnly)

        return copy
    }
}

Now you can use this in your didOutputSampleBuffer method:

guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

let copy = pixelBuffer.copy()

toProcess.append(copy)

But be aware, one such pixelBuffer takes up about 3MB of memory (1080p), which means that in 100 frames you got already about 300MB, which is about the point at which the iPhone says STAHP (and crashes).

Note that you don't actually want to copy the CMSampleBuffer since it only really contains a CVPixelBuffer because it's an image.