user924
user924

Reputation: 12293

AVAssetWriter - first frame of video is often distorted when processing it as CVImageBuffer -> CIImage -> MLTexture -> CVPixelBuffer

So I use the following code to encode video, it works fine expect the first frame is always distorted (see the screenshot).

I use the following logic to process camera frame:

CVImageBuffer -> CIImage -> MLTexture -> CVPixelBuffer (for pixelBufferAdaptor)

Why I use the following:

The code directly from captureOutput function:

guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    print("Failed to get image buffer.")
    return
}

CVPixelBufferLockBaseAddress(imageBuffer, .readOnly)

let ciImage = CIImage(cvPixelBuffer: imageBuffer)

let compositedImage = textOverlayCIImage.composited(over: ciImage)

let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(
    pixelFormat: .bgra8Unorm,
    width: Int(frameSize.width),
    height: Int(frameSize.height),
    mipmapped: false
)
textureDescriptor.usage = [.shaderWrite, .shaderRead, .renderTarget]
// the texture can be reused for better perfomance
// create it once before starting the recording
let texture = metalDevice.makeTexture(descriptor: textureDescriptor)! 

// the commandQueue also can be reused
// create it once before starting the recording / camera
let commandQueue = metalDevice.makeCommandQueue()!
// the commandBuffer has to be created all the time for each new camera frame 
let commandBuffer = commandQueue.makeCommandBuffer()!
    
ciContext.render(
    compositedImage,
    to: texture,
    commandBuffer: commandBuffer,
    bounds: CGRect(x: 0, y: 0, width: compositedImage.extent.width, height: compositedImage.extent.height),
    colorSpace: CGColorSpaceCreateDeviceRGB()
)

commandBuffer.commit()
commandBuffer.waitUntilCompleted()

CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
if let pixelBufferBaseAddress = CVPixelBufferGetBaseAddress(pixelBuffer) {
    // Copy texture data into the pixel buffer's base address
    texture.getBytes(
        pixelBufferBaseAddress,
        bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
        from: MTLRegionMake2D(0, 0, texture.width, texture.height),
        mipmapLevel: 0
    )
}
CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)

CVPixelBufferUnlockBaseAddress(imageBuffer, .readOnly)

if videoWriterInput.isReadyForMoreMediaData {
    pixelBufferAdaptor?.append(pixelBuffer, withPresentationTime:  CMSampleBufferGetPresentationTimeStamp(sampleBuffer))
}

First frame from video player:

enter image description here

The same frame with video player controls (to see that the time/seekbar is 0/0%):

enter image description here

All other frames in the same video are fine (seekbar is more than 0%):

enter image description here

If I remove MLTexture from these steps and use the following steps instead

CVImageBuffer -> CIImage -> CVPixelBuffer

then the first frame is never distorted, so it's something related to Metal, but I have no idea what exactly, all the code is synced

// no issue when processing camera frame 
// and encoding video in the following way
// but it has worse performance

let ciImage = CIImage(cvPixelBuffer: imageBuffer)

let compositedImage = textOverlayCIImage.composited(over: ciImage)

// it takes about ~30 ms for 4K resolution on base iPhone 11
ciContext.render(compositedImage, to: pixelBuffer)

if videoWriterInput.isReadyForMoreMediaData {
    pixelBufferAdaptor?.append(pixelBuffer, withPresentationTime: presentationTime)
}

What can be an issue here?

Upvotes: 1

Views: 40

Answers (0)

Related Questions