Reputation: 12293
So I use the following code to encode video, it works fine expect the first frame is always distorted (see the screenshot).
I use the following logic to process camera frame:
CVImageBuffer -> CIImage -> MLTexture -> CVPixelBuffer (for pixelBufferAdaptor
)
Why I use the following:
The code directly from captureOutput
function:
guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
print("Failed to get image buffer.")
return
}
CVPixelBufferLockBaseAddress(imageBuffer, .readOnly)
let ciImage = CIImage(cvPixelBuffer: imageBuffer)
let compositedImage = textOverlayCIImage.composited(over: ciImage)
let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(
pixelFormat: .bgra8Unorm,
width: Int(frameSize.width),
height: Int(frameSize.height),
mipmapped: false
)
textureDescriptor.usage = [.shaderWrite, .shaderRead, .renderTarget]
// the texture can be reused for better perfomance
// create it once before starting the recording
let texture = metalDevice.makeTexture(descriptor: textureDescriptor)!
// the commandQueue also can be reused
// create it once before starting the recording / camera
let commandQueue = metalDevice.makeCommandQueue()!
// the commandBuffer has to be created all the time for each new camera frame
let commandBuffer = commandQueue.makeCommandBuffer()!
ciContext.render(
compositedImage,
to: texture,
commandBuffer: commandBuffer,
bounds: CGRect(x: 0, y: 0, width: compositedImage.extent.width, height: compositedImage.extent.height),
colorSpace: CGColorSpaceCreateDeviceRGB()
)
commandBuffer.commit()
commandBuffer.waitUntilCompleted()
CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
if let pixelBufferBaseAddress = CVPixelBufferGetBaseAddress(pixelBuffer) {
// Copy texture data into the pixel buffer's base address
texture.getBytes(
pixelBufferBaseAddress,
bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
from: MTLRegionMake2D(0, 0, texture.width, texture.height),
mipmapLevel: 0
)
}
CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)
CVPixelBufferUnlockBaseAddress(imageBuffer, .readOnly)
if videoWriterInput.isReadyForMoreMediaData {
pixelBufferAdaptor?.append(pixelBuffer, withPresentationTime: CMSampleBufferGetPresentationTimeStamp(sampleBuffer))
}
First frame from video player:
The same frame with video player controls (to see that the time/seekbar is 0/0%):
All other frames in the same video are fine (seekbar is more than 0%):
If I remove MLTexture from these steps and use the following steps instead
CVImageBuffer -> CIImage -> CVPixelBuffer
then the first frame is never distorted, so it's something related to Metal, but I have no idea what exactly, all the code is synced
// no issue when processing camera frame
// and encoding video in the following way
// but it has worse performance
let ciImage = CIImage(cvPixelBuffer: imageBuffer)
let compositedImage = textOverlayCIImage.composited(over: ciImage)
// it takes about ~30 ms for 4K resolution on base iPhone 11
ciContext.render(compositedImage, to: pixelBuffer)
if videoWriterInput.isReadyForMoreMediaData {
pixelBufferAdaptor?.append(pixelBuffer, withPresentationTime: presentationTime)
}
What can be an issue here?
Upvotes: 1
Views: 40