CODEWITHSUNDEEP
swiftmachine-learningaugmented-realityarkitapple-vision
Andy Jazz
Andy Jazz

Reputation: 58563

CVPixelBuffer – How to capture every THIRD frame out of 60 fps?

I need only 20 frames out of 60 frames per second for processing (CVPixelBuffer).

How to capture every third ARFrame in ARKit session? I need approximately 20 fps for capture (I understand there may be a drop frame).

Here's a code's snippet:

func updateCoreML() {

    let pixelBuffer: CVPixelBuffer? = (sceneView.session.currentFrame?.capturedImage)

    if pixelBuffer == nil { return }
    let ciImage = CIImage(cvPixelBuffer: pixelBuffer!)
    let imageRequestHandler = VNImageRequestHandler(ciImage: ciImage, options: [:])

    do {
        try imageRequestHandler.perform(self.visionRequests)
    } catch {
        print(error)
    }
}

Upvotes: 1

Views: 963

Answers (2)

Tahir Iqbal
Tahir Iqbal

Reputation: 1

You can either set custom frame rate, so you will get only 20 frames, or you can receive all frames but use only every third frame by taking a flag variable and incrementing it.

  func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
       totalFrameCount += 1
       if totalFrameCount % 3 != 0{ return }
}

or you can set custom frame rate by extending AVCaptureDevice

 lockForConfiguration()
 activeVideoMinFrameDuration = CMTimeMake(1, 20)
 activeVideoMaxFrameDuration = CMTimeMake(1, 20)
 unlockForConfiguration()

Upvotes: 0

Maxim Volgin
Maxim Volgin

Reputation: 4077

The simplest way known to me is to use RxARKit and apply .throttle() operator to session.rx.didUpdateFrame. I believe it is not prudent to skip every so many events because frame rate is not guaranteed to be 60fps at all times, so it is best to use .throttle() to get a frame at most every so many milliseconds, no matter what the actual framerate is. You can plug the result of it into RxVision, which will take care that that very frame will be used by CoreML.

import RxSwift
import RxARKit
import RxVision


let disposeBag = DisposeBag()
let mlRequest: RxVNCoreMLRequest<CVPixelBuffer> = VNCoreMLRequest.rx.request(model: model, imageCropAndScaleOption: .scaleFit)

mlRequest
    .observable
    .subscribe { [unowned self] (event) in
        switch event {
            case .next(let completion):       
                let cvPixelBuffer = completion.value
                if let result = completion.request.results?[0] as? VNClassificationObservation {
                    os_log("results: %@", type: .debug, result.identifier)
                }
            default:
                break
        }
    }
    .disposed(by: disposeBag)

session
    .rx
    .didUpdateFrame
    .throttle(1/20)
    .subscribe { event in
        switch event {
        case .next(let didUpdateFrame):
        let frame: ARFrame = didUpdateFrame.frame        
        let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage, orientation: .up, options: requestOptions)
        do {
            try imageRequestHandler.rx.perform([mlRequest], with: frame.capturedImage) } catch {
            print(error)
        }            
        break
        default:
            break
        }
    }
    .disposed(by: disposeBag)

Upvotes: 1

Related Questions