CODEWITHSUNDEEP
iosobjective-carkitcoreml
gi097
gi097

Reputation: 7721

Set VNImageOptionCameraIntrinsics from ARCamera

I am building an application which combines ARKit with CoreML. I am passing frames to the VNImageRequestHandler using the following lines:

// the frame of the current Scene
CVPixelBufferRef pixelBuffer = _cameraPreview.session.currentFrame.capturedImage;

NSMutableDictionary<VNImageOption, id> *requestOptions = [NSMutableDictionary dictionary];
VNImageRequestHandler *handler = [[VNImageRequestHandler alloc] initWithCVPixelBuffer:pixelBuffer options:requestOptions];

Note the requestOptions. It should contain the VNImageOptionCameraIntrinsics field which passes the camera intrinsics to CoreML.

Before using ARKit I was using a CMSampleBufferRef to get images from the camera. The instrinsics could be retrieved and set using the following:

CFTypeRef cameraIntrinsicData = CMGetAttachment(sampleBuffer, kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, nil);
requestOptions[VNImageOptionCameraIntrinsics] = (__bridge id)(cameraIntrinsicData);

However, I am using an ARFrame now, but I still want to set the correct intrinsics since the pixelBuffer is rotated.

Looking at the docs:

https://developer.apple.com/documentation/vision/vnimageoption?language=objc

https://developer.apple.com/documentation/arkit/arcamera/2875730-intrinsics?language=objc

We can see that ARCamera provides intrinsics as well, however, how do I set this value in the requestOptions properly?

So far it should be something like this:

ARCamera *camera = _cameraPreview.session.currentFrame.camera;
NSMutableDictionary<VNImageOption, id> *requestOptions = [NSMutableDictionary dictionary];
// How to put camera.intrinsics here?
requestOptions[VNImageOptionCameraIntrinsics] = camera.intrinsics;

Upvotes: 2

Views: 1057

Answers (1)

l&#39;L&#39;l
l&#39;L&#39;l

Reputation: 47282

As Giovanni mentioned in the comments, converting UIDeviceOrientation to CGImagePropertyOrientation avoids the need to use VNImageOptionCameraIntrinsics:

Utils.m

+(CGImagePropertyOrientation) getOrientation {
    CGImagePropertyOrientation orientation;
    UIDeviceOrientation deviceOrientation = [[UIDevice currentDevice] orientation];
    switch (deviceOrientation) {
        case UIDeviceOrientationPortrait:
            orientation = kCGImagePropertyOrientationRight;
            break;
        case UIDeviceOrientationPortraitUpsideDown:
            orientation = kCGImagePropertyOrientationLeft;
            break;
        case UIDeviceOrientationLandscapeLeft:
            orientation = kCGImagePropertyOrientationUp;
            break;
        case UIDeviceOrientationLandscapeRight:
            orientation = kCGImagePropertyOrientationDown;
            break;
        default:
            orientation = kCGImagePropertyOrientationRight;
            break;
    }
    return orientation;
}

ViewController.mm

- (void)captureOutput {
    ARFrame *frame = self.cameraPreview.session.currentFrame;
    CVPixelBufferRef pixelBuffer = frame.capturedImage;

    CGImagePropertyOrientation deviceOrientation = [Utils getOrientation];
    NSMutableDictionary<VNImageOption, id> *requestOptions = [NSMutableDictionary dictionary];

    VNImageRequestHandler *handler = [[VNImageRequestHandler alloc] initWithCVPixelBuffer:pixelBuffer orientation:deviceOrientation options:requestOptions];

    [handler performRequests:@[[self request]] error:nil];
}

Upvotes: 2

Related Questions