AVCaptureSession front camera orientation

Question

I'm using AVCaptureSession to record video, using AVAssetWriterInput I'm writing the video to a file.
My problem is with the video orientation, using apple RosyWriter example, I can transform the AVAssetWriterInput so the video will be at the right orientation.

- (CGFloat)angleOffsetFromPortraitOrientationToOrientation:(AVCaptureVideoOrientation)orientation
{
    CGFloat angle = 0.0;

    switch (orientation) {
        case AVCaptureVideoOrientationPortrait:
            angle = 0.0;
            break;
        case AVCaptureVideoOrientationPortraitUpsideDown:
            angle = M_PI;
            break;
        case AVCaptureVideoOrientationLandscapeRight:
            angle = -M_PI_2;
            break;
        case AVCaptureVideoOrientationLandscapeLeft:
            angle = M_PI_2;
            break;
        default:
            break;
    }

    return angle;
}

- (CGAffineTransform)transformFromCurrentVideoOrientationToOrientation:(AVCaptureVideoOrientation)orientation
{
    CGAffineTransform transform = CGAffineTransformIdentity;

    // Calculate offsets from an arbitrary reference orientation (portrait)
    CGFloat orientationAngleOffset = [self angleOffsetFromPortraitOrientationToOrientation:orientation];
    CGFloat videoOrientationAngleOffset = [self angleOffsetFromPortraitOrientationToOrientation:self.videoOrientation];

    // Find the difference in angle between the passed in orientation and the current video orientation
    CGFloat angleOffset = orientationAngleOffset - videoOrientationAngleOffset;
    transform = CGAffineTransformMakeRotation(angleOffset);

    return transform;
}  

The problem is with the front camera orientation, using this code wont work after the user change to front camera.

It seems like what cause the problem is when I'm changing the cam to front, the AVCaptureConnection changes and I get different orientation for the front cam and for the back cam.

So maybe I need to adjust to the differences between the initial orientation for back and front cam.
I dont really want to change the connection's orientation every time the user switch cams, because as Apple say it hurts performance (and indeed it looks bad when im doing it), so instead Apple suggest to use the AVAssetWriterInput transform property to change the output orientation, but I'm not sure I can use the transform, because I want to let the user toggle the camera while recording but I cant change the transform after I start writing (it crashes)...

Any ideas how to solve this?

Answer 1

Not the easiest way, but I think it will definitely work if you intercept the frame and perform your own transforms. You don't need any UIView for that - AVFoundation allows direct access and manipulation of its frame (you can replace with your own buffer when writing to the file, if needed). The only drawback I see is performance. Even just flipping an image might be very slow on old devices and at high resolution.

If this does hurt performance, you can save the video as it is and maintain a separate array of orientations for each frame. After the video is saved, reopen it, perform transformations for each frame, and save it to a new file. I have done something similar and it does work.

On a side note, AVAssetWriterInput's transform property works only on iPhone. If, say, you record a video with AVFoundation and upload it somewhere and watch it in browser, it will have the wrong orientation. So if you're after a thorough solution, your own transformation is the way to go.

If you often do video/image processing (e.g. in this case, flipping / rotating), consider the OpenCV library. It takes some time to learn, but it's worth it.

Answer 2

Since the image from front camera is mirrored you will need to mirror the transformation as well. The function to do this is simply scaling one of the dimensions with -1. Try playing around with this:

transform = CGAffineTransformConcat(CGAffineTransformMakeRotation(angle), CGAffineTransformMakeScale(1.0, -1.0));

Note possible permutations are (beside angle) X scale factor is -1 and or swapping the order of concat

EDIT: explanation

AVCaptureSession will return you the image data that do not rotate as your device does, naturally I believe that would be landscape left or right. If you need your video to have a correct orientation depending on how you are holding the device you need to apply some transform to it, in most cases it is enough to just apply some rotation. However, a front camera is a specific case because it simulates the mirror while you are still receiving non-mirrored frames. As the result to this your video seems to be up-side-down or left-side-right or all the other combinations depending on what rotation you are applying, thus as you said "I tried a lot of angles but I just cant get it right...". Again, to simulate the mirror effect you should scale one of the axis by -1.

EDIT: swapping orientation (from comments)

This is just an idea and I think it is a good one.. Do not use the asset transformation at all, do your own. Create a view with size you want your video to be, then create an image view subview to it with all the transforms, clippings and content modes you need. From the samples you get create images, set them to the image view and get layer snapshot from the view. Then send the snapshot to the asset writer.

I know what you are thinking here is "overhead". Well, I don't think so. As the views and image views do transformation on CPU same does the asset writer, so all you did is you canceled its internal transformation and used your own system. If you will try this I would very much like to hear about your results.

To get the layer image:

- (UIImage *)imageFromLayer:(CALayer *)layer {
    UIGraphicsBeginImageContext([layer frame].size);
    [layer renderInContext:UIGraphicsGetCurrentContext()];
    UIImage *outputImage = UIGraphicsGetImageFromCurrentImageContext();

    UIGraphicsEndImageContext();

    return outputImage;
}