How to keep low latency during the preview of video coming from AVFoundation?

Question

Apple has a sample code called Rosy Writer that shows how to capture video and apply effects to it.

During this section of the code, on the outputPreviewPixelBuffer part, Apple supposedly shows how they keep preview latency low by dropping stale frames.

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription( sampleBuffer );

    if ( connection == _videoConnection )
    {
        if ( self.outputVideoFormatDescription == NULL ) {
            // Don't render the first sample buffer.
            // This gives us one frame interval (33ms at 30fps) for setupVideoPipelineWithInputFormatDescription: to complete.
            // Ideally this would be done asynchronously to ensure frames don't back up on slower devices.
            [self setupVideoPipelineWithInputFormatDescription:formatDescription];
        }
        else {
            [self renderVideoSampleBuffer:sampleBuffer];
        }
    }
    else if ( connection == _audioConnection )
    {
        self.outputAudioFormatDescription = formatDescription;

        @synchronized( self ) {
            if ( _recordingStatus == RosyWriterRecordingStatusRecording ) {
                [_recorder appendAudioSampleBuffer:sampleBuffer];
            }
        }
    }
}

- (void)renderVideoSampleBuffer:(CMSampleBufferRef)sampleBuffer
{
    CVPixelBufferRef renderedPixelBuffer = NULL;
    CMTime timestamp = CMSampleBufferGetPresentationTimeStamp( sampleBuffer );

    [self calculateFramerateAtTimestamp:timestamp];

    // We must not use the GPU while running in the background.
    // setRenderingEnabled: takes the same lock so the caller can guarantee no GPU usage once the setter returns.
    @synchronized( _renderer )
    {
        if ( _renderingEnabled ) {
            CVPixelBufferRef sourcePixelBuffer = CMSampleBufferGetImageBuffer( sampleBuffer );
            renderedPixelBuffer = [_renderer copyRenderedPixelBuffer:sourcePixelBuffer];
        }
        else {
            return;
        }
    }

    if ( renderedPixelBuffer )
    {
        @synchronized( self )
        {
            [self outputPreviewPixelBuffer:renderedPixelBuffer];

            if ( _recordingStatus == RosyWriterRecordingStatusRecording ) {
                [_recorder appendVideoPixelBuffer:renderedPixelBuffer withPresentationTime:timestamp];
            }
        }

        CFRelease( renderedPixelBuffer );
    }
    else
    {
        [self videoPipelineDidRunOutOfBuffers];
    }
}

// call under @synchronized( self )
- (void)outputPreviewPixelBuffer:(CVPixelBufferRef)previewPixelBuffer
{
    // Keep preview latency low by dropping stale frames that have not been picked up by the delegate yet
    // Note that access to currentPreviewPixelBuffer is protected by the @synchronized lock
    self.currentPreviewPixelBuffer = previewPixelBuffer; // A

    [self invokeDelegateCallbackAsync:^{  // B

        CVPixelBufferRef currentPreviewPixelBuffer = NULL; // C
        @synchronized( self ) //D
        {
            currentPreviewPixelBuffer = self.currentPreviewPixelBuffer; // E
            if ( currentPreviewPixelBuffer ) { // F
                CFRetain( currentPreviewPixelBuffer ); // G
                self.currentPreviewPixelBuffer = NULL;  // H
            }
        }

        if ( currentPreviewPixelBuffer ) { // I
            [_delegate capturePipeline:self previewPixelBufferReadyForDisplay:currentPreviewPixelBuffer];  // J
            CFRelease( currentPreviewPixelBuffer );  /K
        }
    }];
}

- (void)invokeDelegateCallbackAsync:(dispatch_block_t)callbackBlock
{
    dispatch_async( _delegateCallbackQueue, ^{
        @autoreleasepool {
            callbackBlock();
        }
    } );
}

After hours of trying to understand this code, my brain is smoking and I cannot see how this is done.

Can someone explain like I am 5 years old, OK, make it 3 years old, how is this code doing that?

thanks.

EDIT: I have labeled the lines of outputPreviewPixelBuffer with letters to make it easy to understand the order the code is being executed.

So, the method starts and A runs and the buffer is stored into the property self.currentPreviewPixelBuffer. B runs and the local variable currentPreviewPixelBuffer is assigned with NULL. D runs and locks self. Then E runs and changes the local variable currentPreviewPixelBuffer from NULL to the value of self.currentPreviewPixelBuffer.

This is the first thing that does not makes sense. Why would I create a variable currentPreviewPixelBuffer assign it to NULL and on the next line assign it to self.currentPreviewPixelBuffer?

The following line is even more insane. Why I am asking if currentPreviewPixelBuffer is not NULL If I just assigned it to a non NULL value on E? Then H is executed and nulls self.currentPreviewPixelBuffer?

One thing I don't get is this: invokeDelegateCallbackAsync: is asynchronous, right? if it is asynchronous then every time outputPreviewPixelBuffer method runs is to set self.currentPreviewPixelBuffer = previewPixelBuffer and dispatch a block for execution, being free to run again.

If outputPreviewPixelBuffer is fired faster, we will have a bunch of blocks piled for execution.

Due to the explanations of Kamil Kocemba, I undestand that these asynchronous blocks are testing somehow if the previous one finished executing and dropping the frames if not.

Also, what exactly is @syncronized(self) locking? Is it preventing self.currentPreviewPixelBuffer from being written or read? or is it locking the local variable currentPreviewPixelBuffer? If the block under @syncronized(self) is synchronous with relation to the scope the line at I will never be NULL because it is being set on E.

Itai Ferber · Accepted Answer

Thank you for highlighting the lines -- this will hopefully make the answer a little bit easier to follow.

Let's go through step by step:

-outputPreviewPixelBuffer: is called. self.currentPreviewPixelBuffer is overwritten not in an @synchronized block: this means that it is forcibly overwritten, effectively for all threads (I'm glossing over the fact that currentPreviewPixelBuffer is nonatomic; this is actually unsafe and there is a race here -- you really need it to be strong, atomic for this to be really true). If there was a buffer in there, it's now gone the next time a thread is going to go looking for it. This is what the documentation implies -- if there was a value in self.currentPreviewPixelBuffer and the delegate has not yet gotten to process the previous value, too bad! It's gone now.
The block is sent to the delegate to process asynchronously. In effect, this will likely happen sometime in the future, with some indeterminate delay. This means that between when -outputPreviewPixelBuffer: is called and when the block is processed, -outputPreviewPixelBuffer: can get called again many, many times! This is how the stale frames are dropped -- if it's taking a long time for the delegate to get to processing the block, the latest self.currentPreviewPixelBuffer will get overwritten with the latest value again and again, effectively dropping the previous frame.
Lines C through H take ownership of self.currentPreviewPixelBuffer. You indeed have a local pixel buffer, initially set to NULL. The @synchronized block around self says, implicitly: "I am going to moderate access to self, to make sure no one edits self while I'm looking at it, and also I will ensure that I grab the most up-to-date value of self's instance variables, even across threads". This is how the delegate ensures that it has the latest self.currentPreviewPixelBuffer; if it was not @synchronized, you could get a stale copy.

Also in the @synchronized block is the overwrite of self.currentPreviewPixelBuffer, after retaining it. This code implicitly says: "hey, if self.currentPreviewPixelBuffer is not NULL, then there must be a pixel buffer to process; if there is (line F), then I'll hold on to it (line E, G), and reset it on self (line H)". In effect, this takes ownership of self's currentPreviewPixelBuffer so that no one else will process it. This is an implicit check for all delegate callback blocks operating on self: the first block to fire that looks at self.currentPreviewPixelBuffer gets to keep it, sets it to NULL for all other blocks looking at self, and does work with it. The others, having read NULL on line F, do nothing.
Lines I and J actually use the pixel buffer, and line K disposes of it properly.

It's true, this code could use some commenting -- it's really lines E through G that do a lot of the implicit work here, taking ownership of self's preview buffer to keep others from processing the block as well. What the comment above line A does not say is, "Note that access to currentPreviewPixelBuffer is protected by @synchronized..., in contrast to here where it's not; because it's not protected by that here, we can overwrite self.currentPreviewPixelBuffer as many times as we want before someone processes it, dropping the intermediate values"

Hope that helps.

How to keep low latency during the preview of video coming from AVFoundation?

Answers (2)

Related Questions