Skip over frames while processing video on iOS

Question

I'm trying to process a local video file and simply do some analysis on the pixel data. Nothing is being output. My current code iterates through each frame of the video but I'd actually like to skip ~15 frames at a time to speed things up. Is there a way to skip over frames without decoding them?

In Ffmpeg, I could simply call av_read_frame without calling avcodec_decode_video2.

Thanks in advance! Here's my current code:

- (void) readMovie:(NSURL *)url
{

    [self performSelectorOnMainThread:@selector(updateInfo:) withObject:@"scanning" waitUntilDone:YES];

    startTime = [NSDate date];

    AVURLAsset * asset = [AVURLAsset URLAssetWithURL:url options:nil];

    [asset loadValuesAsynchronouslyForKeys:[NSArray arrayWithObject:@"tracks"] completionHandler:
     ^{
         dispatch_async(dispatch_get_main_queue(),
                        ^{



                            AVAssetTrack * videoTrack = nil;
                            NSArray * tracks = [asset tracksWithMediaType:AVMediaTypeVideo];
                            if ([tracks count] == 1)
                            {
                                videoTrack = [tracks objectAtIndex:0];

                                videoDuration = CMTimeGetSeconds([videoTrack timeRange].duration);

                                NSError * error = nil;

                                // _movieReader is a member variable
                                _movieReader = [[AVAssetReader alloc] initWithAsset:asset error:&error];
                                if (error)
                                    NSLog(@"%@", error.localizedDescription);       

                                NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
                                NSNumber* value = [NSNumber numberWithUnsignedInt: kCVPixelFormatType_420YpCbCr8Planar];

                                NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key]; 

                                AVAssetReaderTrackOutput* output = [AVAssetReaderTrackOutput 
                                                         assetReaderTrackOutputWithTrack:videoTrack 
                                                         outputSettings:videoSettings];
                                output.alwaysCopiesSampleData = NO;

                                [_movieReader addOutput:output];

                                if ([_movieReader startReading])
                                {
                                    NSLog(@"reading started");

                                    [self readNextMovieFrame];
                                }
                                else
                                {
                                    NSLog(@"reading can't be started");
                                }
                            }
                        });
     }];
}


- (void) readNextMovieFrame
{
    //NSLog(@"readNextMovieFrame called");
    if (_movieReader.status == AVAssetReaderStatusReading)
    {
        //NSLog(@"status is reading");

        AVAssetReaderTrackOutput * output = [_movieReader.outputs objectAtIndex:0];
        CMSampleBufferRef sampleBuffer = [output copyNextSampleBuffer];
        if (sampleBuffer)
        { // I'm guessing this is the expensive part that we can skip if we want to skip frames
            CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); 

            // Lock the image buffer
            CVPixelBufferLockBaseAddress(imageBuffer,0); 

            // Get information of the image
            uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
            size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
            size_t width = CVPixelBufferGetWidth(imageBuffer);
            size_t height = CVPixelBufferGetHeight(imageBuffer); 

            // do my pixel analysis

            // Unlock the image buffer
            CVPixelBufferUnlockBaseAddress(imageBuffer,0);
            CFRelease(sampleBuffer);


            [self readNextMovieFrame];
        }
        else
        {
            NSLog(@"could not copy next sample buffer. status is %d", _movieReader.status);

            NSTimeInterval scanDuration = -[startTime timeIntervalSinceNow];

            float scanMultiplier = videoDuration / scanDuration;

            NSString* info = [NSString stringWithFormat:@"Done\n\nvideo duration: %f seconds\nscan duration: %f seconds\nmultiplier: %f", videoDuration, scanDuration, scanMultiplier];

            [self performSelectorOnMainThread:@selector(updateInfo:) withObject:info waitUntilDone:YES];
        }


    }
    else
    {
        NSLog(@"status is now %d", _movieReader.status);


    }

}


- (void) updateInfo: (id*)message
{
    NSString* info = [NSString stringWithFormat:@"%@", message];

    [infoTextView setText:info];
}

Answer 1

If you want less accurate frame processing (not frame by frame) you should use AVAssetImageGenerator.

This class returns a frame for a specified time you asked.

Specifically, build an Array filled with times between the clip's duration with 0.5s difference between each time (iPhone films at about 29.3 fps if you want every 15 frames its about frame for every 30 seconds) and let the image generator returns your frames.

For each frame you can see the time you requested and the actual time of the frame. It's default value is around 0.5s tolerance from the time you asked but you can also change that by changing the properties:

requestedTimeToleranceBefore and requestedTimeToleranceAfter

I hope I answered your question, Good luck.