CODEWITHSUNDEEP
iosobjective-cmacosavfoundationvideo-processing
kennyc
kennyc

Reputation: 5700

How to composite videos using multiple AVVideoCompositions

I'm trying to figure out how to composite multiple videos (AVAssets) into a single video such that each of the videos goes through its own video composition. However, I can't see a way to accomplish this and was wondering if anyone had any ideas.

Consider the following:

enter image description here

The above picture illustrates what I'm trying to do. I want to take the video track from four different videos and merge them into a single video such that they play in a grid-like layout.

Right now, I'm able to achieve this by doing the following

While this works, it doesn't offer me the ability to do any custom effects on each video track before compositing it into the output clip. For example, in the picture above I'd like to add a unique border color around each track. With a single video (and thus no layer instructions), then I can just add the border using something like AVVideoComposition applyingCIFiltersWithHandler, which works great. But I can't use that because I need the video composition for setting the transform and scale instructions.

(applyingCIFiltersWithHandler doesn't appear to offer a way to get access to the tracks that make up the frame being rendered.)

What I think I want is the ability to do some type of custom rendering within a AVMutableVideoCompositionLayerInstruction, but the only options currently available are transform and opacity ones.

(Note that any solution requires that I have access to the current composition time during rendering for each video track because some of the "effects" I'd like to render are a frame counter and a timecode display.)

Any help, ideas or tips would be appreciated.

Upvotes: 9

Views: 3324

Answers (1)

Tim Bull
Tim Bull

Reputation: 2495

You need to implement a custom compositor that implements the AVVideoCompositing protocol. https://developer.apple.com/reference/avfoundation/avvideocompositing

The startRequest function within that protocol receives an https://developer.apple.com/reference/avfoundation/avasynchronousvideocompositionrequest and this allows you to access the information from each track https://developer.apple.com/reference/avfoundation/avasynchronousvideocompositionrequest/1390379-sourceframe, then combine them with your own custom instructions and composite (put them together) in your own way, returning a final single buffer that has the frame composited from the tracks in the way you like.

Upvotes: 7

Related Questions