How do you add an overlay while recording a video in Swift?

Question

I am trying to record, and then save, a video in Swift using AVFoundation. This works. I am also trying to add an overlay, such as a text label containing the date, to the video.

For example: the video saved is not only what the camera sees, but the timestamp as well.

Here is how I am saving the video:

func fileOutput(_ output: AVCaptureFileOutput, didFinishRecordingTo outputFileURL: URL, from connections: [AVCaptureConnection], error: Error?) {
    saveVideo(toURL: movieURL!)
  }

  private func saveVideo(toURL url: URL) {
    PHPhotoLibrary.shared().performChanges({
      PHAssetChangeRequest.creationRequestForAssetFromVideo(atFileURL: url)
    }) { (success, error) in
      if(success) {
        print("Video saved to Camera Roll.")
      } else {
        print("Video failed to save.")
      }
    }
  }

I have a movieOuput that is an AVCaptureMovieFileOutput. My preview layer does not contain any sublayers. I tried adding the timestamp label's layer to the previewLayer, but this did not succeed.

I have tried Ray Wenderlich's example as well as this stack overflow question. Lastly, I also tried this tutorial, all of which to no avail.

How can I add an overlay to my video that is in the saved video in the camera roll?

Answer 1

Adding on to @Kevin Ng, you can do an overlay on video frames with an UIViewController and an UIView.

UIViewController will have:

1. property to work with video stream

    private var videoSession: AVCaptureSession?
   

2. property to work with overlay(the UIView class)

    private var myOverlay: MyUIView{view as! MyUIView}
   

3. property to work with video output queue

   private let videoOutputQueue = DispatchQueue(label: 
   "outputQueue", qos: .userInteractive)
   

4. method to create video session

5. method to process and display overlay

UIView will have task-specific helper methods needed to to act as overlay. For example, if you are doing hand detection, this overlay class can have helper methods to draw points on coordinates(ViewController class will detect coordinates of hand features, do necessary coordinate conversions, then pass the coordinates to the UIView class to display coordinates as an overlay)

Answer 2

Without more information it sounds like what you are asking for is a WATERMARK. Not an overlay.

A watermark is a markup on the video that will be saved with the video. An overlay is generally showed as subviews on the preview layer and will not be saved with the video.

Check this out here: https://stackoverflow.com/a/47742108/8272698

func addWatermark(inputURL: URL, outputURL: URL, handler:@escaping (_ exportSession: AVAssetExportSession?)-> Void) {
    let mixComposition = AVMutableComposition()
    let asset = AVAsset(url: inputURL)
    let videoTrack = asset.tracks(withMediaType: AVMediaType.video)[0]
    let timerange = CMTimeRangeMake(kCMTimeZero, asset.duration)

        let compositionVideoTrack:AVMutableCompositionTrack = mixComposition.addMutableTrack(withMediaType: AVMediaType.video, preferredTrackID: CMPersistentTrackID(kCMPersistentTrackID_Invalid))!

    do {
        try compositionVideoTrack.insertTimeRange(timerange, of: videoTrack, at: kCMTimeZero)
        compositionVideoTrack.preferredTransform = videoTrack.preferredTransform
    } catch {
        print(error)
    }

    let watermarkFilter = CIFilter(name: "CISourceOverCompositing")!
    let watermarkImage = CIImage(image: UIImage(named: "waterMark")!)
    let videoComposition = AVVideoComposition(asset: asset) { (filteringRequest) in
        let source = filteringRequest.sourceImage.clampedToExtent()
        watermarkFilter.setValue(source, forKey: "inputBackgroundImage")
        let transform = CGAffineTransform(translationX: filteringRequest.sourceImage.extent.width - (watermarkImage?.extent.width)! - 2, y: 0)
        watermarkFilter.setValue(watermarkImage?.transformed(by: transform), forKey: "inputImage")
        filteringRequest.finish(with: watermarkFilter.outputImage!, context: nil)
    }

    guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPreset640x480) else {
        handler(nil)

        return
    }

    exportSession.outputURL = outputURL
    exportSession.outputFileType = AVFileType.mp4
    exportSession.shouldOptimizeForNetworkUse = true
    exportSession.videoComposition = videoComposition
    exportSession.exportAsynchronously { () -> Void in
        handler(exportSession)
    }
}

And heres how to call the function.

let outputURL = NSURL.fileURL(withPath: "TempPath")
let inputURL = NSURL.fileURL(withPath: "VideoWithWatermarkPath")
addWatermark(inputURL: inputURL, outputURL: outputURL, handler: { (exportSession) in
    guard let session = exportSession else {
        // Error 
        return
    }
    switch session.status {
        case .completed:
        guard NSData(contentsOf: outputURL) != nil else {
            // Error
            return
        }

        // Now you can find the video with the watermark in the location outputURL

        default:
        // Error
    }
})

Let me know if this code works for you. It is in swift 3 so some changes will be needed. I currently am using this code on an app of mine. Have not updated it to swift 5 yet

Answer 3

I do not have an actual development environment for Swift that can utilize AVFoundation. Thus, I can't provide you with any example code.

For adding meta data(date, location, timestamp, watermark, frame rate, etc...) as an overlay to the video while recording, you would have to process the video feed, frame by frame, live, while recording. Most likely you would have to store the frames in a buffer and process them before actually record them.

Now when it come to the meta data, there are two type, static and dynamic. For static type such as a watermark, it should be easy enough, as all the frames will get the same thing.

However, for dynamic meta data type such as timestamp or GPS location, there are a few things that needed to be taken into consideration. It takes computational power and time to process the video frames. Thus, depends on the type of dynamic data and how you got them, sometime the processed value may not be a correct value. For example, if you got a frame at 1:00:01, you process it and add a timestamp to it. Just pretend that it took 2 seconds to process the timestamp. The next frame you got is at 1:00:02, but you couldn't process it until 1:00:03 because processing the previous frame took 2 seconds. Thus, depend on how you got that new timestamp for the new frame, that timestamp value may not be the value that you wanted.

For processing dynamic meta data, you should also take into consideration of hardware lag. For example, the software is supposed to add live GPS location data to each frame and there weren't any lags in development or in testing. However, in real life, a user used the software in an area with a bad connection, and his phone lag while obtaining his GPS location. Some of his lags lasted as long as 5 seconds. What do you do in that situation? Do you set a time out for the GPS location and used the last good position? Do you report the error? Do you defer that frame to be process later when the GPS data become available(This may ruin live recording) and using an expensive algorithm to try to predict the user's location for that frame?

Besides those to take into consideration, I have some references here that I think may help you. I thought the one from medium.com looked pretty good.

https://medium.com/ios-os-x-development/ios-camera-frames-extraction-d2c0f80ed05a

Adding watermark to currently recording video and save with watermark

Render dynamic text onto CVPixelBufferRef while recording video