CODEWITHSUNDEEP
mpmovieplayercontrolleravfoundationavplayersubtitletransport-stream
Vasu N
Vasu N

Reputation: 352

Subtitles for AVPlayer/MPMoviePlayerController

I am using m3u8 video format for streaming the video and now I need to display subtitles for the same.

I searched in Apple Documentation and found that I can achieve this by using the closedCaptionDisplayEnabled property of AVPlayer.

I am interested to know what should be the format of subtitles? Will the .srt format do?

Also can I achieve the same using MPMoviePlayerController?

Any help is appreciated.

Upvotes: 17

Views: 23564

Answers (6)

Johannes Fahrenkrug
Johannes Fahrenkrug

Reputation: 44700

Update 03/06/2020: On GitHub, jbweimar has created a sample project that uses the AVAssetResourceLoaderDelegate approach that looks very promising: https://github.com/jbweimar/external-webvtt-example

Update 10/30/2018: It's worth checking this answer by an Apple engineer (Thanks to @allenlini for pointing it out). He suggests a solution involving AVAssetResourceLoaderDelegate. I haven't tried it myself, but it might be a better solution than mine below.

Original Answer:

It seems as if referencing your WebVTT files in your m3u8 streaming description is the officially supported way. Adding them "after the fact" seems to not be officially supported (See this statement by an Apple engineer (bottom of the page)).

That - however - does not mean that you can't get it to work ;-). With the help of this great presentation and sample project (ZIP) by Chris Adamson, this post on the Apple Developer Forums and this Ray Wenderlich tutorial by Abdul Azeem, I was able to get it to work. This is a modified version of Abdul Azeem's sample code and Chris' sample project.

Note how you need to use AVMediaTypeText instead of AVMediaTypeSubtitle. This seems to be a bug in iOS.

    // 1 - Load video asset
    AVAsset *videoAsset = [AVURLAsset assetWithURL:[[NSBundle mainBundle] URLForResource:@"video" withExtension:@"mp4"]];

    // 2 - Create AVMutableComposition object. This object will hold your AVMutableCompositionTrack instances.
    AVMutableComposition *mixComposition = [[AVMutableComposition alloc] init];

    // 3 - Video track
    AVMutableCompositionTrack *videoTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeVideo
                                                                        preferredTrackID:kCMPersistentTrackID_Invalid];
    [videoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                        ofTrack:[[videoAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0]
                         atTime:kCMTimeZero error:nil];

    // 4 - Subtitle track
    AVURLAsset *subtitleAsset = [AVURLAsset assetWithURL:[[NSBundle mainBundle] URLForResource:@"subtitles" withExtension:@"vtt"]];

    AVMutableCompositionTrack *subtitleTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeText
                                                                           preferredTrackID:kCMPersistentTrackID_Invalid];

    [subtitleTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                           ofTrack:[[subtitleAsset tracksWithMediaType:AVMediaTypeText] objectAtIndex:0]
                            atTime:kCMTimeZero error:nil];

    // 5 - Set up player
    AVPlayer *player = [AVPlayer playerWithPlayerItem: [AVPlayerItem playerItemWithAsset:mixComposition]];

Upvotes: 37

clopex
clopex

Reputation: 459

If someone need for lates Swift and custom SwiftUI view (AVplayerController wrapped for SwiftUI) this is the changes, thanks to @perpeer:

let videoAsset = AVURLAsset(url: videoUrl)
    let subtitleAsset = AVURLAsset(url: subtitleUrl)
    
    let mixComposition = AVMutableComposition()
    let videoTrack = mixComposition.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid)
    let audioTrack = mixComposition.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)
    let subtitleTrack = mixComposition.addMutableTrack(withMediaType: .text, preferredTrackID: kCMPersistentTrackID_Invalid)
    
    Task {
        if let videoTrackItem = try await videoAsset.loadTracks(withMediaType: .video).first {
            try await videoTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.load(.duration)),
                                            of: videoTrackItem,
                                            at: .zero)
        }
        if let audioTrackItem = try await videoAsset.loadTracks(withMediaType: .audio).first {
            try await audioTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.load(.duration)),
                                            of: audioTrackItem,
                                            at: .zero)
        }
        if let subtitleTrackItem = try await subtitleAsset.loadTracks(withMediaType: .text).first {
            try await subtitleTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.load(.duration)),
                                               of: subtitleTrackItem,
                                               at: .zero)
        }
        DispatchQueue.main.async {
            let player = AVPlayer(playerItem: AVPlayerItem(asset: mixComposition))
            let vc = AVPlayerViewController()
            vc.player = player
            self.present(vc, animated: true) {
                vc.player?.play()
            }
        }
    }

Tried with local and remote url and it works, main thing is that your subtitle must be with .vtt extension

Upvotes: 0

perpeer
perpeer

Reputation: 61

Audio error occurred when adding subtitles. Updated version as below;

Note: It is only possible to insert local items (assets) into an AVMutableComposition, remote items (like HTTP video streams) will not work before iOS 15.

Video and Subtitle files

Inserting an HTTP stream into a AVMutableComposition

Swift 5

func play() {
    guard let videoUrl = Bundle.main.url(forResource: "ElephantsDream", withExtension: "mp4")?.absoluteURL,
          let subtitleUrl = Bundle.main.url(forResource: "subtitles-en", withExtension: "vtt")?.absoluteURL else {
              return
          }


    let videoAsset = AVURLAsset(url: videoUrl)
    let subtitleAsset = AVURLAsset(url: subtitleUrl)

    // Create AVMutableComposition object. This object will hold your AVMutableCompositionTrack instances.
    let mixComposition = AVMutableComposition()

    let videoTrack = mixComposition.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid)
    let audioTrack = mixComposition.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)
    let subtitleTrack = mixComposition.addMutableTrack(withMediaType: .text, preferredTrackID: kCMPersistentTrackID_Invalid)

    do {
        try videoTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.duration),
                                        of: videoAsset.tracks(withMediaType: .video).first!,
                                        at: .zero)
        try audioTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.duration),
                                        of: videoAsset.tracks(withMediaType: .audio).first!,
                                        at: .zero)
        try subtitleTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: subtitleAsset.duration),
                                           of: subtitleAsset.tracks(withMediaType: .text).first!, at: .zero)
    } catch let err {
        print(err.localizedDescription)
    }

    let player = AVPlayer(playerItem: AVPlayerItem(asset: mixComposition))
    let vc = AVPlayerViewController()
    vc.player = player
    present(vc, animated: true) {
        vc.player?.play()
    }
}

If you want to work with server-side video and subtitles you need to use like this.

iOS 15

func play() {
    if #available(iOS 15.0.0, *) {
        let videoUrl = URL(string: "https://wwww.acme.com/video/elephantsdream.mov")!
        let subtitleUrl = URL(string: "https://wwww.acme.com/subtitle/subtitles-en.vtt")!
        
        let videoAsset = AVURLAsset(url: videoUrl)
        let subtitleAsset = AVURLAsset(url: subtitleUrl)

        let mixComposition = AVMutableComposition()
        let videoTrack = mixComposition.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid)
        let audioTrack = mixComposition.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)
        let subtitleTrack = mixComposition.addMutableTrack(withMediaType: .text, preferredTrackID: kCMPersistentTrackID_Invalid)

        Task {
            if let videoTrackItem = try await videoAsset.loadTracks(withMediaType: .video).first {
                try videoTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.duration),
                                                of: videoTrackItem,
                                                at: .zero)
            }
            if let audioTrackItem = try await videoAsset.loadTracks(withMediaType: .audio).first {
                try audioTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.duration),
                                                of: audioTrackItem,
                                                at: .zero)
            }
            if let subtitleTrackItem = try await subtitleAsset.loadTracks(withMediaType: .text).first {
                try subtitleTrack?.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.duration),
                                                   of: subtitleTrackItem,
                                                   at: .zero)
            }
            DispatchQueue.main.async {
                let player = AVPlayer(playerItem: AVPlayerItem(asset: mixComposition))
                let vc = AVPlayerViewController()
                vc.player = player
                self.present(vc, animated: true) {
                    vc.player?.play()
                }
            }
        }
    }
}

Upvotes: 2

jmartinalonso
jmartinalonso

Reputation: 2482

This is the Swift version of @JohannesFahrenkrug answer. Hope this is useful for someone:

    let localVideoAsset = Bundle.main.url(forResource: "L1C1P1C3", withExtension: "mp4")

    //Create AVMutableComposition
    let videoPlusSubtitles = AVMutableComposition()

    //Adds video track
    let videoTrack = videoPlusSubtitles.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid)

    try? videoTrack?.insertTimeRange(CMTimeRangeMake(kCMTimeZero, localVideoAsset.duration),
                                of: localVideoAsset.tracks(withMediaType: .video)[0],
                                at: kCMTimeZero)

    //Adds subtitle track
    let subtitleAsset = AVURLAsset(url: Bundle.main.url(forResource: "L1C1P1C3", withExtension: ".mp4.vtt")!)

    let subtitleTrack = videoPlusSubtitles.addMutableTrack(withMediaType: .text, preferredTrackID: kCMPersistentTrackID_Invalid)

    try? subtitleTrack?.insertTimeRange(CMTimeRangeMake(kCMTimeZero, localVideoAsset.duration),
                                        of: subtitleAsset.tracks(withMediaType: .text)[0],
                                        at: kCMTimeZero)

Upvotes: 7

Peter Hosey
Peter Hosey

Reputation: 96323

The WWDC 2013 iOS app uses WebVTT files for its subtitles (credit to Nicholas Riley for discovering this). For whatever reason, there are about 50 (exact number varies) for each session.

I have no idea whether WebVTT is supported at the AVFoundation or MPMoviePlayer level, or whether the app downloads and parses the subtitles files itself.

A quick search for “webvtt m3u8” turned up a reference to this HTTP Live Streaming Draft, which suggests that you may be able to make this work by simply referencing the WebVTT files in your m3u8 playlist. I don't have an example for you, though, since I wasn't able to trivially guess the m3u8 URL for a WWDC session.

Upvotes: 0

Dave Batton
Dave Batton

Reputation: 8845

Both AVPlayer and MPMoviePlayerController can display subtitles.

The difference seems to be that with AVPlayer, you control whether or not the subtitles are displayed using the closedCaptionDisplayEnabled property.

With MPMoviePlayerController, the user controls whether or not subtitles are displayed using a switch in the Setting app. You get no programatic control of this in the app.

Upvotes: 2

Related Questions