CODEWITHSUNDEEP
javascripthtml5-video
sheodox
sheodox

Reputation: 835

Playing one of multiple audio tracks in sync with a video

I'm trying to play a video in a web browser, the original video comes with two or more audio streams, each in a different language. I want to give the user the option to switch which audio track they're listening to.

I tried using audioTracks on the video element, but despite saying it's supported behind a flag in most browsers, at least in Firefox and Chrome I wouldn't say it's working at all (in Firefox it only shows the first track and the metadata was wrong, and in Chrome the video would pause as soon as you muted the main track, and you had to seek the video to get it to actually continue playing).

I tried using ffmpeg to save the individual audio tracks separately and tried playing them in sync with the video (setting audio.currentTime = video.currentTime in response to several events on the video like play, playing, pause, seeked, stalled), playing both audio tracks in <audio> elements connected to GainNodes using the Web Audio API (switching audio tracks sets the gain to 1 for the track you want and 0 for the rest). This seems to be working flawlessly in Chrome, but Firefox is all over the place and even after syncing the currentTime properties the actual audio is off by a second or more.

I saw other people complaining about similar issues with the timing being off with MP3, but I'm using AAC. The solution in those cases was to not use variable bitrates for the audio but that didn't seem to improve it (ffmpeg -i video.mkv -map 0:a:0 -acodec aac -b:a 128k track-0.aac)

Is there any good strategy for doing this? I'd rather not have to have duplicate video files for each audio track if I can avoid it.

Upvotes: 1

Views: 3581

Answers (1)

Kaiido
Kaiido

Reputation: 136707

The best in your case is probably to use the Media Source Extension (MSE) API.
This will allow you to switch only the audio source while keeping playing the original video.
Since we will replace the whole audio SourceBuffer's content with the other audio source, we won't have sync issues, for the player, it will be just as if there was a single audio source.

(async() => {
  const vid = document.querySelector( "video" );
  const check = document.querySelector( "input" );
  // video track as ArrayBuffer
  const bufvid = await getFileBuffer( "525d5ltprednwh1/test.webm" );
  // audio track one
  const buf300 = await getFileBuffer( "p56kvhwku7pdzd9/beep300hz.webm" );
  // audio track two
  const buf800 = await getFileBuffer( "me3y69ekxyxabhi/beep800hz.webm" );
  
  const source = new MediaSource();
  // load our MediaSource into the video
  vid.src = URL.createObjectURL( source );
  // when the MediaSource becomes open
  await waitForEvent( source, "sourceopen" );

  // append video track
  const vid_buffer = source.addSourceBuffer( "video/webm;codecs=vp8" );
  vid_buffer.appendBuffer( bufvid );

  // append one of the audio tracks
  const aud_buffer =  source.addSourceBuffer( "audio/webm;codecs=opus" );
  aud_buffer.appendBuffer( check.checked ? buf300 : buf800 );
  // wait for both SourceBuffers to be ready
  await Promise.all( [
    waitForEvent( aud_buffer, "updateend" ),
    waitForEvent( vid_buffer, "updateend" )
  ] );
  // Tell the UI the stream is ended (so that 'ended' can fire)
  source.endOfStream();
  
  check.onchange = async (evt) => {
    // remove all the data we had in the Audio track's buffer
    aud_buffer.remove( 0, source.duration );
    // it is async, so we need to wait it's done
    await waitForEvent( aud_buffer, "updateend" );
    // no we append the data of the other track
    aud_buffer.appendBuffer( check.checked ? buf300 : buf800 );
    // also async
    await waitForEvent( aud_buffer, "updateend" );
    // for ended to fire
    source.endOfStream();
  };

})();

// helpers
function getFileBuffer( filename ) {
  return fetch( "https://dl.dropboxusercontent.com/s/" + filename )
    .then( (resp) => resp.arrayBuffer() );
}
function waitForEvent( target, event ) {
  return new Promise( res => {
    target.addEventListener( event, res, { once: true } );
  } );
}
video { max-width: 100%; max-height: 100% }
<label>Use 300Hz audio track instead of 800Hz <input type="checkbox"></label><br>
<video controls></video>

Upvotes: 5

Related Questions