CODEWITHSUNDEEP
javascripthtmlvideovideo-streamingwebrtc
user6233283
user6233283

Reputation:

WebRTC MediaRecorder on remote stream cuts when the stream hangs

The Problem:

During a WebRTC unicast video conference, I can successfully stream video from a mobile device's webcam to a laptop/desktop. I would like to record the remote stream on the laptop/desktop side. (The setup is that a mobile device streams to a laptop/desktop).

However, it is usual for the video stream to hang from time to time. That's not a problem, for the "viewer" side will catch up. However, the recording of the remote stream will stop at the first hang.

Minimal and Removed Implementation (Local Recording):

I can successfully record the local stream from navigator.mediaDevices.getUserMedia() as follows:

const recordedChunks = [];

navigator.mediaDevices.getUserMedia({
    video: true,
    audio: false
}).then(stream => {
    const localVideoElement = document.getElementById('local-video');
    localVideoElement.srcObject = stream;
    return stream;
}).then(stream => {
    const mediaRecorder = new MediaRecorder(stream);
    mediaRecorder.ondataavailable = (event) => {
        if(event.data && event.data.size > 0) {
            recordedChunks.push(event.data);
        }
    };
    mediaRecorder.start({ mimeType: 'video/webm;codecs=vp9' }, 10);
});

I can download this quite easily as follows:

const blob = new Blob(recordedChunks, { type: 'video/webm' });

const url = URL.createObjectURL(blob);
const a = document.createElement('a');
document.body.appendChild(a);
a.style = 'display: none';
a.href = url;
a.download = 'test.webm';
a.click();
window.URL.revokeObjectURL(url);

Minimal and Removed Implementation (Remote Recording):

The setup I am using requires recording the remote stream, not the local stream, for IOS Safari does not support the MediaRecorder API. I included the above to show that the recording is working on the local side. The implementation of the remote stream recording is no different except I manually add a 0 Hz audio track to the video, for Chrome appears to have a bug where it won't record without an audio track.

const mediaStream = new MediaStream();
const audioContext = new AudioContext();

const destinationNode = audioContext.createMediaStreamDestination();

const oscillatorNode = audioContext.createOscillator();
oscillatorNode.frequency.setValueAtTime(0, audioContext.currentTime);
oscillatorNode.connect(destinationNode);

const audioTrack = destinationNode.stream.getAudioTracks()[0];
const videoTrack = remoteStream.getVideoTracks()[0]; // Defined somewhere else.

mediaStream.addTrack(videoTrack);
mediaStream.addTrack(audioTrack);

And then I perform the exact same operations that I do on the local stream example above to record the mediaStream variable.

As mentioned, at the first point where the remote stream hangs (due to network latency, perhaps), the remote recording ceases, such that on download, the duration of the .webm file converted to .mp4, via ffmpeg, is only as long as to where the first hang occurred.

Attempts to Mitigate:

One attempt to mitigate this issue I have tried is, rather than recording the remote stream that is attained in the callback for the ontrack event from WebRTC, I use the video stream from the remote video element instead, via remoteVideoElement.captureStream(). This does not work to fix the issue.

Any help would be much appreciated. Thank you.

Upvotes: 2

Views: 2620

Answers (1)

Brad
Brad

Reputation: 163234

Hopefully, someone is able to post an actual fix for you. In the mean time, a nasty, inefficient, totally-not-recommended workaround:

  1. Route the incoming MediaStream to a video element.
  2. Use requestAnimationFrame() to schedule drawing frames to a canvas. (Note that this removes any sense of genlock from the original video, and is not something you want to do. Unfortunately, we don't have a way of knowing when incoming frames occur, as far as I know.)
  3. Use CanvasCaptureMediaStream as the video source.
  4. Recombine the video track from CanvasCaptureMediaStream along with the audio track from the original MediaStream in a new MediaStream.
  5. Use this new MediaStream for MediaRecorder.

I've done this with past projects where I needed to programatically manipulate the audio and video. It works!

One big caveat is that there's a bug in Chrome where even though a capture stream is attached to a canvas, the canvas won't be updated if the tab isn't active/visible. And, of course, requestAnimationFrame is severely throttled at best if the tab isn't active, so you need another frame clock source. (I used audio processors, ha!)

Upvotes: 1

Related Questions