CODEWITHSUNDEEP
javascriptgetusermediabrowser-extensionmediastream
Dov Rine
Dov Rine

Reputation: 840

Can a browser plugin "mitm" the local webcam?

I want to create a browser extension that would allow users to add effects to their video/audio streams, without special plugins, on any site that uses the javascript web apis.

Google searching has not been particularly helpful so I'm starting to wonder if this is even possible.

I have 2 primary questions here:

  1. Is this possible with javascript+chrome?

  2. Any links to additional resources are greatly appreciated.

Upvotes: 1

Views: 111

Answers (1)

Kaiido
Kaiido

Reputation: 136588

I am not really into web-extensions, so there may even be a simpler API available and I won't go into details about the implementation but theoretically you can indeed do it.

All it takes is to override the methods from where you'd get your MediaStream, to draw the original MediaStream to an HTML canvas where you'd be able to apply your filter, and then simply to return a new MediaStream made of the VideoTrack of a MediaStream from the canvas element's captureStream(), and possibly other tracks from the original MediaStream.

A very basic proof of concept implementation for gUM could look like:

// overrides getUserMedia so it applies an invert filter on the videoTrack
{
  const mediaDevices = navigator.mediaDevices;
  const original_gUM = mediaDevices.getUserMedia.bind(mediaDevices);
  
  mediaDevices.getUserMedia = async (...args) => {
    const original_stream = await original_gUM(...args);

    // no video track, no filter
    if( !original_stream.getVideoTracks().length ) {
      return original_stream;
    }

    // prepare our DOM elements
    const canvas = document.createElement('canvas');
    const ctx = canvas.getContext('2d');
    const video = document.createElement('video');
    // a flag to know if we should keep drawing on the canvas or not
    let should_draw = true;

    // no need for audio there
    video.muted = true;
    // gUM video tracks can change size
    video.onresize = (evt) => {
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;
    };
    // in case users blocks the camera?
    video.onpause = (evt) => {
      should_draw = false;
    };
    video.onplaying = (evt) => {
      should_draw = true;
      drawVideoToCanvas();
    };
    
    video.srcObject = original_stream;
    
    await video.play();

    const canvas_track = canvas.captureStream().getVideoTracks()[0];
    const originalStop = canvas_track.stop.bind(canvas_track);
    // override the #stop method so we can revoke the camera stream
    canvas_track.stop = () => {
      originalStop();
      should_draw = false;
      original_stream.getVideoTracks()[0].stop();
    };

    // merge with audio tracks
    return new MediaStream( original_stream.getAudioTracks().concat( canvas_track ) );
    
    // the drawing loop
    function drawVideoToCanvas() {
      if(!should_draw) {
        return;
      }
      ctx.filter = "none";
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      ctx.filter = "invert(100%)";
      ctx.drawImage(video,0,0);
      requestAnimationFrame( drawVideoToCanvas );
    }

  };
}

And then every scripts that would call this method would receive a filtered videoTrack.

Outsourced example since gUM is not friend with StackSnippets.

Now I'm not sure how to override methods from web-extensions, you'll have to learn that by yourself, and beware this script is really just a proof of concept and not ready for production. I didn't put any though in handling anything than the demo case.

Upvotes: 1

Related Questions