How to record audio (mic+tab audio) from Google Meet with Chrome Extension?

Question

I want to record a Google Meet meeting with a Browser Extension, but any tab audio, such as YouTube, can be recorded, including my mic.

I know how to record the tab audio.

I don't know how to combine the tab audio with the mic to record the GMeet fully. Include functionality that respects muting the microphone on GMeet.

My current code is on GitHub: https://github.com/prokopsimek/chrome-extension-recording

What's already implemented:

• record tab audio
• opens a new tab after the audio is stopped with the audio file

my offscreen.tsx:

    const media = await navigator.mediaDevices.getUserMedia({
      audio: {
        mandatory: {
          chromeMediaSource: 'tab',
          chromeMediaSourceId: streamId,
        },
      },
      video: false,
    } as any);
    console.error('OFFSCREEN media', media);

    // FIXME: this causes error in recording, stops recording the offscreen
    // const micMedia = await navigator.mediaDevices.getUserMedia({
    //   audio: {
    //     mandatory: {
    //       chromeMediaSource: 'tab',
    //       chromeMediaSourceId: micStreamId,
    //     },
    //   },
    //   video: false,
    // } as any);

    // Continue to play the captured audio to the user.
    const output = new AudioContext();
    const source = output.createMediaStreamSource(media);

    const destination = output.createMediaStreamDestination();
    // const micSource = output.createMediaStreamSource(micMedia);

    source.connect(output.destination);
    source.connect(destination);
    // micSource.connect(destination);
    console.error('OFFSCREEN output', output);

    // Start recording.
    recorder = new MediaRecorder(destination.stream, { mimeType: 'video/webm' });
    recorder.ondataavailable = (event: any) => data.push(event.data);
    recorder.onstop = async () => {
      const blob = new Blob(data, { type: 'video/webm' });

      // delete local state of recording
      chrome.runtime.sendMessage({
        action: 'set-recording',
        recording: false,
      });

      window.open(URL.createObjectURL(blob), '_blank');

my popup.tsx useEffect:

  const handleRecordClick = () => {
    if (isRecording) {
      console.log('Attemping to stop recording');
      chrome.tabs.query({ active: true, currentWindow: true }, (tabs) => {
        const currentTab = tabs[0];
        if (currentTab.id) {
          chrome.runtime.sendMessage({
            action: 'stopRecording',
            tabId: currentTab.id,
          });
          setIsRecording(false);
        }
      });
    } else {
      chrome.tabs.query({ active: true, currentWindow: true }, (tabs) => {
        const currentTab = tabs[0];
        if (currentTab.id) {
          chrome.runtime.sendMessage({
            action: 'startRecording',
            tabId: currentTab.id,
          });
          setIsRecording(true);
        }
      });
    }
  };

my background.ts initializing offscreen:

const startRecordingOffscreen = async (tabId: number) => {
  const existingContexts = await chrome.runtime.getContexts({});
  let recording = false;

  const offscreenDocument = existingContexts.find((c) => c.contextType === 'OFFSCREEN_DOCUMENT');

  // If an offscreen document is not already open, create one.
  if (!offscreenDocument) {
    console.error('OFFSCREEN no offscreen document');
    // Create an offscreen document.
    await chrome.offscreen.createDocument({
      url: 'pages/offscreen/index.html',
      reasons: [chrome.offscreen.Reason.USER_MEDIA, chrome.offscreen.Reason.DISPLAY_MEDIA],
      justification: 'Recording from chrome.tabCapture API',
    });
  } else {
    recording = offscreenDocument.documentUrl?.endsWith('#recording') ?? false;
  }

  if (recording) {
    chrome.runtime.sendMessage({
      type: 'stop-recording',
      target: 'offscreen',
    });
    chrome.action.setIcon({ path: 'icons/not-recording.png' });
    return;
  }

  // Get a MediaStream for the active tab.
  console.error('BACKGROUND getMediaStreamId');

  const streamId = await new Promise<string>((resolve) => {
    // chrome.tabCapture.getMediaStreamId({ consumerTabId: tabId }, (streamId) => {
    chrome.tabCapture.getMediaStreamId({ targetTabId: tabId }, (streamId) => {
      resolve(streamId);
    });
  });
  console.error('BACKGROUND streamId', streamId);

  const micStreamId = await new Promise<string>((resolve) => {
    chrome.tabCapture.getMediaStreamId({ consumerTabId: tabId }, (streamId) => {
      resolve(streamId);
    });
  });
  console.error('BACKGROUND micStreamId', micStreamId);

  // Send the stream ID to the offscreen document to start recording.
  chrome.runtime.sendMessage({
    type: 'start-recording',
    target: 'offscreen',
    data: streamId,
    micStreamId,
  });

  chrome.action.setIcon({ path: '/icons/recording.png' });
};

What's missing:

• record a mic
• combine mic with tab audio
• respect when the mic is on/off in GMeet

What I am confused about:

1. Should I really use the offscreen document?

• I am getting permissions denied in the offscreen (ref)

2. What to use for the recording? Offscreen, content script, popup, or anything else?
3. How to combine the audio streams from both sources into one file?

My goal:

• Record audio from Google Meets (tab + my mic)
• Record audio only
• Use the v3 manifest (I expect that the older versions will be deprecated sooner than v3)
• I should allow the mic in a tab where I want to start the recording if not allowed yet
• When I stop recording, a new tab will open with the blob file so that it can be saved

Reference links:

• https://github.com/muaz-khan/Chrome-Extensions/issues/75 (doesn't work)
• https://github.com/muaz-khan/Chrome-Extensions/tree/master/screen-recording (manifest v2 but I'd like v3 - do I really need the v3? 🤷)
• https://groups.google.com/a/chromium.org/g/chromium-extensions/c/V09VMCLzvWM
• https://github.com/GoodDollar/GoodDAPP/issues/1727
• https://developer.mozilla.org/en-US/docs/Web/API/MediaStream
• https://developer.chrome.com/docs/extensions/reference/api/tabCapture#method-capture
• https://developer.chrome.com/docs/extensions/reference/api/tabCapture#method-getMediaStreamId
• Chrome extension include mic audio to recorded video
• How to record audio from a meeting tab in google chrome? (no response)

Answer 1

The answer is quite simple but it took me a while to find:

1. First request audio recording permissions from another extensions page that is not the offscreen, which can be popup, options, etc.

const audioStream = await navigator.mediaDevices.getUserMedia({
  audio: true,
  video: false,
});

A popup will pop up asking "(Extension Name) wants to use your microphone." Once the user grants the permission, you're good to request a userMedia audio stream in the offscreen document.

You can then use display media and user media as you will inside an offscreen document. Here is an example of me recording both screen capture, screen audio, and mic audio:

async function getRecordingStream() {
  // 1. ask user to record their screen and possibly audio from a tab
  const recorderStream = await navigator.mediaDevices.getDisplayMedia({
        audio: true,
        video: true,
      });

  // 2. ask user to record their mic. Permission should already be granted
  // if you ran this exact code before in another process, like popup or options - NOT content script
  const audioStream = await navigator.mediaDevices.getUserMedia({
    audio: true,
    video: false,
  });

  // if recording window (no system audio), then just join with mic.
  if (recorderStream.getAudioTracks().length === 0) {
        const combinedStream = new MediaStream([
          ...recorderStream.getVideoTracks(),
          ...audioStream.getAudioTracks(),
        ]);
        return combinedStream;
  }
  
  // else if recording tab audio, you need to merge both mic and system audio
  // into a single track (if you want to)
        const audioContext = new AudioContext();
      const destination = audioContext.createMediaStreamDestination();

      // Add tab audio to the destination
      const tabAudioSource =
        audioContext.createMediaStreamSource(recorderStream);
      tabAudioSource.connect(destination);

      // Add mic audio to the destination
      const micAudioSource = audioContext.createMediaStreamSource(audioStream);
      micAudioSource.connect(destination);

      const combinedStream = new MediaStream([
        ...recorderStream.getVideoTracks(),
        ...destination.stream.getTracks(),
      ]);

      return combinedStream;
  
}

If you want you can just use this giant ass class I made lol:

interface StartRecording {
  onStop?: () => void;
  onRecordingCanceled?: () => void;
}

class RecordingError extends Error {
  constructor(message: string, public stream: MediaStream) {
    super(message);
    this.name = "RecordingError";
  }

  log() {
    console.error(this.name, this.message);
    console.error("The offending stream", this.stream);
    console.error(this.stack);
  }
}

class MicNotEnabledError extends RecordingError {
  constructor(stream: MediaStream) {
    super("Mic not enabled", stream);
    this.name = "MicNotEnabledError";
  }
}

export class ScreenRecorder {
  stream?: MediaStream;
  recorder?: MediaRecorder;
  recorderStream?: MediaStream;
  micStream?: MediaStream;

  static async checkMicPermission() {
    const result = await navigator.permissions.query({
      name: "microphone" as PermissionName,
    });
    return result.state;
  }

  private async getStream({ recordMic }: { recordMic: boolean }) {
    const recorderStream = await navigator.mediaDevices.getDisplayMedia({
      audio: true,
      video: true,
    });
    this.recorderStream = recorderStream;

    // if video ends, audio should too.
    this.recorderStream.getTracks()[0].addEventListener("ended", async () => {
      await this.stopRecording();
    });

    // if recording window (no system audio), then just join with mic.
    if (recorderStream.getAudioTracks().length === 0) {
      const audioStream = await navigator.mediaDevices.getUserMedia({
        audio: true,
        video: false,
      });
      const combinedStream = new MediaStream([
        ...recorderStream.getVideoTracks(),
        ...audioStream.getAudioTracks(),
      ]);
      return combinedStream;
    }
    if (recordMic) {
      const audioStream = await navigator.mediaDevices.getUserMedia({
        audio: true,
        video: false,
      });
      if (audioStream.getAudioTracks().length === 0) {
        throw new MicNotEnabledError(audioStream);
      }

      const audioContext = new AudioContext();
      const destination = audioContext.createMediaStreamDestination();

      // Add tab audio to the destination
      const tabAudioSource =
        audioContext.createMediaStreamSource(recorderStream);
      tabAudioSource.connect(destination);

      // Add mic audio to the destination
      const micAudioSource = audioContext.createMediaStreamSource(audioStream);
      micAudioSource.connect(destination);

      const combinedStream = new MediaStream([
        ...recorderStream.getVideoTracks(),
        ...destination.stream.getTracks(),
      ]);

      return combinedStream;
    } else {
      return recorderStream;
    }
  }

  async startAudioRecording(options?: StartRecording) {
    if (this.recorder) {
      this.recorder.stop();
    }
    let audioStream: MediaStream;
    try {
      audioStream = await navigator.mediaDevices.getUserMedia({
        audio: true,
        video: false,
      });
    } catch (e) {
      if (e instanceof DOMException) {
        options?.onRecordingCanceled?.();
        return false;
      }
    }
    this.stream = audioStream;
    this.recorder = new MediaRecorder(this.stream);

    // Start recording.
    this.recorder.start();
    this.recorder.addEventListener("dataavailable", async (event) => {
      let recordedBlob = event.data;
      let url = URL.createObjectURL(recordedBlob);

      let a = document.createElement("a");

      a.style.display = "none";
      a.href = url;
      a.download = "audio-recording.webm";

      document.body.appendChild(a);
      a.click();

      document.body.removeChild(a);

      URL.revokeObjectURL(url);

      options?.onStop?.();
    });
    return true;
  }

  async startVideoRecording({
    onStop,
    recordMic = false,
    onRecordingCanceled,
    onRecordingFailed,
  }: {
    onStop?: () => void;
    recordMic?: boolean;
    onRecordingCanceled?: () => void;
    onRecordingFailed?: () => void;
  }) {
    if (this.recorder) {
      this.recorder.stop();
    }
    try {
      this.stream = await this.getStream({
        recordMic,
      });
      console.log("stream", this.stream);
    } catch (e) {
      if (e instanceof DOMException) {
        console.warn("Permission denied: user canceled recording");
        onRecordingCanceled?.();
        return false;
      } else if (e instanceof RecordingError) {
        e.log();
        onRecordingFailed?.();
        return false;
      } else {
        console.error(e);
        onRecordingFailed?.();
        return false;
      }
    }
    this.recorder = new MediaRecorder(this.stream, {
      mimeType: "video/webm;codecs=vp9,opus",
    });

    // Start recording.
    this.recorder.start();
    this.recorder.addEventListener("dataavailable", async (event) => {
      let recordedBlob = event.data;
      let url = URL.createObjectURL(recordedBlob);

      let a = document.createElement("a");

      a.style.display = "none";
      a.href = url;
      a.download = "screen-recording.webm";

      document.body.appendChild(a);
      a.click();

      document.body.removeChild(a);

      URL.revokeObjectURL(url);
      onStop && onStop();
    });
    return true;
  }

  async isRecording() {
    return Boolean(this.recorder && this.recorder.state === "recording");
  }

  /**
   * For programmatically stopping the recording.
   */
  async stopRecording() {
    console.log("stopping recording");
    console.log("combined stream tracks", this.stream.getTracks());
    this.stream.getTracks().forEach((track) => track.stop());
    // this.micStream?.getTracks().forEach((track) => track.stop());
    // this.recorderStream?.getTracks().forEach((track) => track.stop());
    this.recorder.stop();
    this.recorder = undefined;

    // this.micStream = undefined;
    // this.recorderStream = undefined;
  }
}

Answer 2

I will attempt to address the remaining requirements individually

1. Capture the audio from specific tab.

    const media = await navigator.mediaDevices.getUserMedia({
     audio: {
       mandatory: {
         chromeMediaSource: 'tab',
         chromeMediaSourceId: streamId,
       },
     },
     video: false,
   } as any);
   

Let's assume we have created a separate function named getTabAudioStream that returns the above.

2. • record a mic

   return await navigator.mediaDevices.getUserMedia({
       audio: true,  // Request microphone audio
       video: false,
   });   
   

Let's assume we have created a separate function named getMicrophoneAudioStream that returns the above.

3. • combine mic with tab audio

   async function setupAudioCombination(tabStreamId) {
     const tabStream = await getTabAudioStream(tabStreamId);
     const micStream = await getMicrophoneAudioStream();
   
     const audioContext = new AudioContext();
     const tabSource = audioContext.createMediaStreamSource(tabStream);
     const micSource = audioContext.createMediaStreamSource(micStream);
     const destination = audioContext.createMediaStreamDestination();
     const micGainNode = audioContext.createGain(); // Create a GainNode to control mic volume when mute state updates
   
   
     // Connect both sources to the destination
     tabSource.connect(destination);
     micSource.connect(micGainNode); // Connect micSource to the GainNode
     micGainNode.connect(destination);
   
     return {
       combinedStream: destination.stream, // record this
       micGainNode,
       audioContext,
     };
   }
   

4. • respect when the mic is on/off in GMeet

   You can track the microphone's "muted" state in Google Meet by selecting the mute button using the [data-mute-button] attribute and setting up a MutationObserver to listen for changes in the data-is-muted attribute. This attribute toggles between "true" and "false" to indicate whether the microphone is muted or unmuted.

   Here's a simple implementation:

   const muteButton = document.querySelector('[data-mute-button]');
   let isMuted = muteButton.getAttribute('data-is-muted') === 'true';
   
   // Create a MutationObserver to watch for changes in the `data-is-muted` attribute
   const observer = new MutationObserver((mutationsList) => {
     for (const mutation of mutationsList) {
       if (mutation.type === 'attributes' && mutation.attributeName === 'data-is-muted') {
         isMuted = muteButton.getAttribute('data-is-muted') === 'true';
         console.log('Microphone mute state changed:', isMuted ? 'Muted' : 'Unmuted');
          handleMicMuteState(isMuted);
       }
     }
   });
   
   // Start observing the mute button for attribute changes
   observer.observe(muteButton, { attributes: true, attributeFilter: ['data-is-muted'] });
   

   where handleMicMuteState mutes the mic audio stream alone through the micGainNode and the recording of the tab audio continues:

   function handleMicMuteState(isMuted) {
     if (micGainNode) {
       micGainNode.gain.value = isMuted ? 0 : 1; // Set gain to 0 to mute, 1 to unmute
     }
   }