How do I record musical notes as they are played with the Web Audio and Media Recorder api's?

Question

I'm trying to create a little app that records sounds from audio nodes as they are played using the Web Audio API, but have run into a few problems.

I have two buttons ('start recording' & 'stop recording') a media recorder and a keydown event listener attached to the window.

I don't want to record through the device's speaker but directly from audioNodes

What I want to happen:

1. When I click the 'start recording' button, the media player should start recording (which it does)
2. When I press key "1" the function playNote should play a sound.
3. When I click the 'stop recording' button, the media player should stop recording.
4. When I click the 'play' button on the media player, it should play back the note, or notes that have been played.

What's actually happening so far:

Scenario 1:

Coded so the media player starts to record when the 'start recording' button is clicked

1. I press 'start recording' and the media player starts recording.
2. I press key "1" and the playNote function does not run, I can hear nothing through the speakers.
3. I click 'stop recording' and the media player stops recording.
4. I click the play button on the media player and the media plays back a silent recording that lasts for the duration of the recorded session.

Result: The media recorder didn't record my sound!

Scenario 2:

Coded so the media recorder starts recording when the playNote function runs:

1. I press key "1" and the note plays and the media recorder starts recording.
2. I click the 'stop recording' button and the media player stops recording.
3. I click the play button on the media player and the recording is played back.

My problem with this second scenario is that I want to play more than one note/sound with successive keystrokes throughout the duration of the recording period so, starting the startRecording function in my playNote function does not achieve this, because as soon as key "1" is pressed again, a new sound and new recording session are started.

So what I actually want to happen is this:

Scenario 3:

Coded so the media recorder starts recording when the 'start recording' button is clicked:

1. I click the 'start recording' button and the media player starts a recording session.
2. I press key "1" a few times, the notes played and recorded
3. I click the 'stop recording' button and the media player stops recording.
4. I click the play button on the media player and the recording is played back with all notes present

If anyone can help me solve this problem, I would be very grateful. The only way I can imagine getting this to work at the moment is to somehow create a Blob of each sound played and then chain them all together by pushing them into an array to play back the final recording.

What I'm wondering is, is there a way to inject (for want of another term) outputs from audioNodes into an already running media stream? It would make life so much simpler! :-)

The code I have so far is below. If you comment out this line:

startRecording(); // If this isn't here, the note doesn't play when the media player has been told to start recording

You can see how Scenario 1 pans out, as described above.

Thanks for taking the time to look at this.

// Variables
let audioTag = document.getElementById("audioTag"),
  started = false,
  stopped,
  startBtn = document.getElementById("startBtn"),
  stopBtn = document.getElementById("stopBtn"),
  actx,
  recorder = false,
  recordingStream = false,
  mainVol;

// Define the global context, recording stream & gain nodes
actx = new AudioContext();

recordingStream = actx.createMediaStreamDestination();

mainVol = actx.createGain();
mainVol.gain.value = 0.1;

// Connect the main gain node to the recording stream and speakers
mainVol.connect(recordingStream);
mainVol.connect(actx.destination);

// Function to run when we want to start recording
function startRecording() {
  recorder = new MediaRecorder(recordingStream.stream);
  recorder.start();
  started = true;
}

// Function to run when we want to terminate the recording
function stopRecording() {
  recorder.ondataavailable = function(e) {
    audioTag.src = URL.createObjectURL(e.data);

    recorder = false;
    stopped = true;
  };
  recorder.stop();
}

// Event listeners attached to the start and stop buttons
startBtn.addEventListener(
  "click",
  (e) => {
    e.target.disabled = true;
    console.log("start button clicked");
    startRecording();
  },
  false
);

stopBtn.addEventListener("click", (e) => {
  console.log("stop button clicked");
  startBtn.disabled = false;
  stopRecording();
});

// A function to play a note
function playNote(freq, decay = 1, type = "sine") {
  let osc = actx.createOscillator();
  osc.frequency.value = freq;

  osc.connect(mainVol); // connect to stream destination via main gain node
  startRecording(); // // If this isn't here, the note doesn't play when the media player has been told to start recording
  console.log(mainVol);
  osc.start(actx.currentTime);
  osc.stop(actx.currentTime + 1);
}

// keydown evennt listener attached to the window
window.addEventListener("keydown", keyDownHandler, false);

// The keydown handler
function keyDownHandler(e) {
  if (e.key === "1") {
    console.log(e.key, "pressed");
    playNote(440);
  }
}

<p>
  <button id="startBtn">Start Recording</button>
  <button id="stopBtn">Stop Recording</button>
</p>
<audio id="audioTag" controls="true"></audio>

Answer 1

OK! After a few days of unsuccessful research and hair loss, I had a thought! I have them occasionally ::oP

If we look at the code above we can see that we are creating a new MediaRecorder every time the startRecording function is called:

function startRecording() {
  recorder = new MediaRecorder(recordingStream.stream);
  recorder.start();
  started = true;
}

So I thought:

pull the recorder constructor out into the global scope

get rid of the recorder = false when stopRecording is called, so we can record again using the same MediaRecorder again if we wish:

function stopRecording() {
  recorder.ondataavailable = function(e) {
    audioTag.src = URL.createObjectURL(e.data);

    //recorder = false;
    stopped = true;
  };
  recorder.stop();
}

Then, in our playNote function, add a conditional statement to only start the MediaRecorder if it is not already recording:

function playNote(freq, decay = 1, type = "sine") {
  let osc = actx.createOscillator();
  osc.frequency.value = freq;

  osc.connect(mainVol); // connect to stream destination via main gain node

  // Only start the media recorder if it is not already recording
  if (recorder.state !== "recording") {
    startRecording();
  } // If this isn't here, the note doesn't play
  console.log(mainVol);
  osc.start(actx.currentTime);
  osc.stop(actx.currentTime + decay);
}

This works :-)

I also removed the startBtn and its event listener so that we don't accidentally press it and overwrite our recording.

And just for fun, added a new note to our keyDownHandler

function keyDownHandler(e) {
  if (e.key === "1") {
    console.log(e.key, "pressed");
    playNote(440);
  }

  if (e.key === "2") {
    playNote(600);
  }
}

The final result is that we can now play notes repeatedly, stop recording when we are finished by clicking the stopBtn and then play back the recording by clicking the play button on the media recorder.

Here's a working snippet:

// Variables
let audioTag = document.getElementById("audioTag"),
  started = false,
  stopped,
  //   startBtn = document.getElementById("startBtn"),
  stopBtn = document.getElementById("stopBtn"),
  actx,
  recorder = false,
  streamDest = false,
  mainVol;

// Define the global context, recording stream & gain nodes
actx = new AudioContext();

streamDest = actx.createMediaStreamDestination();

mainVol = actx.createGain();
mainVol.gain.value = 1;

// Create a new MediaRecorder and attached it to our stream
recorder = new MediaRecorder(streamDest.stream);

// Connect the main gain node to the recording stream and speakers
mainVol.connect(streamDest);
mainVol.connect(actx.destination);

// Function to run when we want to start recording
function startRecording() {
  recorder.start();
  started = true;

  console.log(recorder.state);
}

// Function to run when we want to terminate the recording
function stopRecording() {
  recorder.ondataavailable = function(e) {
    audioTag.src = URL.createObjectURL(e.data);

    // recorder = false;
    stopped = true;
  };
  recorder.stop();
}

// Event listeners attached to the start and stop buttons
// startBtn.addEventListener(
//   "click",
//   (e) => {
//     e.target.disabled = true;
//     console.log("start button clicked");
//     startRecording();
//   },
//   false
// );

stopBtn.addEventListener("click", (e) => {
  console.log("stop button clicked");
  //   startBtn.disabled = false;
  stopRecording();
});

// A function to play a note
function playNote(freq, decay = 1, type = "sine") {
  let osc = actx.createOscillator();
  osc.frequency.value = freq;

  osc.connect(mainVol); // connect to stream destination via main gain node

  // Only start the media recorder if it is not already recording
  if (recorder.state !== "recording") {
    startRecording();
  }
  osc.start(actx.currentTime);
  osc.stop(actx.currentTime + decay);
}

// keydown evennt listener attached to the window
window.addEventListener("keydown", keyDownHandler, false);

// The keydown handler
function keyDownHandler(e) {
  if (e.key === "1") {
    console.log(e.key, "pressed");
    playNote(440);
  }

  if (e.key === "2") {
    playNote(600);
  }
}

* {
margin: 0;
padding: 4px;
}

<h6>Keys #1 & #2 play sounds</h6>
<p>Recording starts when the first key is pressed</p>
<h6>Press the 'Stop Recording' button when finished</h6>
<h6>
  Click the play button on the media recorder to play back the recording
</h6>
<p>
  <!-- <button id="startBtn">Start Recording</button> -->
  <button id="stopBtn">Stop Recording</button>
</p>
<audio id="audioTag" controls="true"></audio>

So in the end, all we needed to do was:

create the MediaRecorder in the global scope

Start the MediaRecorder only if it is not already recording.