How to get amplitude from opus audio?

Question

I use webapi MediaRecorder to capture voice.

var options = {mimeType: "audio/webm;codecs=opus", audioBitsPerSecond:16000};
mediaRecorder = new MediaRecorder(stream, options);
mediaRecorder.addEventListener("dataavailable", function(event) {
    var reader = new FileReader();
    reader.addEventListener("loadend", function() {
        var int8View = new Int8Array(reader.result);
    });
    reader.readAsArrayBuffer(event.data);
});
mediaRecorder.start(200);

so every 200 miliseconds, I will get a blob containing an audio clip. The int8View of that data looks like:

-5,-127,36,84,-128,123,-125,37,35,-109,-94,120,111,-110,40,-93,-7,77,35,-62,83,-36,-12,47,127,47,-75,-35,89,55,65,-75,-106,96,-86,30,118,37,51,-28,-2,-38,124,-95,102,-91,-109,.....

My question is how can I get the amplitude on each sampling point or an average amplitude of this clip? My intention is to use it to detect voice vs silence.

The clip is encoded in opus so I assume adding up the absolute value of each number won't work, right?

Thanks!

Answer 1

As you said you can't just absolute value the array to get the amplitude, since the values will still be in compressed opus format. So in my mind there are two steps:

1. Decode your opus audio

This can be done using many libraries im sure, but here is just one I found : opus-to-pcm. This suggests either using their library, or the Web-Audio API to decode opus. An example with their lib:

var decoder = new Decoder.OpusToPCM({
  channels: 1,
  fallback: true 
});
decoder.on('decode', function(pcmData) {
     //do whatever you want to do with PCM data
});

// single opus packet and it is a typedArray
decoder.decode(opus_packet); 

Seems simple to use! Although I haven't had the chance to use it myself yet.

2. Get your Amplitude

You mentioned how would you get the amplitude for the entire clip / the individual samples, but to get the entire clip you really need the individual samples (at least for the calculation).

Once you have decoded the audio, to find an individual sample's linear amplitude simply take the absolute value of the value in the index wanted of the buffer:

var sampleAmplitude = Math.abs(buffer[index]);

More commonly used, the average across a whole block is taken using the RMS (Root Mean Squared) Value.

var rms = 0;

for (var i = 0; i < buffer.length; i++) {
  rms += buffer[i] * buffer[i];
}

rms /= buffer.length;
rms = Math.sqrt(rms);

This iterates through the entire buffer and sums all of the squared values of the buffer. The average is then taken by dividing it by the length of samples, and finally the squareroot is taken.

Both of the calculations I mentioned return the value to you in linear terms (i.e. between 0 and 1), but in audio we tend to use dB (Decibels). To convert the linear terms you have calculated simply use:

var dBAmplitude = 20*Math.log10(linAmplitude);

Usually the RMS value is used in place of the linAmplitude in this equation.

3. Sample Program

This is an example of the code that you were looking for with comments (using opus-to-pcm. Please note that this is not the ideal way to do it as there is no need to encode the audio to opus in the first place (see this link for a clear tutorial on skipping the opus format all together)! also this example will create readers every time you decode audio, but I think this explains your particular problem's solution clearer. Also, according to the opus docs the audio is in int16 format (I changed your array type).

// Calculate RMS of block (Linear)
function calcrms_lin(buffer){

    var rms = 0;

    for(var bufferIndex = 0; bufferIndex < buffer.length; bufferIndex++){
        rms+= buffer[bufferIndex]*buffer[bufferIndex];
    }

    rms /= buffer.length;
    rms = Math.sqrt(rms);

    return rms;

}

// Calculate RMS of block db
function calcrms_db(buffer){
    return 20*Math.log10(calcrms_lin(buffer));
}

// Create opus-to-pcm decoder
var decoder = new Decoder.OpusToPCM({
    channels: 1,
    fallback: true 
  });

// Assign function to decode callback
decoder.on('decode', function(pcmData) {

    // Get amplitude of entire block rms (in dB) everytime its decoded
    var dBAmplitude = calcrms_db(pcmData);

    // Do what you want with the dBAmplitude variable e.g. display it to the screen or whatever

});

// Create options for media recorder
var options = {mimeType: "audio/webm;codecs=opus", audioBitsPerSecond:16000};

// Construct media recorder
mediaRecorder = new MediaRecorder(stream, options);

// Add callback for when data available from recorder
mediaRecorder.addEventListener("dataavailable", function(event) {

    // New file 
    var reader = new FileReader();

    // Assign callback
    reader.onload = function(){
        var audioBuffer = new Int16Array(reader.result);
        decoder.decode(audioBuffer);
    }

    // Read data into file reader (will start the onload function above)
    reader.readAsArrayBuffer(event.data);

});

// Start media recorder process
mediaRecorder.start(200);

Note: this code is untested and should just serve as an example

Extra

If you are unsure about this stuff maybe its worth checking out general theory of JS callbacks more, since audio especially tends to be very callback based as client side JS is 'single threaded'.

I'm not entirely sure if this is the process you want to follow but if you need the audio in opus format then fair enough. If you just wanted to record audio to display the data however, I would definately be looking at checking out the Web-Audio API - Here's an example to get you started (it has built in audio FX, and is really easy for visualizing audio data!).