CODEWITHSUNDEEP
javascripthtmlaudioweb-audio-api
JVE999
JVE999

Reputation: 3517

Web Audio API: How to use FFT to convert from time domain and use iFFT to convert the data back

I've been trying to convert audio data to frequency domain data, edit that data, and reconstruct the audio from that data.

I followed the instructions of:

  1. OfflineAudioContext in order to get a buffer to perform the analysis on,
  2. AnalyserNode to perform the analysis, and
  3. PeriodicWave to reconstruct the wave.

The audio rendered by the OfflineAudioContext should match the audio of the PeriodicWave, but it clearly doesn't. The instructions say that it should, though, so I clearly am missing something.

(Additionally, I don't know what to use for the real and imaginary value inputs of the PeriodicWave. From the instructions, the real values are sines and the imaginary values are cosines, so I set all of the imaginary values to 0 as I have no cosine values from the FFT analysis from AnalyserNode, and there seemed to be no other way.)

So far the simplest and closest I've gotten is the following script (https://jsfiddle.net/k81w04qv/1/):

<!DOCTYPE html>
<html>

<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
  <meta name="viewport" content="width=device-width">

  <title>Audio Test</title>
  <link rel="stylesheet" href="">
  <!--[if lt IE 9]>
      <script src="//html5shiv.googlecode.com/svn/trunk/html5.js"></script>
    <![endif]-->
</head>

<body>
  <h1>Audio Test</h1>
  <button id='0'>Play original sound</button>
  <button id='1'>Play reconstructed sound</button>
  <pre></pre>
</body>
<script id='script'>
  var pre = document.querySelector('pre');
  var myScript = document.getElementById('script');
  pre.innerHTML = myScript.innerHTML;

  var buttonOriginal = document.getElementById('0');
  var buttonReconstr = document.getElementById('1');
  var audioCtx = new (window.AudioContext || window.webkitAudioContext)();

  var channels = 2;
  var sampleRate = audioCtx.sampleRate;
  var frameCount = sampleRate * 2.0;

  var offlineCtx = new OfflineAudioContext(channels, frameCount, sampleRate);
  var myArrayBuffer = offlineCtx.createBuffer(channels, frameCount, sampleRate);
  var offlineSource = offlineCtx.createBufferSource();

  var analyser = offlineCtx.createAnalyser();

  var pi = Math.PI;
  var songPos = [0, 0];

  for (var channel = 0; channel < channels; channel++) {
    var nowBuffering = myArrayBuffer.getChannelData(channel);
    for (var i = 0; i < frameCount; i++) {
      songPos[channel]++;
      nowBuffering[i] = synth(channel);
    }
  }

  analyser.connect(offlineCtx.destination);
  offlineSource.connect(analyser);
  offlineSource.buffer = myArrayBuffer;
  offlineSource.start();
  offlineCtx.startRendering().then(function (renderedBuffer) {
    console.log('Rendering completed successfully');
    analyser.fftSize = 2048;
    var bufferLength = analyser.frequencyBinCount;
    var dataArray = new Float32Array(bufferLength);

    analyser.getFloatFrequencyData(dataArray);
    console.log(dataArray);
    // Remove -infinity
    for (var i = 0; i < dataArray.length; i++) {
      if(dataArray[i]==-Infinity)
      dataArray[i] = -255;
    }
    /// Reconstruct
    // Create array of zeros
    var imagArray = new Float32Array(bufferLength);
    for (var i = 0; i < imagArray.length; i++) imagArray[i] = 0;

    var wave = audioCtx.createPeriodicWave(dataArray, imagArray, {disableNormalization: true});
    console.log(wave);


    buttonReconstr.onclick = function() {
      var wave = audioCtx.createPeriodicWave(dataArray, imagArray, {disableNormalization: true});
      var osc = audioCtx.createOscillator();
      osc.setPeriodicWave(wave);
      osc.connect(audioCtx.destination);
      osc.start();
      osc.stop(2);
      osc.onended = () => {
        console.log('Reconstructed sound finished');
      }
    }

    buttonOriginal.onclick = function() {
      var song = audioCtx.createBufferSource();
      song.buffer = renderedBuffer;
      song.connect(audioCtx.destination);
      song.start();
      song.onended = () => {
        console.log('Original sound finished');
      }
    }


  })/*.catch(function (err) {
    console.log('Rendering failed: ' + err);
    // Note: The promise should reject when startRendering is called a second time on an OfflineAudioContext
  });*/




  function freqSin(freq, time) {
    return Math.sin(freq * (2 * pi) * time);
  }
  function synth(channel) {
    var time = songPos[channel] / sampleRate;
    switch (channel) {
      case 0:
        var freq = 200 + 10 * freqSin(9, time);;
        var amp = 0.7;
        var output = amp * Math.sin(freq * (2 * pi) * time);
        break;
      case 1:
        var freq = 900 + 10 * freqSin(10, time);
        var amp = 0.7;
        var output = amp * Math.sin(freq * (2 * pi) * time);
        break;
    }
    //console.log(output)
    return output;
  }


</script>

</html>

An interesting side-issue with this script is that after you've played the original sound, you cannot play the reconstructed sound (although you can play the original sound as much as you'd like). In order to play the reconstructed sound, you have to play it first, then it will only play on refresh. (You can also play it after you've played the original sound if you play it while the original sound is playing.)

Upvotes: 1

Views: 444

Answers (1)

Raymond Toy
Raymond Toy

Reputation: 6048

For this to work, you need both the real and imaginary parts of the FFT of the time-domain signal. The AnalyserNode only gives you the magnitude; you're missing the phase component.

Sorry, this isn't going to work.

Upvotes: 1

Related Questions