Issue accessing HoloLens 2 audio data using Microsoft WindowsMicrophoneStream

Question

I am trying to access the raw (float[]) values of the HoloLens 2's embedded microphone in real time. I do not need to record this data or playback this data, purely to sample is the user is speaking at any given slice in time, as recorded by the HL2. I am using the MicrophoneAmplitudeDemo.cs demo here nearly verbatim. I attach this script to a Unity GameObject, and have modified the script only to print the average amplitude every update, purely as a way to debug the output. When the script runs, the returned float values are always 0. I have already double checked the permissions for the microphone in the manifest and the initial popup permission windows are answered "yes". The code, modified from the original MS sample only to print the average amplitude, is below.

TO try to fix the problem, I have already diabled all other functionality in this program (eye tracking, onboard ML inference, etc) to ensure that wasn't the problem. I have also tried another MS sample (MicStreamDemo) with the exact same result. The debugging window throws no errors, but merely print zeros when I print the current values of the mic stream.

using System.Collections; using System.Collections.Generic; using UnityEngine;

[RequireComponent(typeof(AudioSource))]
public class AudioCaptureUtility : MonoBehaviour
{


[SerializeField]
[Tooltip("Gain to apply to the microphone input.")]
[Range(0, 10)]
private float inputGain = 1.0f;

[SerializeField]
[Tooltip("Stream Type")]
public WindowsMicrophoneStreamType streamType= WindowsMicrophoneStreamType.HighQualityVoice;

/// <summary>
/// Class providing microphone stream management support on Microsoft Windows based devices.
/// </summary>
private WindowsMicrophoneStream micStream = null;

/// <summary>
/// The average amplitude of the sound captured during the most recent microphone update.
/// </summary>
private float averageAmplitude = 0.0f;

private void Awake()
{
    // We do not wish to play the ambient room sound from the audio source.
    //gameObject.GetComponent<AudioSource>().volume = 0.0f;

    micStream = new WindowsMicrophoneStream();
    if (micStream == null)
    {
        Debug.Log("Failed to create the Windows Microphone Stream object");
    }

    micStream.Gain = inputGain;

    // Initialize the microphone stream.
    WindowsMicrophoneStreamErrorCode result = micStream.Initialize(streamType);
    if (result != WindowsMicrophoneStreamErrorCode.Success)
    {
        Debug.Log($"Failed to initialize the microphone stream. {result}");
        return;
    }

    // Start the microphone stream.
    // Do not keep the data and do not preview.
    result = micStream.StartStream(false, false);
    if (result != WindowsMicrophoneStreamErrorCode.Success)
    {
        Debug.Log($"Failed to start the microphone stream. {result}");
    }
}

private void OnDestroy()
{
    if (micStream == null) { return; }

    // Stop the microphone stream.
    WindowsMicrophoneStreamErrorCode result = micStream.StopStream();
    if (result != WindowsMicrophoneStreamErrorCode.Success)
    {
        Debug.Log($"Failed to stop the microphone stream. {result}");
    }

    // Uninitialize the microphone stream.
    micStream.Uninitialize();
    micStream = null;

}

// Start is called before the first frame update
void Start()
{
    
}

// Update is called once per frame
private void Update()
{
    if (micStream == null) { return; }

    // Update the gain, if changed.
    if (micStream.Gain != inputGain)
    {
        micStream.Gain = inputGain;
    }
    float[] tempBuffer = new float[5];
    OnAudioFilterRead(tempBuffer, 2);


    if(averageAmplitude == 0.0f)
    {
        Debug.Log("Average Amp is Zero");
        //Debug.Log(averageAmplitude.ToString("F9"));
    }
    

}

private void OnAudioFilterRead(float[] buffer, int numChannels)
{
    if (micStream == null) { return; }

    // Read the microphone stream data.
    WindowsMicrophoneStreamErrorCode result = micStream.ReadAudioFrame(buffer, numChannels);
    if (result != WindowsMicrophoneStreamErrorCode.Success)
    {
        Debug.Log($"Failed to read the microphone stream data. {result}");
    }

    float sumOfValues = 0;

    // Calculate this frame's average amplitude.
    for (int i = 0; i < buffer.Length; i++)
    {
        if (float.IsNaN(buffer[i]))
        {
            buffer[i] = 0;
        }

        buffer[i] = Mathf.Clamp(buffer[i], -1.0f, 1.0f);
        sumOfValues += Mathf.Clamp01(Mathf.Abs(buffer[i]));
    }

    averageAmplitude = sumOfValues / buffer.Length;
}

}

EDIT: The pictures below are screenshots of the errors. I was able to get some raw float data printed, but the data stream ends during initialization each time. I simply print the current value of averageAmplitude each Update(). The InitializeFrameReader message are from a Windows MediaCapture instance. To ensure that this isn't the culprit, I remove this functionality and the issues remain. The float values cease and never return. I have waited as long as 5 minutes to ensure they never did come back.

Answer 1

I run into an issue after a few tests, and I am not sure if it is the same issue you have. When initialization of micStream, it sometimes returns 'Already Running', and OnAudioFilterRead() will return 'NotEnoughData'. Then I made some modifications of micStream initialization to solve this issue. You can refer to it.

WindowsMicrophoneStreamErrorCode result = micStream.Initialize(WindowsMicrophoneStreamType.HighQualityVoice); 
if (result != WindowsMicrophoneStreamErrorCode.Success && result!=WindowsMicrophoneStreamErrorCode.AlreadyRunning)
{
    Debug.Log($"Failed to initialize the microphone stream. {result}");
    return;
}

Also, you can get some log information on Windows Device Portal to troubleshoot the issue. Or you can try remote debugging on Unity.