XAudio2 - Source voice hangs when active audio device gets removed

Question

I have a problem I am not able to solve. My application should be able to switch the default audio device during runtime. To achieve this I am using XAudio2 from the DirectXTK.

I implemented the IMMNotificationClient into my audio class to be able to react on default device changes.

For instance when the default device changes I stop my current source voice, reset the audio engine and start the source voice again. Everything works as expected.

However when my default device is a USB soundcard and I unplug it during the source voice is playing the application freezes.

The reason for this is that the source voice hangs when stopping the voice. Sometimes also when flushing the source buffer. Seems like the source voice could not be stopped anymore when the audio device was removed the source voice was using.

Does someone had the same problem and was able to solve this?

Here's the function I am using to reset the audio engine.

HRESULT DX::XAudioEngine::ResetAudioDevice()
{
    HRESULT hr = S_OK;

    this->m_retryAudio = TRUE;

    if (SUCCEEDED(hr) && m_pSourceVoice)
    {
        hr = m_pSourceVoice->Stop();
    }

    if (SUCCEEDED(hr))
    {
        hr = m_pSourceVoice->FlushSourceBuffers();
    }

    if (m_audEngine && m_pSourceVoice)
    {
        // Get the source voice back from the smart pointer
        IXAudio2SourceVoice* ptmpSrcVoice = nullptr;
        ptmpSrcVoice = m_pSourceVoice.release();

        // Destroy the voice
        ptmpSrcVoice->DestroyVoice();
    }

    m_audEngine->Reset(&m_waveFormat, NULL);

    // Create source voice
    IXAudio2SourceVoice* ptmpSrcVoice = nullptr;

    m_audEngine->AllocateVoice(
        (WAVEFORMATEX*)&m_waveFormat,
        SoundEffectInstance_Default,
        false,
        &ptmpSrcVoice
    );

    // Add source voice to smart pointer
    m_pSourceVoice.reset(ptmpSrcVoice);

    // Set the input volume
    if (this->m_inputVolume != 1.0f) {
        hr = m_pSourceVoice->SetVolume(this->m_inputVolume);
    }

    hr = m_pSourceVoice->Start(0);

    this->m_retryAudio = FALSE;

    return hr;
}