pyaudio - Faster way to reopen microphone input device on remote desktop

Question

I've been working with pyaudio and recording data from default microphone input, in my PC works fine for a long time period, the thing is that I've been also working in remote desktop using windows rdp file (this file has been configured to use the mic in local resources tab > settings > play on remote computer).

In remote connection is other story, when disconnecting from remote or auto disconnecting by idle timer, I got OsError: [errno -9999] unanticipated host error, I've tried everything, from configuring regedit either group policies, nothing works, so I tried to open again the stream with try/except.

import pyaudio

CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000
RECORD_SECONDS = 1
global stream

p = pyaudio.PyAudio()
def open():
    global stream
    stream = p.open(format=FORMAT,
                    channels=CHANNELS,
                    rate=RATE,
                    input=True,
                    frames_per_buffer=CHUNK,
                    input_device_index=1)

open()
while True:

    frames = []
    try:
        for i in range(0, int(RATE / CHUNK * RECORD_SECONDS)):
            data = stream.read(CHUNK)
            frames.append(data)
    except Exception as e:
        stream.close()
        try:
            stream = p.open(format=FORMAT,
                    channels=CHANNELS,
                    rate=RATE,
                    input=True,
                    frames_per_buffer=CHUNK,
                    input_device_index=1)
        except Exception as e:
            continue

This works for me, but the reconnection is slow, it takes like 2 seconds to start recording again, is there another way to open faster the mic input? (this also can be reproduced on local PC, disconnecting the audio external usb)