Nginx reverse proxy makes NAudio play white noise audio (with occasional normal audio inbetween)

Question

I'm currently trying to switch from direct IP access to my tts tool to a nginx controlled domain access.

All works as intended. Except that my Naudio in my program doesn't play the audio errorfree anymore.

The code I'm using didn't change and calling the streaming function from URL directly results in an errorfree audio. Even saving the stream locally AFTER playing it with whitenoise is still a perfect audiofile. So my big question. Does Nginx/domain access change anything in the response of this request or do you know of any way to verify that myself?

    HttpClient httpClient = new HttpClient();
    httpClient.BaseAddress = new Uri(data.BaseUrl);
    httpClient.Timeout = TimeSpan.FromSeconds(2);

    HttpResponseMessage res = null;
    try
    {
        var uriBuilder = new UriBuilder(data.BaseUrl);
        uriBuilder.Path = data.StreamPath;
        var query = HttpUtility.ParseQueryString(uriBuilder.Query);
        query["text"] = voiceLine;
        query["voice"] = voice;
        query["language"] = getAlltalkLanguage(language);
        query["output_file"] = "ignoreme.wav";
        uriBuilder.Query = query.ToString();
        LogHelper.Debug(MethodBase.GetCurrentMethod().Name, $"Requesting... {uriBuilder.Uri}", eventId);
        using var req = new HttpRequestMessage(HttpMethod.Get, uriBuilder.Uri);

        res = await httpClient.SendAsync(req, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
        EnsureSuccessStatusCode(res);

        // Copy the sound to a new buffer and enqueue it
        LogHelper.Info(MethodBase.GetCurrentMethod().Name, "Getting response...", eventId);
        var responseStream = await res.Content.ReadAsStreamAsync().ConfigureAwait(false);

        LogHelper.Info(MethodBase.GetCurrentMethod().Name, "Done", eventId);
        return responseStream ;
    }
    catch (Exception ex)
    {
        LogHelper.Error(MethodBase.GetCurrentMethod().Name, ex.ToString(), eventId);
    }

The code I'm using to play the stream:

    CurrentlyPlayingStream = queueItem;
    var stream = new RawSourceWaveStream(queueItem, new NAudio.Wave.WaveFormat(24000, 16, 1));
    LogHelper.Info(MethodBase.GetCurrentMethod().Name, "Playing next queue item", eventId);
    var volumeSampleProvider = new VolumeSampleProvider(stream.ToSampleProvider());
    volumeSampleProvider.Volume = Volume;

    ActivePlayer = new WasapiOut(AudioClientShareMode.Shared, 0);
    ActivePlayer.PlaybackStopped += SoundOut_PlaybackStopped;
    ActivePlayer.Init(volumeSampleProvider);
    ActivePlayer.Play();

EDIT 1: I'm using the Nginx Proxy Manager (NPM) in a docker Environment with these settings: Service: click me SSL/HTTPS: click me

No custom configuration.

What I tried: Disabling Nginx/Accessing via Domain:Port to check if the domain's the problem. It's not.

gzip off; <- Read somewhere this can cause problems.

Disabling SSL <- No change

EDIT 2: After hours of trying around I'm getting the feeling that some kind of data packets aren't delivered the way Naudio expects them for playing directly from stream but are fine if saved via naudio.

Could it be that Nginx is too slow to deliver sometimes? (the playing audio is randomly filled with white noise, different each time)