CODEWITHSUNDEEP
angularspring-bootaudio-streaming
Idorasi Paul
Idorasi Paul

Reputation: 77

How play audio before download is complete?

I have an Angular client and one SpringBoot server. On the server, I have an endpoint that generates an audio file from text. The audio file generation depends on another API that take too long (5 seconds+) and I don't want to make the user wait for more than 5 seconds for the audio to start.

To fix the time issue, I'm returning a Flux as I receive the bytes from the API (supports streaming). Until here, everything is good. Endpoint code:

    @PostMapping(value = "/speech")
    public Flux<DataBuffer> generateSpeechAudio(@RequestBody @Valid SpeechRequest speechRequest) {
        return audioService.generateSpeech(speechRequest);
    }

On the client side (Angular), I want to call the API and start the audio before the byte download finished. For that, I'm using the reportProgress and I observe the events in my client call:

    transcribeTextToAudio(toAudio: ITranscript): Observable<HttpEvent<string>> {
        return this.httpClient.post<string>(`${this.transcriptionLink}/speech`, toAudio, {
            observe: 'events',
            responseType: 'arrayBuffer' as 'json',
            reportProgress: true,
        });
    }

I'm not sure how to get the bytes and "play them" or even if such a thing is possible. Using MediaSource works, but only of I wait for the entire file to download.

Upvotes: 0

Views: 64

Answers (2)

PPFromInfy
PPFromInfy

Reputation: 189

Can you please try this?

First of all, it's best that you have setup your backend to return Flux. It's the best option for streaming audio.

Coming back to the problem, change the return type of transcribeTextToAudio to Observable<ArrayBuffer>. Something like this:

transcribeTextToAudio(toAudio: ITranscript): Observable<ArrayBuffer> {
    return this.httpClient.post(`${this.transcriptionLink}/speech`, toAudio, {
        responseType: 'arraybuffer',
        reportProgress: true,
        observe: 'events'
    }).pipe(
        filter(event => event.type === HttpEventType.DownloadProgress),
        map(event => (event as HttpProgressEvent).partialText as ArrayBuffer)
    );
}

Now in your component, handle this response to play audio:

import { Component } from '@angular/core';
import { Observable } from 'rxjs';
import { filter, map } from 'rxjs/operators';

import { TestService } from './test.service';

@Component({
    selector: 'app-test',
    templateUrl: './test.component.html',
    styleUrls: ['./test.component.css']
})
export class TestComponent {
    private _media: MediaSource;
    private _source: SourceBuffer;
    private _audio: HTMLAudioElement;

    constructor(private _serv: TestService) {
        this._audio = document.createElement('audio');
        this._media = new MediaSource();
        this._audio.src = URL.createObjectURL(this.mediaSource);

        this._media.addEventListener('sourceopen', () => {
            this._source = this._media.addSourceBuffer('audio/mpeg');
        });
    }

    public playAudio(toAudio: ITranscript) {
        this._serv.transcribeTextToAudio(toAudio).subscribe({
            next: (chunk: ArrayBuffer) => {
                if (this._source.updating) {
                    this._source.addEventListener('updateend', () => {
                        this._source.appendBuffer(chunk);
                    }, { once: true });
                } else {
                    this._source.appendBuffer(chunk);
                }
            },
            error: (err) => console.error('Error streaming audio:', err),
            complete: () => {
                this._media.endOfStream();
                this._audio.play();
            }
        });
    }
}

Note: I haven't been able to test this. If it also plays once download has completed, please do let me know. I'll look for a workaround.

Thanks!

Upvotes: 0

Brad
Brad

Reputation: 163528

Using MediaSource is one way, but it only works with a couple codecs and containers, and can be tricky.

A better method is to let the browser handle the playback and streaming. To do this, don't make a POST request, but a GET request instead. Try something like this:

const audioUrl = new URL('speech', this.transcriptionLink);
audioUrl.searchParams.set('text', 'text to speak goes here');

const audio = new Audio(audioUrl);
audio.play(); // Do this on-click or some similar user action, at least at first to avoid autoplay policy issues.

In this case, the browser makes the GET request for you by way of the audio element. It will start playback as soon as it thinks it has enough buffered.

Upvotes: 0

Related Questions