Streaming screen record from WebRTC app - route audio to internal

Question

TL;DR

How to route remote participants voice in WebRTC call so they can be treated as internal sounds by Streamlabs or similar streaming apps?

Long story

My game is using Google WebRTC library for the main functionality - audio/video chat. I want to promote my players to stream the screen cast to YouTube and Twitch right from their Android phones. I tested a few apps and found that Streamlabs works great for my purpose.

My players can spectate a game and stream the app screen and sounds to Twitch/YouTube using Streamlabs app.

Streamlabs can pick audio from 2 channels - Internal & Microphone (you can adjust volume for each one). All ambient game sounds, like clicks, music etc. goes to Internal which is great. The WebRTC sounds from other participants (players) routed by my app (using Android AudioManager) to Speakerphone, then they picked up by phone's Microphone. So, this works, but this is not great.

The problem is that the streamer should always keep his mic on, even if he doesn't say anything because WebRTC speech from remote players goes this route (WebRTC -> Speakerphone -> Mic -> Streamlabs). And he also has always be in quite environment. Plus, the quality of sound obviously degraded.

Is there a way to route WebRTC sounds in a way, so they will be treated as Internal by Streamlabs?

Note that the problem is common for all WebRTC / VOIP call / conference call apps on Android. This can be easily reproduced with WhatsApp call, or Facebook Messenger call or any similar app. You can also use YouTube or Twitch app instead of Streamlabs and get similar results.

I tried different setting with Android AudioManager (using USAGE_MEDIA instead of USAGE_VOICE_COMMUNICATION, setting Audio Mode to NORMAL instead of MODE_IN_COMMUNICATION and more), but none of them works. I guess this is because the streaming app (Streamlabs etc) grabs Internal sounds even before it reaches AudioManager, probably on Android MediaPlayer level.

It would be great if I can route remote WebRTC participants voice in a way that it can be picked up by Streamlabs as Internal sounds, so it won't need to travel by air from Speakerphone to the Mic loosing quality and mixing with environment sounds.

I can alter Google WebRTC if needed (I build it myself anyway).

Answer 1

From the look of things, you’re trying to route WebRTC audio directly to the internal audio stream that Steamlabs can capture rather than having it go through the speaker-microphone path. You should try these approaches:

Investigate AudioTrack since WebRTC likely uses AudioTrack for audio output. You might be able to modify the audio sink:

// Example approach using AudioTrack
AudioTrack audioTrack = new AudioTrack.Builder()
    .setAudioAttributes(new AudioAttributes.Builder()
        .setUsage(AudioAttributes.USAGE_MEDIA)  // Instead of VOICE_COMMUNICATION
        .setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
        .build())
    .setAudioFormat(/* your format */)
    .setTransferMode(AudioTrack.MODE_STREAM)
    .build();

Implement a virtual audio device in WebRTC:

public class VirtualAudioDevice implements AudioDeviceModule {
    private MediaProjection mediaProjection;
    private AudioRecord audioRecord;
    private AudioTrack audioTrack;
    private boolean isCapturing;
    private boolean isPlaying;
    
    // Buffer for audio routing
    private final LinkedBlockingQueue<ByteBuffer> audioBuffer = new LinkedBlockingQueue<>();
    
    @Override
    public int init() {
        // Initialize virtual device
        AudioFormat format = new AudioFormat.Builder()
            .setSampleRate(48000)
            .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
            .setChannelMask(AudioFormat.CHANNEL_OUT_STEREO)
            .build();
            
        // Create a virtual output device that other apps can capture
        audioTrack = new AudioTrack.Builder()
            .setAudioAttributes(new AudioAttributes.Builder()
                .setUsage(AudioAttributes.USAGE_MEDIA)
                .setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
                .setFlags(AudioAttributes.FLAG_LOW_LATENCY)
                .build())
            .setAudioFormat(format)
            .setTransferMode(AudioTrack.MODE_STREAM)
            .setPerformanceMode(AudioTrack.PERFORMANCE_MODE_LOW_LATENCY)
            .build();
            
        return 0;
    }
    
    @Override
    public boolean startPlayout() {
        if (isPlaying) {
            return true;
        }
        
        audioTrack.play();
        isPlaying = true;
        
        // Start audio routing thread
        new Thread(() -> {
            while (isPlaying) {
                try {
                    ByteBuffer buffer = audioBuffer.take();
                    if (buffer != null) {
                        byte[] audio = new byte[buffer.remaining()];
                        buffer.get(audio);
                        audioTrack.write(audio, 0, audio.length);
                    }
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    break;
                }
            }
        }).start();
        
        return true;
    }
    
    @Override
    public boolean stopPlayout() {
        isPlaying = false;
        if (audioTrack != null) {
            audioTrack.stop();
            audioTrack.flush();
        }
        return true;
    }
    
    // Method to receive audio data from WebRTC
    public void onWebRTCAudioFrame(ByteBuffer audioData) {
        if (isPlaying) {
            try {
                audioBuffer.put(audioData);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }
    }
    
    // Implement other required methods...
    
    @Override
    public void release() {
        stopPlayout();
        if (audioTrack != null) {
            audioTrack.release();
            audioTrack = null;
        }
    }
}

Other alternatives are:

• Use Android's AudioMix API to create a virtual audio source
• Implement a custom audio sink in WebRTC that writes to a virtual audio device
• Use Android's AudioRecord to capture the WebRTC audio and redirect it

What I recommend you try first are:

Modify your WebRTC build to use the virtual audio device implementation provided above. This creates a virtual audio output that should be captured as internal audio.

In your WebRTC initialization:

VirtualAudioDevice audioDevice = new VirtualAudioDevice();
PeerConnectionFactory.initialize(
    PeerConnectionFactory.InitializationOptions.builder(context)
        .setAudioDeviceModule(audioDevice)
        .createInitializationOptions()
);

Route the WebRTC audio through this device:

// In your WebRTC audio track callback
@Override
public void onWebRTCAudioFrame(AudioTrack.Buffer buffer) {
    audioDevice.onWebRTCAudioFrame(buffer.data);
}

Some key considerations to note:

• The virtual device needs to present itself as a media output rather than communication output
• You'll need to handle proper audio mixing if you have multiple WebRTC streams
• Consider latency and buffer size carefully to avoid audio glitches
• Test with different Android versions as audio routing behaviors can vary

Answer 2

I solved the problem by editing WebRTC sources to:

1. Get raw PCM buffer (to add it to my own AudioTrack) and
2. Destroy (or do not create) internal AudioTrack.

It appears essential to destroy existing internal WebRTC AudioTrack, because normally it should be only one AudioTrack per app. Even if all the sounds go to my AudioTrack, the internal AudioTrack, if exists, may confuse other apps. With 2 active AudioTracks the streaming works fine with Twitch app, but appears completely broken with YouTube app (even the Mic stop working properly). Need to mention, it works normally locally in either case (i.e. you can hear everything just fine), but when streaming, having just one AudioTrack with correct settings (i.e. USAGE_MEDIA) is essential.

Another thing I want to share, is that it's very important to set the correct size of audio buffer for AudioTrack. Initially I thought that as bigger as better, but it's not. Actually you should select the as smaller buffer as permitted, otherwise the the sound lag behind video.

Answer 3

It’s tricky to route WebRTC audio correctly on Android to make it usable for streaming apps like Streamlabs without losing quality.

If modifying the WebRTC code is an option for you, consider implementing custom audio routing or integrating a virtual audio device within your app. This would likely give you better control over how the sound is handled before it reaches Streamlabs.