CODEWITHSUNDEEP
iosswiftavplayeravaudiosession
Sebastian Misas
Sebastian Misas

Reputation: 71

Record audio while playing video and only record from microphone

I have an app that records the user's audio session while they are playing a video simultaneously. The issue is the playback from the video comes out of the speaker.

Ideally, I would like to cancel out the audio coming from the speaker and only record the user's voice through the microphone.

Example App: FaceTime Facetime cancels out the music you are playing when you are on a FaceTime call and only records the user's voice. (https://www.quora.com/How-does-an-iPhone-cancel-out-the-music-you-are-playing-when-you-FaceTime-someone)

Below is my code to add the audio session to record the user:

let audioInputDevice = AVCaptureDevice.default(
    .builtInMicrophone, 
    for: .audio, 
    position: .unspecified
)
.default(for: AVMediaType.audio)!
let session = self.previewLiveCamera.cameraLayer!.session!
        
//TODO: fix Audio Input, do not mix with other video audio
do {
    audioInput = try AVCaptureDeviceInput(device: audioInputDevice!)
    session.addInput(audioInput!)
} catch {
    print("failed to add audio input")
}
        
for output in session.outputs {
    let connectionOutput = output as! AVCaptureMovieFileOutput
    self.connectionOutput = connectionOutput
    print("Started reacording to \(fileURL!)")
    self.connectionOutput.startRecording(to: fileURL!, recordingDelegate: self)        
}

Furthermore, I'm setting the videplayer for the video the user views simultaneously:

let postPlayer = AVPlayer(url: postVideoUrl!)
cell.playerLayer.player = postPlayer

Upvotes: 7

Views: 5386

Answers (2)

kakaiikaka
kakaiikaka

Reputation: 4487

For those looking for a swift version Active Echo Cancellation [AEC] component, here is the repo I wrote: AECAudioStream

enter image description here

But you can no longer use high-level APIs like AVCaptureDevice to capture audio. You have to store the AVAudioPCMBuffers with AVAudioFile

Here are the steps to use it:

Create a AECAudioStream

To create a standard AudioStream without any special renderrer callback, you initialize a new instance of the AECAudioStream class, and supply a sampling rate for the recorded Audio, as the following code shows:

  /// Audio Samping at 16000Hz
  let audioUnit = AECAudioStream(sampleRate: 16000)

Get AEC Filtered Audio Data from AECAudioStream

After an AudioStream object is created, you can listen for recorded audio data by calling AECAudioStream/AECAudioStream/startAudioStream(enableAEC:) it returns an AsyncThrowingStream that yields AVAudioPCMBuffer objects containing the captured audio data.

for try await pcmBuffer in audioUnit.startAudioStream(enableAEC: true) {
  // here you get a ``AVAudioPCMBuffer`` data
  audioFile?.write(from: pcmBuffer)
}

Upvotes: 1

Joshua Ostrom
Joshua Ostrom

Reputation: 451

After adding the input audio source you'll need to set the audio mode:

do {
    audioInput = try AVCaptureDeviceInput(device: audioInputDevice!)
    session.addInput(audioInput!)
} catch {
    print("failed to add audio input")
}

// Add the following lines:
try! AVAudioSession.sharedInstance().setCategory(.playAndRecord, mode: .voiceChat)
try! AVAudioSession.sharedInstance().setActive(true)

.voiceChat will enable the device’s tonal equalization is optimized for voice and the set of allowable audio routes is reduced to only those appropriate for voice chat

https://developer.apple.com/documentation/avfoundation/avaudiosession/mode/1616455-voicechat

If the audio still has too much of the speaker audio present after setting the category you'd then need to look at enabling Active Echo Cancellation [AEC]. Here's an example of doing so under iOS:

https://github.com/twilio/video-quickstart-ios/blob/master/AudioDeviceExample/AudioDevices/ExampleAVAudioEngineDevice.m#L802 Note the VoiceProcessingIO et al.

Upvotes: 3

Related Questions