Implement PIP mode for my agora video call app

Question

Problem

Unable to Use Native PiP Mode with AVSampleBufferDisplayLayer in Agora iOS SDK

I am working on an Agora Live Streaming feature in an iOS Swift app, where I am using AVSampleBufferDisplayLayer to render Agora's onRenderVideoFrame delegate frames. The live video stream works well in the foreground, but I am unable to enable native Picture-in-Picture (PiP) mode when the app goes into the background.

What I have done so far

1. Agora Kit Initialization

   I have initialized Agora with the following configurations:

   agoraKit.setParameters("{\"che.video.keep_last_frame\": true}")
   agoraKit.setVideoFrameDelegate(self)
   agoraKit.setParameters("{\"che.video.force_renderer_background\": true}")
   agoraKit.enableVideo()
   agoraKit.setVideoFrameDelegate(self)
   agoraKit.enableLocalVideo(true)
   agoraKit.setChannelProfile(.liveBroadcasting)
   agoraKit.setClientRole(.broadcaster)
   // agoraKit.enableWebSdkInteroperability(true)
   agoraKit.setParameters("{\"rtc.video.pip\": true}")
   agoraKit.setParameters("{\"rtc.video.background_mode\": 1}")
   agoraKit.setParameters("{\"rtc.video.enable_ios_background_mode\": true}")
   

2. Processing Video Frames in onRenderVideoFrame I am processing the incoming frames and rendering them using AVSampleBufferDisplayLayer.

   private var videoLayerRemote: AVSampleBufferDisplayLayer?

   func onRenderVideoFrame(_ videoFrame: AgoraOutputVideoFrame, uid: UInt, channelId: String) -> Bool {
       frameCount += 1
   
       // Ignore local video frames
       if uid == localUid {
           return true
       }
   
       var sampleBuffer: CMSampleBuffer?
   
       if videoFrame.type == 1 { // I420 Format
           guard let yBuffer = videoFrame.yBuffer,
                 let uBuffer = videoFrame.uBuffer,
                 let vBuffer = videoFrame.vBuffer else {
               return true
           }
   
           sampleBuffer = convertI420ToCMSampleBuffer(yPlane: yBuffer,
                                                      uPlane: uBuffer,
                                                      vPlane: vBuffer,
                                                      width: Int(videoFrame.width),
                                                      height: Int(videoFrame.height))
       } else {
           guard let pixelBuffer = videoFrame.pixelBuffer else {
               return true
           }
           sampleBuffer = createSampleBuffer(from: pixelBuffer)
       }
   
       guard let videoLayer = videoLayerRemote else {
           return true
       }
   
       if videoLayer.status == .failed {
           videoLayer.flush()
       }
   
       if !videoLayer.isReadyForMoreMediaData {
           return true
       }
   
       videoLayer.enqueue(sampleBuffer!)
   
       return true
   }
   

3. PiP Configuration I have done the following to enable Picture-in-Picture (PiP):

   Enabled Background Modes:

   • Audio, AirPlay, and Picture in Picture
   • Background Processing
   • Background Fetch
   • Multiprocessing Added Capability for Multi-Camera Access

Issue

When the app goes into the background, the video stops rendering. PiP mode does not activate. The app does not display the AVSampleBufferDisplayLayer when in the background.

I want to

• Keep AVSampleBufferDisplayLayer active when the app goes into the background.
• Enable native iOS Picture-in-Picture (PiP) mode. Automatically hide AVSampleBufferDisplayLayer when the app comes to the foreground.

What I have tried

Tried Using AVPictureInPictureController with AVSampleBufferDisplayLayer, but no success

Since AVSampleBufferDisplayLayer does not directly support PiP, I tried wrapping it in a AVPlayerLayer, but this did not work. Using Agora setParameters("{\"rtc.video.pip\": true}")

No effect on enabling PiP mode.

Handling app state transitions

I tried handling UIApplication.didEnterBackgroundNotification to manually configure PiP but couldn't get AVSampleBufferDisplayLayer to display.

Questions

1. How can I keep rendering the Agora video stream using AVSampleBufferDisplayLayer in the background?

2. How can I enable native Picture-in-Picture (PiP) mode using AVSampleBufferDisplayLayer in an Agora iOS app?splay it when the app goes in background and hide when app comes in foreground