Controlling Frame Rate of VirtualDisplay

Question

I'm writing an Android application, and in it, I have a VirtualDisplay to mirror what is on the screen and I then send the frames from the screen to an instance of a MediaCodec. It works, but, I want to add a way of specifying the FPS of the encoded video, but I'm unsure how to do so.

From what I've read and experimented with, dropping encoded frames (based on the presentation times) doesn't work well as it ends up with blocky/artifact ridden video as opposed to a smooth video at a lower framerate. Other reading suggests that the only way to do what I want (limit the FPS) would be to limit the incoming FPS to the MediaCodec, but the VirtualDisplay just receives a Surface which is constructed from the MediaCodec as below

mSurface = <instance of MediaCodec>.createInputSurface();
mVirtualDisplay = mMediaProjection.createVirtualDisplay(
    "MyDisplay",
    screenWidth,
    screenHeight,
    screenDensity,
    DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
    mSurface,
    null,
    null);

I've also tried subclassing Surface and limit the frames that are fed to the MediaCodec via the unlockCanvasAndPost(Canvas canvas) but the function never seems to be called on my instance, so, there may be some weirdness in how I extended Surface and the interaction with the Parcel as writeToParcel function is called on my instance, but that is the only function that is called in my instance (that I can tell).

Other reading suggests that I can go from encoder -> decoder -> encoder and limit the rate in which the second encoder is fed frames, but that's a lot of extra computation that I'd rather not do if I can avoid it.

Has anyone successfully limited the rate at which a VirtualDisplay feeds its Surface? Any help would be greatly appreciated!

Answer 1

You can control the frame rate of a virtual display by

1. Creating an ImageReader.
2. Getting the surface from an ImageReader and submitting it to the VirtualDisplay.
3. On the ImageReader callback, acquire the latest Image frame, and store it.
4. In another thread/event tick grab that runs at your desired frame rate, grab the latest frame from the intermedia variable.
5. Take the surface of the encoder and grab the hardware canvas.
6. Convert the Image to a Bitmap.
7. Draw the Bitmap to the hardware canvas (optionally downscale or other drawing things you want), then unlock it.

You'll need to close images that aren't actually drawn, but this will be obvious as the application will crash if you don't do this but the stack trace will allow you to find the solution (along with the documentation) pretty fast.

This will work with recent versions of Android and is measured at around 11ms per frame. Most of the time spent will be in conversion of Image -> Bitmap which requires the pixels being downloaded from the GPU -> CPU, but then drawing will be done via GPU and very is fast.

If you are working with Android 31 and above you can replace steps 5-7 with an ImageWriter which will keep everything in the GPU. However prior to API 31 the ImageWriter will not be able to write to an encoder surface it expects pixel format RGB888 and the VirtualDisplay will produce images in pixel format RGBA8888 space and the work around for this was introduced in API 31.

Keep in mind that using pure EGL Surface's may be faster. But this works well enough to work at 30 fps. Multiple threads may be able to get it working at 60fps.

Here is some example Kotlin code that does this:

import android.annotation.SuppressLint
import android.graphics.Bitmap
import android.graphics.Paint
import android.graphics.PixelFormat
import android.graphics.Rect
import android.media.Image
import android.media.ImageReader
import android.view.Surface
import java.lang.Double.max
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicReference

// An intermedia surface that downscales the input surface to the destination surface. Note that
// this is all done in software, so it is not very efficient and is measured at about 13ms per frame
// for the Samsung SM-S901U and ~13ms for Motorola Moto E6. However it is fast enough given that the
// capture rate is ~1 fps.
class SurfaceDownscaler(
    private val srcWidth: Int, private val srcHeight: Int,
    private val dstWidth: Int, private val dstHeight: Int,
    private val fps: Float) {

    companion object {
        private const val TAG = "SurfaceDownscaler"
        private const val LOG_PERFORMANCE = true
    }

    var inputSurface: Surface

    fun setDestinationSurface(surface: Surface?) {
        synchronized(surfaceLock) {
            surfaceDestination = surface
        }
    }

    fun pause() {
        paused.set(true)
    }
    fun resume() {
        paused.set(false)
    }

    private val paused = AtomicBoolean(false)
    private val surfaceLock = Any()
    private var surfaceDestination: Surface? = null
    private var imageFormat: Int = PixelFormat.RGBA_8888
    private var imageReader: ImageReader
    private val recentImage: AtomicReference<Image?> = AtomicReference(null)
    private val timerAllowImage = TimerIntervalThreaded(TAG, this::onTick)
    private lateinit var largeBitmap: Bitmap
    private val paint = Paint(Paint.ANTI_ALIAS_FLAG)

    private val imageListener: ImageReader.OnImageAvailableListener = ImageReader.OnImageAvailableListener {
        val image: Image? = it.acquireLatestImage()
        if (image == null) {
            return@OnImageAvailableListener
        }
        val prevImage: Image? = recentImage.getAndSet(image)
        if (prevImage != null) {
            prevImage.close()
        }
    }

    init {
        assert(srcWidth > 0)
        assert(srcHeight > 0)
        assert(dstWidth > 0)
        assert(dstHeight > 0)
        @SuppressLint("WrongConstant")
        imageReader = ImageReader.newInstance(srcWidth, srcHeight, imageFormat, 3)
        imageReader.setOnImageAvailableListener(imageListener, null)
        timerAllowImage.start()
        inputSurface = imageReader.surface
        paint.isFilterBitmap = true
    }

    fun close() {
        timerAllowImage.stop()
        imageReader.close()
        recentImage.getAndSet(null)?.close()
        if (::largeBitmap.isInitialized) {
            largeBitmap.recycle()
        }
    }


    private fun drawImageAndRelease(image: Image) {
        val startTime = System.currentTimeMillis()
        val planes = image.planes
        val pixelStride = planes[0].pixelStride
        val rowStride = planes[0].rowStride
        val rowPadding: Int = rowStride - pixelStride * image.width
        if (!::largeBitmap.isInitialized) {
            largeBitmap = Bitmap.createBitmap(
                srcWidth + rowPadding / pixelStride, srcHeight,
                Bitmap.Config.ARGB_8888
            )
        }
        largeBitmap.copyPixelsFromBuffer(planes[0].buffer)
        image.close()
        synchronized(surfaceLock) {
            val surface = surfaceDestination
            if (surface != null) {
                val canvas = surface.lockHardwareCanvas()
                val dstRect = Rect(0, 0, dstWidth, dstHeight)
                canvas.drawBitmap(largeBitmap, null, dstRect, paint)
                surface.unlockCanvasAndPost(canvas)
            }
        }
        val duration = System.currentTimeMillis() - startTime
        if (LOG_PERFORMANCE) {
            Log.d(TAG, "drawImageAndRelease duration: $duration")
        }
    }

    private fun onTick(): Long {
        val startTimeAll = System.currentTimeMillis()
        val image = recentImage.getAndSet(null)
        if (image != null) {
            if (paused.get()) {
                image.close()
            } else {
                drawImageAndRelease(image)
            }
        }
        val endTime = System.currentTimeMillis()
        val duration = endTime - startTimeAll
        val sleepTime = max(0.0, (1000.0 / fps) - duration.toDouble())
        if (LOG_PERFORMANCE) {
            Log.d(TAG, "duration: $duration")
        }
        return sleepTime.toLong()
    }

}

Answer 2

You can reference the code sample from saki4510t's ScreenRecordingSample or RyanRQ's ScreenRecoder, they are all use the additional EGL Texture between the virtual display and media encoder, and the first one can keep at least 15 fps for the output video. You can search the keyword createVirtualDisplay from their code base for more details.

Answer 3

Starting off with what you can't do...

You can't drop content from the encoded stream. Most of the frames in the encoded stream are essentially "diffs" from other frames. Without knowing how the frames interact, you can't safely drop content, and will end up with that corrupted macroblock look.

You can't specify the frame rate to the MediaCodec encoder. It might stuff that into metadata somewhere, but the only thing that really matters to the codec is the frames you're feeding into it, and the presentation time stamps associated with each frame. The encoder will not drop frames.

You can't do anything useful by subclassing Surface. The Canvas operations are only used for software rendering, which is unrelated to feeding in frames from a camera or virtual display.

What you can do is send the frames to an intermediate Surface, and then choose whether or not to forward them to the MediaCodec's input Surface. One approach would be to create a SurfaceTexture, construct a Surface from it, and pass that to the virtual display. When the SurfaceTexture's frame-available callback fires, you either ignore it, or render the texture onto the MediaCodec input Surface with GLES.

Various examples can be found in Grafika and on bigflake, none of which are an exact fit, but all of the necessary EGL and GLES classes are there.