Unity: How can I get realtime image capturing at ~60fps?

Question

I am writing an application in Unity which will be required to capture an image from a camera every frame (at ~60fps), and send the resultant data to another service running locally.

The issue is, I am aware that capturing the rendered data from the camera can cause massive frame rate drops (as explained in this article) when using the GetPixels() method. The article explains that "GetPixels() blocks for ReadPixels() to complete" and "ReadPixels() blocks while flushing the GPU" which is why the GPU and CPU have to sync up, resulting in a lag.

I have produced a sample project with a script attached which simply outputs frames to a file as a PNG to replicate the functionality of the program I wish to create. I have done my best to implement what is described in the article, namely allowing the GPU to render a frame, then wait a few frames before calling GetPixels() so as not to cause the GPU and CPU to forcefully sync up. However, I really haven't made any progress with it. The project still plays at about 10-15fps.

How can I achieve a realtime capture of 60 frames per second in Unity?

using System;
using System.Collections;
using System.IO;
using UnityEngine;

namespace Assets
{
    public class MyClass: MonoBehaviour
    {
        private const float reportInterval = 0.5f;
        private int screenshotCount = 0;
        private const float maxElapsedSecond = 20;
        private string screenshotsDirectory = "UnityHeadlessRenderingScreenshots";
        public Camera camOV;
        public RenderTexture currentRT;
        private int frameCount = 0;
        private Texture2D resultantImage;

        public void Start()
        {
            camOV.forceIntoRenderTexture = true;
            if (Directory.Exists(screenshotsDirectory))
            {
                Directory.Delete(screenshotsDirectory, true);
            }
            if (!Application.isEditor)
            {
                Directory.CreateDirectory(screenshotsDirectory);
                camOV.targetTexture = currentRT;
            }
        }

        // Update is called once per frame
        public void Update()
        {
            //Taking Screenshots
            frameCount += 1;
            if (frameCount == 1)
            {
                TakeScreenShot();
            }
            else if (frameCount == 3)
            {
                ReadPixelsOut("SS_"+screenshotCount+".png");
            }

            if (frameCount >= 3)
            {
                frameCount = 0;
            }
        }

        public void TakeScreenShot()
        {
            screenshotCount += 1;
            RenderTexture.active = camOV.targetTexture;
            camOV.Render();
            resultantImage = new Texture2D(camOV.targetTexture.width, camOV.targetTexture.height, TextureFormat.RGB24, false);
            resultantImage.ReadPixels(new Rect(0, 0, camOV.targetTexture.width, camOV.targetTexture.height), 0, 0);
            resultantImage.Apply();
        }

        private void ReadPixelsOut(string filename)
        {
            if (resultantImage != null)
            {
                resultantImage.GetPixels();
                RenderTexture.active = currentRT;

                byte[] bytes = resultantImage.EncodeToPNG();
                // save on disk
                var path = screenshotsDirectory + "/" + filename;
                File.WriteAllBytes(path, bytes);

                Destroy(resultantImage);
            }
        }
    }
}

The article implies that it is possible, but I haven't managed to get it to work.

Many thanks in advance for your help.

Unity: How can I get realtime image capturing at ~60fps?

Answers (1)

Related Questions