Tomer Holtzman
Tomer Holtzman

Reputation: 1

Activating MediaFrameReader severly lowers Unity's framerate on Hololens2

I am developing a CV app for Hololens with Unity and I require capturing video frames using mediacapture. I followed a bunch of examples and tutorials online and started receiving video frames. Then, I noticed Unity's MRTK profiler showing a drop in framerate (from the expected 60 to about 48-49).

I disabled everything in the app and verified the framerate achieved is 60Hz). Then I only enabled MediaFrameReader and its FrameArrived callback without any usage of the frame it provides. And saw the frame rate drop again.

I printed the current ManagedThreadIdfrom within the FrameArrived callback and noticed that it's a different thread each time, they are not from the runtime's threadpool and they are not being reused.

It seems like a stretch but is it possible MediaFrameReader creates a new thread each time it wants to notify me of a new frame? If so i think this could certainly explain both the drop in frame rate and the inconsistency of the timing in frame arrival.

public static async void StartVideoCaptureAsync(VideoFormatHandler formatHandler, VideoFrameHandler frameHandler, int? maxWidth = null, int? maxHeight = null, int? maxFrameRate = null, string device = null, int? source = null)
{
    log(string.Format("Attempting to create a MediaCapture object (w:{0} h:{1} fr:{2})", maxWidth, maxHeight, maxFrameRate));

    var videoConfig = await enumerateMediaCaptureVideoProfilesAsync(device, source, maxWidth, maxHeight, maxFrameRate);

    if (videoConfig != null)
    {
        log(string.Format("Configuring MediaCapture for {0}x{1} {2} frames at {4}FPS from {3}",
            videoConfig.profile.Width, videoConfig.profile.Height, videoConfig.profile.Subtype,
            videoConfig.info.DeviceInformation.Name, videoConfig.profile.FrameRate));

        mediaCapture = new MediaCapture();
        await mediaCapture.InitializeAsync(new MediaCaptureInitializationSettings()
        {
            SourceGroup = videoConfig.group,
            SharingMode = MediaCaptureSharingMode.ExclusiveControl,
            MemoryPreference = MediaCaptureMemoryPreference.Cpu,
            StreamingCaptureMode = StreamingCaptureMode.Video
        });

        mediaCapture.VideoDeviceController.Focus.TrySetAuto(true);

        log("Successfully created MediaCapture interface");

         mediaFrameReader = await mediaCapture.CreateFrameReaderAsync(
         mediaCapture.FrameSources[videoConfig.info.Id], videoConfig.profile.Subtype);

         mediaFrameReader.FrameArrived += frameHandler;
         await mediaFrameReader.StartAsync();

         log("Successfully started media frame reader");
    }
    else
    {
        log("Failed to configure media capture");
    }
}

private static void frameHandler(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
{
    Debug.Log(string.Format("{0}: notified of new frame {1} in thread {2} -{3}- pool[{4}]", currentqueue, currentqueue - lastqueue, Thread.CurrentThread.ManagedThreadId, Thread.CurrentThread.Name, Thread.CurrentThread.IsThreadPoolThread)); 
}

These are the prints I am getting from this:

1907: notified of new frame 1907 in thread 9 -- pool[False]
1973: notified of new frame 66 in thread 10 -- pool[False]
2007: notified of new frame 34 in thread 11 -- pool[False]
2028: notified of new frame 21 in thread 12 -- pool[False]
2054: notified of new frame 26 in thread 13 -- pool[False]

and it just goes on like this per frame - seemingly creating a new thread for each one.

Can anyone help shed some light on this phenomenon?

Thanks in advance!

EDIT: I deployed the app to hololens and the measurements were taken from the frameHandler() callback (namely i checked the frequency of the calls to it and the thread they came on). The MemoryPreference is set to CPU as I need the frames in CPU memory for processing.

However, since posting I've found the cause of the framerate reduction. The frames were coming in using the camera's 30FPS profile and not the 15FPS profile i wanted - because of misconfiguration on my part. So that overloaded my processing queue and everything went downhill from there. Now it's ok.

However, the question still remains why is every callback coming from a different thread - it makes no sense to create a new thread per frame. I ran the same thing on PC and there it is reusing the same 4 or 5 threads (judging by their IDs).

It's strange...

Upvotes: 0

Views: 257

Answers (0)

Related Questions