UI Freezes while processing frames from Kinect using EmguCV in WPF C# Application

Question

I am working on a project which involves extracting features from color and depth frames from Kinect Camera. The problem I am facing is that whenever I try displaying 2 Images, the UI hangs. When I tried debugging, the depthFrame and colorFrame were coming as null. If enable only the color steam then both colorImage and featureImage1 are displayed properly and if I enable only the depth stream, it works as it should. But when I enable them both, the UI hangs. I have no idea what is causing the problem. I have the following the following code for my Kinect Application. What is the cause of this problem and how can I fix it? Config: Windows 8 Pro 64bit, 2Ghz Core2Duo, VisualStudio 2012 Ultimate, EmguCV 2.4.0.

using System;
using System.Collections.Generic;
using System.Drawing;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Kinect;
using Emgu.CV;
using Emgu.CV.WPF;
using Emgu.CV.Structure;
using Emgu.Util;
namespace features
{
public partial class MainWindow : Window
{  
    public MainWindow()
    {
        InitializeComponent();
    }
    private Image<Bgra, Byte> cvColorImage;
    private Image<Gray, Int16> cvDepthImage;
    private int colorWidth = 640;
    private int colorHeight = 480;
    private int depthWidth = 640;
    private int depthHeight = 480;
    private static readonly int Bgr32BytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;
    private byte[] colorPixels;
    private byte[] depthPixels;
    private short[] rawDepthData;
    private bool first = true;
    private bool firstDepth = true;
    Image<Bgra, byte> image2;
    private void Window_Loaded(object sender, RoutedEventArgs e)
    {
        kinectSensorChooser.KinectSensorChanged += new DependencyPropertyChangedEventHandler(kinectSensorChooser_KinectSensorChanged);
    }
    void kinectSensorChooser_KinectSensorChanged(object sender, DependencyPropertyChangedEventArgs e)
    {
        KinectSensor oldSensor = (KinectSensor)e.OldValue;
        KinectStop(oldSensor);
        KinectSensor _sensor = (KinectSensor)e.NewValue;
        _sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
        _sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
        _sensor.DepthFrameReady += new EventHandler<DepthImageFrameReadyEventArgs>(_sensor_DepthFrameReady);
        _sensor.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(_sensor_ColorFrameReady);
        _sensor.DepthStream.FrameHeight);
        try
        {
            _sensor.Start();
        }
        catch
        {
            kinectSensorChooser.AppConflictOccurred();
        }
    }
    void KinectStop(KinectSensor sensor)
    {
        if (sensor != null)
        {
            sensor.Stop();
        }
    }

    private void Window_Closed(object sender, EventArgs e)
    {
        KinectStop(kinectSensorChooser.Kinect);
    }

    void _sensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
    {
        using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
        {
            if (colorFrame == null) return;
            if (first)
            {
                this.colorPixels = new Byte[colorFrame.PixelDataLength];
                first = false;
            }
            colorFrame.CopyPixelDataTo(this.colorPixels); //raw data in bgrx format
            processColor();
        }
    }
    void _sensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
    {
        using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
        {
            if (depthFrame == null) return;
            if (firstDepth)
            {
                this.rawDepthData = new short[depthFrame.PixelDataLength];
                firstDepth = false;
            }
            depthFrame.CopyPixelDataTo(rawDepthData);
            processDepth();
        }
    }
    private void processColor(){...}
    private void processDepth(){...}
}
}

The processDepth function is as follows. I am just making an Image from the RAW depth data.

private void processDepth() {
    GCHandle pinnedArray = GCHandle.Alloc(this.rawDepthData, GCHandleType.Pinned);
    IntPtr pointer = pinnedArray.AddrOfPinnedObject();
    cvDepthImage = new Image<Gray, Int16>(depthWidth, depthHeight, depthWidth << 1, pointer);
    pinnedArray.Free();
    depthImage.Source = BitmapSourceConvert.ToBitmapSource(cvDepthImage.Not().Bitmap);
}

The processColor function is as follows. Here just for the sake of it, I am trying to display the cloned image instead of extracting features, just to check the lag. When both streams are enabled (color and depth) the following function displays the colorImage properly, but as soon as I uncomment the commented lines, the UI hangs.

private void processColor() {
    GCHandle handle = GCHandle.Alloc(this.colorPixels, GCHandleType.Pinned);
    Bitmap image = new Bitmap(colorWidth, colorHeight, colorWidth<<2, System.Drawing.Imaging.PixelFormat.Format32bppRgb, handle.AddrOfPinnedObject());
    handle.Free();
    cvColorImage = new Image<Bgra, byte>(image);
    image.Dispose();
    BitmapSource src = BitmapSourceConvert.ToBitmapSource(cvColorImage.Bitmap);
    colorImage.Source = src;
    //image2 = new Image<Bgra, byte>(cvColorImage.ToBitmap()); //uncomment and it hangs
    //featureImage1.Source = BitmapSourceConvert.ToBitmapSource(image2.Bitmap); //uncomment and it hangs
}

Answer 1

I see code that do a lot of work in event handlers. I almost certain that handlers is called in GUI thread. I suggest you to extract your code to the background thread routine.
Don't forget that updating of the Form's controls (depthImage and controlImage) should be done using BeginInvoke method of the parent form,