Image Processing with Kinect and AForge

Question

I'm working on a project where I want to track a dice with the Microsoft Kinect using the AForge.NET-Library. The project itself contains only the fundamentals such as initializing the Kinect, obtaining a Colorframe and applying one color filter but there already the problem occurs. So here is the main part of the program:

void ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
    {
        using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
        {
            if (colorFrame != null)
            {
                colorFrameManager.Update(colorFrame);
                BitmapSource thresholdedImage =
                    diceDetector.GetThresholdedImage(colorFrameManager.Bitmap);

                if (thresholdedImage != null)
                {
                    Display.Source = thresholdedImage;
                }
            }

        }
    }

The 'Update'-method of the 'colorFrameManager'-object looks like this:

public void Update(ColorImageFrame colorFrame)
    {
        byte[] colorData = new byte[colorFrame.PixelDataLength];
        colorFrame.CopyPixelDataTo(colorData);

        if (Bitmap == null)
        {
            Bitmap = new WriteableBitmap(colorFrame.Width, colorFrame.Height,
                96, 96, PixelFormats.Bgr32, null);
        }
        int stride = Bitmap.PixelWidth * Bitmap.Format.BitsPerPixel / 8;
        imageRect.X = 0;
        imageRect.Y = 0;
        imageRect.Width = colorFrame.Width;
        imageRect.Height = colorFrame.Height;
        Bitmap.WritePixels(imageRect, colorData, stride, 0);
    }

And the 'getThresholdedImage'-method looks like this:

public BitmapSource GetThresholdedImage(WriteableBitmap colorImage)
    {
        BitmapSource thresholdedImage = null;
        if (colorImage != null)
        {
            try
            {
                Bitmap bitmap = BitmapConverter.ToBitmap(colorImage);
                HSLFiltering filter = new HSLFiltering();
                filter.Hue = new IntRange(335, 0);
                filter.Saturation = new Range(0.6f, 1.0f);
                filter.Luminance = new Range(0.1f, 1.0f);
                filter.ApplyInPlace(bitmap);
                thresholdedImage = BitmapConverter.ToBitmapSource(bitmap);
            }
            catch (Exception ex)
            {
                System.Console.WriteLine(ex.Message);
            }
        }
        return thresholdedImage;
    }

Now the program slows down a lot/ doesn't respond when this line is executed:

filter.ApplyInPlace(bitmap);

So I already read this thread (C# image processing on Kinect video using AForge) and I tried EMGU but I couldn't get it to work because of inner exceptions and as the thread-starter wasn't online since four months my question to have a look at his working code wasn't answered. Now firstly I'm intereseted in how the reason for the slow execution can be

filter.ApplyInPlace(bitmap);

Is this image processing really so complex? Or could this be a problem with my enviroment? Secondly I would like to ask if skipping frames is a good solution? Or is it better to use polling and open frames only every - for instance - 500 milliseconds. Thank you very much!

Answer 1

The HSL filter would not slow down the computation, is not an complex Filter. Im utilizing it in 320x240 images with 30 fps without problems.

The problem may be in the resolution of the computed image or in a too high frame rate!

If the resolution of the image is high, i suggest to resize it before any filter application. And i think a framerate of 20 (and maybe less) is enough to tracking a dice.