CODEWITHSUNDEEP
c#kinect
andyopayne
andyopayne

Reputation: 1368

Kinect V2 Color Stream Byte Order

I'm working on an application which will stream the color, depth, and IR video data from the Kinect V2 sensor. Right now I'm just putting together the color video part of the app. I've read through some tutorials and actually got some video data coming into my app - the problem seems to be that the byte order seems to be in the wrong order which gives me an oddly discolored image (see below).

enter image description here

So, let me explain how I got here. In my code, I first open the sensor and also instantiate a new multi source frame reader. After I've created the reader, I create an event handler called Reader_MultiSourceFrameArrived:

void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
    {
        if (proccessing || gotframe) return;

        // Get a reference to the multi-frame
        var reference = e.FrameReference.AcquireFrame();

        // Open color frame
        using (ColorFrame frame = reference.ColorFrameReference.AcquireFrame())
        {
            if (frame != null)
            {
                proccessing = true;
                var description = frame.ColorFrameSource.FrameDescription;

                bw2 = description.Width / 2;
                bh2 = description.Height / 2;
                bpp = (int)description.BytesPerPixel;

                if (imgBuffer == null)
                {
                    imgBuffer = new byte[description.BytesPerPixel * description.Width * description.Height];
                }

                frame.CopyRawFrameDataToArray(imgBuffer);
                gotframe = true;
                proccessing = false;
            }
        }
    }

Now, every time a frame is received (and not processing) it should copy the frame data into an array called imgBuffer. When my application is ready I then call this routine to convert the array into a Windows Bitmap that I can display on my screen.

if (gotframe)
        {
            if (theBitmap.Rx != bw2 || theBitmap.Ry != bh2) theBitmap.SetSize(bw2, bh2);

            int kk = 0;
            for (int j = 0; j < bh2; ++j)
            {
                for (int i = 0; i < bw2; ++i)
                {
                    kk = (j * bw2 * 2 + i) * 2 * bpp;
                    theBitmap.pixels[i, bh2 - j - 1].B = imgBuffer[kk];
                    theBitmap.pixels[i, bh2 - j - 1].G = imgBuffer[kk + 1];
                    theBitmap.pixels[i, bh2 - j - 1].R = imgBuffer[kk + 2];
                    theBitmap.pixels[i, bh2 - j - 1].A = 255;
                }
            }
            theBitmap.needupdate = true;
            gotframe = false;
        }
}

So, after this runs theBitmap now contains the image information needed to draw the image on the screen... however, as seen in the image above - it looks quite strange. The most obvious change is to simply change the order of the pixel B,G,R values when they get assigned to the bitmap in the double for loop (which I tried)... however, this simply results in other strange color images and none provide an accurate color image. Any thoughts where I might be going wrong?

Upvotes: 2

Views: 3166

Answers (2)

Olloxan
Olloxan

Reputation: 201

I was stuck on this problem forever. Problem is, that all almost all examples you find, are WPF examples. But for Windows Forms its a different story. 

frame.CopyRawFrameDataToArray(imgBuffer);

gets you the rawdata whitch is

ColorImageFormat.Yuy2

By converting it to RGB you should be able to fix your color problem. The transformation from YUY2 to RGB is very expensive, you might want to use a Parallel foreach loop to maintain your framerate

Upvotes: 1

16per9
16per9

Reputation: 502

Is this Bgra? The normal "RGB" in Kinect v2 for C# is BGRA.

Using the Kinect SDK 2.0, you don't need all of those "for" cycles. The function used to allocate the pixels in the bitmap is this one: 

colorFrame.CopyConvertedFrameDataToIntPtr(
                            this.colorBitmap.BackBuffer,
                            (uint)(colorFrameDescription.Width * colorFrameDescription.Height * 4),
                            ColorImageFormat.Bgra);

1) Get the Frame From the kinect, using Reader_ColorFrameArrived (go see ColorSamples - WPF);

2) Create the colorFrameDescription from the ColorFrameSource using Bgra format;

3) Create the bitmap to display;

If you have any problems, please say. But if you follow the sample it's actually pretty clean there how to do it.

Upvotes: 1

Related Questions