Tracking Blobs with Microsoft Kinect

Question

I am working on a people counter. For this I have the Microsoft Kinect installed over the door. I am working with C# and EmguCV. I have extracted the heads of the people, so that they appear as white blobs on a black image. Then I have created a bounding box around the heads. That works fine. So I now how many blobs I have per frame and I also now their position. This works fine. But now I want to track the blobs because I want to count how much people come in and go out, but I don't know how to do this. Can anyone help me? The problem is that every frame, new blobs can appear and old blobs can disappear. Can anyone give me an algorithm or maybe some code? or a paper. Thanks a lot!

---

Sure. This is the code for the blobs:

using (MemStorage stor = new MemStorage())
        {



            Contour<System.Drawing.Point> contours = head_image.FindContours(Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE, Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_EXTERNAL, stor);



            for (int i = 0; contours != null; contours = contours.HNext)
            {

                i++;



                //if ((contours.Area > Math.Pow(sliderMinSize.Value, 2)) && (contours.Area < Math.Pow(sliderMaxSize.Value, 2)))
                {

                    MCvBox2D box = contours.GetMinAreaRect();

                    blobCount++;

                    contour_image.Draw(box, new Bgr(System.Drawing.Color.Red), 1);


                    new_position = new System.Drawing.Point((int)(box.center.X), (int)(box.center.Y));
                    new_x = box.center.X;
                    new_y = box.center.Y;
                }

            }
        }

Answer 1

Please see Emgu CV Blob Detection for more information. Assuming you are using Emgu CV 2.1 or higher, then the answer will work. If you are using version 1.5 or higher, see this thread on how to easily detect blobs. Or look at the code below

     Capture capture = new Capture();

     ImageViewer viewer = new ImageViewer();

     BlobTrackerAutoParam param = new BlobTrackerAutoParam();
     param.ForgroundDetector = new ForgroundDetector(Emgu.CV.CvEnum.FORGROUND_DETECTOR_TYPE.FGD);
     param.FGTrainFrames = 10;
     BlobTrackerAuto tracker = new BlobTrackerAuto(param);

     Application.Idle += new EventHandler(delegate(object sender, EventArgs e)
     {
        tracker.Process(capture.QuerySmallFrame().PyrUp());
        Image<Gray, Byte> img = tracker.GetForgroundMask();
        //viewer.Image = tracker.GetForgroundMask();

        MCvFont font = new MCvFont(Emgu.CV.CvEnum.FONT.CV_FONT_HERSHEY_SIMPLEX, 1.0, 1.0);
        foreach (MCvBlob blob in tracker)
        {
           img.Draw(Rectangle.Round(blob), new Gray(255.0), 2);
           img.Draw(blob.ID.ToString(), ref font, Point.Round(blob.Center), new Gray(255.0));
        }
        viewer.Image = img;
     });
     viewer.ShowDialog();

Hope this helps!

EDIT

I think you should use this code every ten frames or so (~3 times a second) and do something like this:

     Capture capture = new Capture();

     ImageViewer viewer = new ImageViewer();

     BlobTrackerAutoParam param = new BlobTrackerAutoParam();
     param.ForgroundDetector = new ForgroundDetector(Emgu.CV.CvEnum.FORGROUND_DETECTOR_TYPE.FGD);
     param.FGTrainFrames = 10;
     BlobTrackerAuto tracker = new BlobTrackerAuto(param);
     int frames = 0;
     Application.Idle += new EventHandler(delegate(object sender, EventArgs e)
     {
        frames++;//Add to number of frames
        if (frames == 10)
        {
        frames = 0;//if it is after 10 frames, do processing and reset frames to 0
        tracker.Process(capture.QuerySmallFrame().PyrUp());
        Image<Gray, Byte> img = tracker.GetForgroundMask();
        //viewer.Image = tracker.GetForgroundMask();

        int blobs = 0;

        MCvFont font = new MCvFont(Emgu.CV.CvEnum.FONT.CV_FONT_HERSHEY_SIMPLEX, 1.0, 1.0);
        foreach (MCvBlob blob in tracker)
        {
           //img.Draw(Rectangle.Round(blob), new Gray(255.0), 2);
           //img.Draw(blob.ID.ToString(), ref font, Point.Round(blob.Center), new Gray(255.0));
           //Only uncomment these if you want to draw a rectangle around the blob and add text
           blobs++;//count each blob
        }
        blobs = /*your counter here*/;
        blobs = 0; //reset 
        viewer.Image = img;//get next frame
     });
     viewer.ShowDialog();

EDIT 2

It sounds like you just want to identify the blobs, it sounds like you want McvBlob.ID. This is the ID of the blob and you can check which ID's are still there and which are not. I would still do this every ten frames to not slow it down as much. You just need a simple algorithm that can observe what the ID's are, and if they have changed. I would store the IDs in a List<string> and check that list for changes every few frames. Example:

List<string> LastFrameIDs, CurrentFrameIDs;

         Capture capture = new Capture();

     ImageViewer viewer = new ImageViewer();

     BlobTrackerAutoParam param = new BlobTrackerAutoParam();
     param.ForgroundDetector = new ForgroundDetector(Emgu.CV.CvEnum.FORGROUND_DETECTOR_TYPE.FGD);
     param.FGTrainFrames = 10;
     BlobTrackerAuto tracker = new BlobTrackerAuto(param);
     int frames = 0;
     Application.Idle += new EventHandler(delegate(object sender, EventArgs e)
     {
        frames++;//Add to number of frames
        if (frames == 10)
        {
        frames = 0;//if it is after 10 frames, do processing and reset frames to 0
        tracker.Process(capture.QuerySmallFrame().PyrUp());
        Image<Gray, Byte> img = tracker.GetForgroundMask();
        //viewer.Image = tracker.GetForgroundMask();

        int blobs = 0, i = 0;

        MCvFont font = new MCvFont(Emgu.CV.CvEnum.FONT.CV_FONT_HERSHEY_SIMPLEX, 1.0, 1.0);
        foreach (MCvBlob blob in tracker)
        {
           i++;
           //img.Draw(Rectangle.Round(blob), new Gray(255.0), 2);
           //img.Draw(blob.ID.ToString(), ref font, Point.Round(blob.Center), new Gray(255.0));
           //Only uncomment these if you want to draw a rectangle around the blob and add text
           CurrentFrameIDs.Add(blob.ID.ToString());
           if (CurrentFrameIDs[i] == LastFrameIDs[i])
               img.Draw(Rectangle.Round(blob), new Gray(0,0), 2);//mark the new/changed blob
           blobs++;//count each blob
        }
        blobs = /*your counter here*/;
        blobs = 0; //reset 
        i = 0;
        LastFrameIDs = CurrentFrameIDs;
        CurrentFrameIDs = null;
        viewer.Image = img;//get next frame
     });
     viewer.ShowDialog();