OpenCV : Object detection and tracking based on feature detection

Question

I have just ventured into computer vision and trying to demystify various intricacies of it. I am trying to augment the kalman filter with a surf feature detector. But I do not understand how to call and use the kalman method after the homography and bounded rectangle has been constructed on the detected frame using surf features. I have detected the features and extracted the keypoints using a reference image after comparing with the incoming frames. Then I used the flann matcher.

Now, is it feasible to use the kalman filter since I want to track the motion and get the predicted motion. I have searched a lot but have not found that surf features can be used with kalman filter. All I am getting are suggestions to use cvBlobs for tracking. But, theoretically kalman filters are used for tracking purposes. However, I am confused since the several implementations of video based tracking using surf shows that surf can be used for tracking in itself. But my question is

• If kalman filter cannot be used with surf, then how do I implement the moments to get the coordinate measurement values as I need the information for motion prediction.

• can surf be used with kalman filter for tracking and if yes how to implement it after the object has been detected and bounded with a rectangle using the following code.

  Example : the object to be tracked . Some frames

  /* Object Detection and recognition from video*/

  int main() { Mat object = imread( "book1.png",CV_LOAD_IMAGE_GRAYSCALE );

  if( !object.data )
  {
      std::cout<< "Error reading object " << std::endl;
      return -1;
  }
  
  //Detect the keypoints using SURF Detector
  int minHessian = 500;
  
  SurfFeatureDetector detector( minHessian );
  std::vector<KeyPoint> kp_object;
  
  detector.detect( object, kp_object );
  
  //Calculate descriptors (feature vectors)
  SurfDescriptorExtractor extractor;
  Mat des_object;
  
  extractor.compute( object, kp_object, des_object );
  
  FlannBasedMatcher matcher;        
  
  namedWindow("Good Matches");
  namedWindow("Tracking");
  
  std::vector<Point2f> obj_corners(4);
  
  //Get the corners from the object
  obj_corners[0] = cvPoint(0,0);
  obj_corners[1] = cvPoint( object.cols, 0 );
  obj_corners[2] = cvPoint( object.cols, object.rows );
  obj_corners[3] = cvPoint( 0, object.rows );
  
  char key = 'a';
  int framecount = 0;
  VideoCapture cap("booksvideo.avi");
  
  for(; ;)
  {
      Mat frame;
      cap >> frame;
      imshow("Good Matches", frame);
  
  
      Mat des_image, img_matches;
      std::vector<KeyPoint> kp_image;
      std::vector<vector<DMatch > > matches;
      std::vector<DMatch > good_matches;
      std::vector<Point2f> obj;
      std::vector<Point2f> scene;
      std::vector<Point2f> scene_corners(4);
      Mat H;
      Mat image;
  
      //cvtColor(frame, image, CV_RGB2GRAY);
  
      detector.detect( image, kp_image );
      extractor.compute( image, kp_image, des_image );
  
      matcher.knnMatch(des_object, des_image, matches, 2);
  
      //THIS  LOOP IS SENSITIVE TO SEGFAULTS
      for(int i = 0; i < min(des_image.rows-1,(int) matches.size()); i++) 
      {
          if((matches[i][0].distance < 0.6*(matches[i][4].distance)) && ((int) matches[i].size()<=2 && (int) matches[i].size()>0))
          {
              good_matches.push_back(matches[i][0]);
          }
      }
  
      //Draw only "good" matches
      drawMatches( object, kp_object, image, kp_image, good_matches, img_matches, Scalar::all(-1), Scalar::all(-1), vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );
  
      if (good_matches.size() >= 4)
      {
          for( int i = 0; i < good_matches.size(); i++ )
          {
              //Get the keypoints from the good matches
              obj.push_back( kp_object[ good_matches[i].queryIdx ].pt );
              scene.push_back( kp_image[ good_matches[i].trainIdx ].pt );
          }
  
          H = findHomography( obj, scene, CV_RANSAC );
  
          perspectiveTransform( obj_corners, scene_corners, H);
  
          //Draw lines between the corners (the mapped object in the scene image )
          line( img_matches, scene_corners[0] + Point2f( object.cols, 0), scene_corners[1] + Point2f( object.cols, 0), Scalar(0, 255, 0), 4 );
          line( img_matches, scene_corners[1] + Point2f( object.cols, 0), scene_corners[2] + Point2f( object.cols, 0), Scalar( 0, 255, 0), 4 );
          line( img_matches, scene_corners[2] + Point2f( object.cols, 0), scene_corners[3] + Point2f( object.cols, 0), Scalar( 0, 255, 0), 4 );
          line( img_matches, scene_corners[3] + Point2f( object.cols, 0), scene_corners[0] + Point2f( object.cols, 0), Scalar( 0, 255, 0), 4 );
  
      }
  
      //Show detected matches
      imshow( "Good Matches", img_matches );
      for( int i = 0; i < good_matches.size(); i++ )
      { 
          printf( "-- Good Match [%d] Keypoint 1: %d  -- Keypoint 2: %d  \n", i,    good_matches[i].queryIdx, good_matches[i].trainIdx ); 
      }
  
      waitKey(0);
  

  }

  return 0;

  }

Answer 1

I haven't seen the combination of feature matching and filtering that you are describing. One idea I had is to keep track of the center of mass (and size) with a kalman filter and use that information to mask out the exterior region before running the feature matching on the next frame. I'm not sure what your constraints are, but you might consider a template matching or camshift-type tracking that could also use a kalman filter for helping with search.