Extract an object on a sheet of paper

Question

From pictures of tools on a sheet of paper, I'm asked to find their outline contour to vectorize them.

I'm a total beginner in computer-vision-related problems and the only thing I thought about was OpenCV and edge detection.

The result is better than what I've imagined, this is still very unreliable, especially if the source picture isn't "perfect".

I took 2 photographies of a wrench they gave me.

After playing around with opencv bindings for node, I get this:

Then, I've tried with the less-good picture:

That's totally inexploitable. I can get something a little better by changing the Canny thresold, but that must be automatized (given that the picture is relatively correct).

So I've got a few questions:

• Am I taking the right approach? Is GrabCut better for this? A combination of Grabcut and Canny edge detection? I still need vertices at the end, but I feel that GrabCut does what I want too.
• The borders are rough and have certain errors. I can augment approxPolyDP's multiplier, but without a loss of precision on good parts.
• Related to the above point, I'm thinking of integrating Savitzky-Golay algorithm to smooth the outline, instead of polygon simplification with approxPolyDP. Is it a good idea?
• Normally, the line of the outer border must form a simple, cuttable block. Is there a way in OpenCL to avoid that line to do impossible things, like passing on itself? - Or, simply, detect the problem? Those configurations are, of course, impossible but happen when the detection is failed (like in the second pic).
• I'm searching a way to do automatic Canny thresold calculation, since I must tweak it manually for each image. Do you have a good example for that?
• I noticed that converting the image to grayscale before edge detection sometimes deteriorates the result, and sometimes makes it better. Which one should I choose? (tools can be of any color btw!)

here is the source for my tests:

const cv = require('opencv');

const lowThresh = 90;
const highThresh = 90;
const nIters = 1;

const GRAY   = [120, 120, 120];
const WHITE = [255, 255, 255];

cv.readImage('./files/viv1.jpg', function(err, im) {
  if (err) throw err;

  width = im.width()
  height = im.height()
  if (width < 1 || height < 1) throw new Error('Image has no size');

  const out = new cv.Matrix(height, width);
  im.convertGrayscale();
  im_canny = im.copy();
  im_canny.canny(lowThresh, highThresh);
  im_canny.dilate(nIters);

  contours = im_canny.findContours();

  let maxArea = 0;
  let biggestContour;

  for (i = 0; i < contours.size(); i++) {
    const area = contours.area(i);
    if (area > maxArea) {
      maxArea = area;
      biggestContour = i;
    }

    out.drawContour(contours, i, GRAY);
  }

  const arcLength = contours.arcLength(biggestContour, true);
  contours.approxPolyDP(biggestContour, 0.001 * arcLength, true);

  out.drawContour(contours, biggestContour, WHITE, 5);

  out.save('./tmp/out.png');
  console.log('Image saved to ./tmp/out.png');
});

Answer 1

You might also try running your images through an adaptive threshold first. This type of binarization is fairly adept at segmenting foreground and background in cases like this, even with inconsistent lighting/shadows (which seems to be the issue in your second example above). Adathresh will require some parameter fine-tuning, but once the entire tool is segmented from the background, Canny edge detection should produce more consistent results.

As for the roughness in your contours, you could try setting your findContours mode to one of the CV_CHAIN_APPROX methods described here.

Answer 2

You'll need to add some pre-processing to clean up the image. Because you have a large variation in intensities in the image because of shadow, poor lighting, high shine on tools, etc you should equalize the image. This will help you get a better response in the regions that are currently poorly lit or have high shine.

Here's an opencv tutorial on Histogram equalization in C++: http://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/histogram_equalization/histogram_equalization.html

Hope this helps

EDIT: You can have an automatic threshold based on some loss function(?). For eg: If you know that the tool will be completely captured in the frame, you know that you should get a high value at every column from x = 10 to x = 800(say). You could then keep reducing the threshold until you get a high value at every column from x = 10 to x = 800. This is a very naive way of doing it, but its an interesting experiment, I think, since you are generating the images yourself and have control over object placement.