Detecting hand and segmenting image in Android application

Question

I have developed an android application which captures and processes images. I would like to add the following feature:

• Detect where a user's hand is in an image. Specifically, the app will use this to approximate where the user is pointing.
• Segment the image so that only the part of the image with the user's hand remains. It would be even better if this wasn't just some predefined radius, but instead it segmented using some object near the user's hand.

It seems like Android OpenCV could work, but I'm not sure if it is a) powerful enough to do this and b) how to use it for this feature.

Answer 1

This is a very broad question, but let my try give some suggestions.

Generally you can see this as a pattern detection problem. The crucial thing is to define what pattern exaclty means in your situation, ie what kind of hand gestures you would like to detect on an image? If your goal is to detect a hand in general and you can assume specific gesture which will be detected, I would suggest hand with straight closed fingers:

And here's an explanation why. This pattern is practical detectible for one good reason: it has an internal feature which is always observed in that pattern and this feature is invariant to the background. The feature is "lines" between fingers. As you can observe, with the closed straight fingers there are couple of straight, parallel gradient vectors to be detected:

They are invariant to the background. Especially, you cannot rely on the detecting borders of the hand since the gradient there is dependent to the background. I think that method will be good also for different skin colors as the feature is clearly a gradient, which is irrelevent to the skin color.

In the past I've been working on such haar classificator which I trained on such examples. The results were satisfactory good. I trained my classificator on a few hundreds positive and a few thousands negative patterns.