How to Detect Physical object in Android Augmented Reality?

Question

I found many ways to detect different shapes. But hard luck when I am going for a physical object. From what I read we should have a black border around images to make a pattern file. If I follow this concept and generate a pattern then my application detects images on printout. But in the real world a physical object not necessarily has a black border square shape around it.

Update

Although I accept an answer, my question remains unsolved. As there is still no solution for detecting physical object.

Any further research and links are welcome!

Answer 1

Since 2018 you can detect objects in Android apps & ARCore apps.

In May 2018 Google announced a mobile SDK – ML Kit, that's a machine learning API for Android and iOS developers that you can use along with ARCore SDK (what now has Augmented Images API and Augmented Faces API). At the moment it's still in beta stage but it's got the following working features:

Google ML Kit engineers say about it the following:

ML Kit brings Google's machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you're new or experienced in machine learning, you can implement the functionality you need in just a few lines of code. There's no need to have deep knowledge of neural networks or model optimization to get started. On the other hand, if you are an experienced ML developer, ML Kit provides convenient APIs that help you use your custom TensorFlow Lite models in your mobile apps.

Answer 2

Have you considered OpenCV for detecting and tracking object? Take a look at Find Objects with a Webcam tutorial (C++ / Qt). OpenCV is available for Android - OpenCV 4 Android.

Answer 3

The bad news is, you can't use AndAR to detect physical objects. AndAR is based on a fiducial marker approach, where the marker is made of two components: a solid border and an interior pattern. The pattern encodes a value that can be used to address a particular model to render on the marker, and the border makes it easy to determine the relative orientation of the marker to the device. Clearly this is just planar image recognition.

To do object recognition on a 3D object is a more complicated problem, and I don't know of any Android libraries that provide a turn-key solution, but recognizing just one object is probably feasible on a mobile device.

One possibility might be to investigate the available Android AR toolkits (Layar, Junaio, Qualcomm AR SDK) which all now support some image recognition. It may be that by taking images of your teapot at various rotations and using those as the images you want your app to match against that you might get this solution working, but keep in mind they are only designed to do planar matching on images, not real 3D objects, so the performance might not be great. (Well, the Metaio Mobile SDK Pro does 3D recognition and tracking, but it's very expensive).

While object recognition is perhaps best done by comparing camera frames with images of the object you wish to recognize (or by comparing image features from the camera frames with pre-computed image features etc), tracking is a different matter. If you want to accurately track your 3D object in 3D space you'll really need to have or build a 3D model of it, and for each frame determine point correspondences between the camera image and the 3D object for tracking. True unassisted (i.e. no depth-camera) 3D tracking is hard.

I hope this gives you some background you can use to evaluate your next steps.

Update: Qualcomm's Vuforia SDK allows you to track "multi targets", which are objects with a set of planar tracking surfaces with a fixed spatial relationship. If you made a "cube" different photos of the 6 sides of your object (teapot) that might work somewhat. https://ar.qualcomm.at/qdevnet/developer_guide/Trackables

End of 2013 Update:

I have no experience with these, but:

Metaio now offer 3D tracking of CAD models: https://dev.metaio.com/sdk/tutorials/3d-tracking-based-on-cad-data/

ARLab.com's LinkAR promises object matching. http://www.arlab.com/objecttracking

I would note the use of the word "matching" - I think the use case here is you know the object you want to overlay (a toy-box, and engine etc). Differentiating between multiple 3D objects may be entirely out of scope.

Answer 4

I recently read about research being doing on hierarchical shape vocabularies used for object representation. Of course, there is no library available for download, but in case you are interested in the general approach here you can find some papers.

Also you might be interested in this paper. It describes an algorithm for detecting objects based on a set of contours.