CODEWITHSUNDEEP
augmented-realityarcore
SZwinsor
SZwinsor

Reputation: 171

Is ARCore object recognition possible?

My goal is to overlay material/texture on a physical object (it would be an architectural model) that I would have an identical 3d model of. The model would be static (on a table if that helps), but I obviously want to look at the object from any side. The footprint area of my physical models would tend to be no smaller than 15x15cm and could be as large as 2-3m^2, but I would be willing to change the size of the model to work with ARCore's capability.

I know ARCore is mainly designed to anchor digital objects to flat horizontal planes. My main question is, in its current state, is it capable of accompliahing my end goal? If i have this right, it would record physical point cloud data and attempt to match it to point cloud data of my digital model, then overlapping the two on the phone screen?

If that really isn't what ARCore is for, is there an alternative that I should be focusing on? In my head this sounded fairly straightforward, but I'm sure I'll get way out of my depth if I go about it an inefficient way. Speaking of depth, I would prefer not to use a depth sensor, since my target devices are phones.

Upvotes: 17

Views: 16931

Answers (2)

Andy Jazz
Andy Jazz

Reputation: 58583

Updated: 11th May, 2023.

Scene Semantics API

At the moment there's still no customizable 3D Object Recognition API in Google ARCore. However, a brand-new Scene Semantics API, which is a part of ARCore 1.37, allows you to automatically recognize eleven types of such outdoor scene components as:

  • MAIN COMPONENTS – sky, building, tree, road, vehicle

  • MAJOR COMPONENTS – sidewalk, terrain, structure, water

  • MINOR COMPONENTS – object, person

Scene Semantics runs an ML model on the camera image feed and provides a semantic image with each pixel corresponding to one of 11 labels of outdoor concepts.

In addition to the above, you can use ML Kit framework and Augmented Images API for various tasks. And, according to Google's documentation, you can use ARCore as input for ML models.

Upvotes: 7

PhilLab
PhilLab

Reputation: 5027

I most definitely hope that it will be possible in the future - after all an AR toolkit without Computer Vision is not that helpful.

Unfortunately, according to the ARCore employee Ian, this is currently not directly supported but you could try to access the pixels via glReadPixels and then use OpenCV with these image bytes.

Quote from Ian:

I can't speak to future plans, but I agree that it's a desirable capability. Unfortunately, my understanding is that current Android platform limitations prevent providing a single buffer that can be used as both a GPU texture and CPU-accessible image, so care must be taken in providing that capability.

Upvotes: 9

Related Questions