CODEWITHSUNDEEP
augmented-realityscenekitarkitarcorerealitykit
Yaro
Yaro

Reputation: 148

Augmented Reality – Lighting Real-World objects with Virtual light

Is it possible to import a virtual lamp object into the AR scene, that projects a light cone, which illuminates the surrounding space in the room and the real objects in it, e.g. a table, floor, walls?

For ARKit, I found this SO post.

For ARCore, there is an example of relighting technique. And this source code.

I have also been suggested that post-processing can be used to brighten the whole scene.

However, these examples are from a while ago and perhaps threre is a newer or a more straight forward solution to this problem?

Upvotes: 2

Views: 1041

Answers (2)

闪电狮
闪电狮

Reputation: 556

At the low level, RealityKit is only responsible for rendering virtual objects and overlaying them on top of the camera frame. If you want to illuminate the real scene, you need to post-process the camera frame.

Here are some tutorials on how to do post-processing: Tutorial1⃣️ Tutorial2⃣️

If all you need is an effect like This , then all you need to do is add a CGImage-based post-processing effect for the virtual object (lights).

More specifically, add a bloom filter to the rendered image(You can also simulate bloom filters with Gaussian blur).

In this way, the code is all around UIImage and CGImage, so it's pretty simple😎

If you want to be more realistic, consider using the depth map provided by LiDAR to calculate which areas can be illuminated for a more detailed brightness.

Or If you're a true explorer, you can use Metal to create a real world Digital Twin point cloud in real time to simulate occlusion of light.

Upvotes: 1

Andy Jazz
Andy Jazz

Reputation: 58153

There's nothing new in relighting techniques based on 3D compositing principles in 2021. At the moment, when you're working with RealityKit or SceneKit, you have to personally implement the relighting functionality with the help of two additional render passes (RGB pass is always needed) - Normals pass and PointPosition pass. Both AOVs must be 32-bit.

enter image description here

However, in the near future, when Apple engineers finally implement texture capturing in Scene Reconstruction – any inexperienced AR developer will be able to apply a relighting procedure.

Watch this Vimeo Video to find out how relighting can be achieved in The Foundry NUKE.

A crucial point here, when implementing the Relighting effect, is the presence of a LiDAR scanner (or iToF sensor if you're using ARCore). In other words, today's relighting solution for iOS is Metal + RealityKit.

Upvotes: 1

Related Questions