CODEWITHSUNDEEP
augmented-realityarkitarcore
jiawen
jiawen

Reputation: 1208

How to estimate ambient light intensity and color from camera image for Augmented reality application?

ARKit and ARCore has the feature to estimate the ambient light intensity and color for realistic rendering.

ARKit: https://developer.apple.com/documentation/arkit/arlightestimate?language=objc

ARCore: https://developers.google.com/ar/reference/java/arcore/reference/com/google/ar/core/LightEstimate#getColorCorrection(float[],%20int)

They both expose an ambient intensity and an ambient color. In ARKit, the color is in degrees kelvin while in ARCore it is RGB color correction.

Question 1: What's the difference between kelvin and color correction and how can they be applied to rendering?

Question 2: What's the algorithm to estimate the light intensity and color from camera frames? Are there existing code or research papers we can refer to if we want to implement it ourselves?

Upvotes: 1

Views: 1207

Answers (3)

Ivan Tishchenko
Ivan Tishchenko

Reputation: 106

Question 2: What's the algorithm to estimate the light intensity and color from camera frames? Are there existing code or research papers we can refer to if we want to implement it ourselves?

Unfortunately, neither ARCore nor ARKit disclose which algorithms they use for light estimation. Judging from the publicly available sources, it seems that they could estimate an equirectangular HDRI map using a deep neural network:

In any case, it could be helpful to search for recent literature on light estimation and select the approach relevant for your usecase. Here is an example list: https://github.com/waldenlakes/Awesome-Illumination-Estimation.

Upvotes: 0

Rom4in
Rom4in

Reputation: 562

Providing you have added a light node called 'light' with a SCNLight attached to it in your "ship.scn" SCNScene and that your ViewController conforms to ARSessionDelegate so you can get the light estimate per frame: 

class ViewController: UIViewController, ARSCNViewDelegate, ARSessionDelegate {

@IBOutlet var sceneView: ARSCNView!

override func viewDidLoad() {
    super.viewDidLoad()
    sceneView.delegate = self
    let scene = SCNScene(named: "art.scnassets/ship.scn")!
    sceneView.scene = scene
}

override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    let configuration = ARWorldTrackingConfiguration()
    configuration.isLightEstimationEnabled = true
    sceneView.session.run(configuration)
    sceneView.session.delegate = self
}

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    guard let lightEstimate = frame.lightEstimate,
    let light = sceneView.scene.rootNode.childNode(withName: "light", recursively: false)?.light else {return}
    light.temperature = lightEstimate.ambientColorTemperature
    light.intensity = lightEstimate.ambientIntensity
}

}

As a result, if you dim the lights in your room, SceneKit will dim the virtual light too .

ARKit light estimation

Upvotes: 1

mulan
mulan

Reputation: 176

Question 2 for ARCore: Here is a research paper on How Environmental HDR works

Here is a short summary about environmental HDR in ARCore + Sceneform

Hope it helps you in your search :)

Upvotes: 2

Related Questions