iOS SceneKit to UIView projection issue

Question

I have a SCNView with SCNNode of SCNPlane geometry in front of my camera root node.

Over SCNView, in UIView, I am adding UIImages (Markers) - orange circles.

In Motion listener I am trying to position the Markers in a way so they will stick to the center of each edge of the Plane.

A proper Markers alignment - while device is in straight position:

I'm doing this using projection from SceneKit objects to UIView:

//world coordinates
let v1w =  sm.node.convertPosition(sm.node.boundingBox.min, 
    to: self.sceneView.scene?.rootNode)
let v2w =  sm.node.convertPosition(sm.node.boundingBox.max, 
    to: self.sceneView.scene?.rootNode)

//projected coordinates
let v1p = self.sceneView.projectPoint(v1w)
let v2p = self.sceneView.projectPoint(v2w)

//frame rectangle
let rect = CGRect.init(x: CGFloat(v1p.x), y: CGFloat(v2p.y), 
    width: CGFloat(v2p.x - v1p.x), height: CGFloat(v1p.y - v2p.y))

var frameOld = sm.marker.frame

switch sm.position
{
case .Top:
    frameOld.origin.y = rect.minY - frameOld.size.height/2
    frameOld.origin.x = rect.midX - frameOld.size.width/2
case .Bottom:
    frameOld.origin.y = rect.maxY - frameOld.size.height/2
    frameOld.origin.x = rect.midX - frameOld.size.width/2
case .Left:
    frameOld.origin.y = rect.midY - frameOld.size.height/2
    frameOld.origin.x = rect.minX - frameOld.size.width/2
case .Right:
    frameOld.origin.y = rect.midY - frameOld.size.height/2
    frameOld.origin.x = rect.maxX - frameOld.size.width/2
}

sm.marker.frame = frameOld
self.view.layoutSubviews()

Similar approach you can find here: Scene Kit: projectPoint calculated is displaced

So I would like those Markers to stick to the edges of the Plane during device motion. But there is the issue: when rotating device - Markers are drifting from the Plane edges

See video of an issue: https://youtu.be/XBgNDDX5ZI8

I have created a basic project on github to reproduce an issue: https://github.com/mgontar/SceneKitProjectionIssue

Answer 1

To make proper projections you should calculate mid points in world coordinates and then project points to scene view

        let min = node.boundingBox.min
        let max = node.boundingBox.max
        
        //world coordinates
        var mp1w = SCNVector3Make((min.x+max.x)/2, max.y, (min.z+max.z)/2) //top
        var mp2w = SCNVector3Make((min.x+max.x)/2, min.y, (min.z+max.z)/2) //bottom
        var mp3w = SCNVector3Make(min.x, (min.y+max.y)/2, (min.z+max.z)/2) //left
        var mp4w = SCNVector3Make(max.x, (min.y+max.y)/2, (min.z+max.z)/2) //right
        
        mp1w = node.convertPosition(mp1w, to: self.sceneView.scene.rootNode)
        mp2w = node.convertPosition(mp2w, to: self.sceneView.scene.rootNode)
        mp3w = node.convertPosition(mp3w, to: self.sceneView.scene.rootNode)
        mp4w = node.convertPosition(mp4w, to: self.sceneView.scene.rootNode)

        //projected coordinates
        let mp1p = self.sceneView.projectPoint(mp1w) //top
        let mp2p = self.sceneView.projectPoint(mp2w) //bottom
        let mp3p = self.sceneView.projectPoint(mp3w) //left
        let mp4p = self.sceneView.projectPoint(mp4w) //right

Answer 2

The problem here is that CGRect you use to calculate the mid points is based on the projected coordinates of the bounding box. The two corner points of the bounding box are transformed using the model view projection matrix, to get the correct view space coordinates for the mid points you need to perform the same transformation.

Hopefully the code is a bit clearer.

//world coordinates
let v1w =  sm.node.convertPosition(sm.node.boundingBox.min, to: self.sceneView.scene?.rootNode)
let v2w =  sm.node.convertPosition(sm.node.boundingBox.max, to: self.sceneView.scene?.rootNode)

//calc center of BB in world coordinates
let center = SCNVector3Make(
    (v1w.x + v2w.x)/2,
    (v1w.y + v2w.y)/2,
    (v1w.z + v2w.z)/2)

//calc each mid point
let mp1w = SCNVector3Make(v1w.x, center.y, center.z)
let mp2w = SCNVector3Make(center.x, v2w.y, center.z)
let mp3w = SCNVector3Make(v2w.x, center.y, center.z)
let mp4w = SCNVector3Make(center.x, v1w.y, center.z)

//projected coordinates
let mp1p = self.sceneView.projectPoint(mp1w)
let mp2p = self.sceneView.projectPoint(mp2w)
let mp3p = self.sceneView.projectPoint(mp3w)
let mp4p = self.sceneView.projectPoint(mp4w)

var frameOld = sm.marker.frame

switch sm.position
{
case .Top:
    frameOld.origin.y = CGFloat(mp1p.y) - frameOld.size.height/2
    frameOld.origin.x = CGFloat(mp1p.x) - frameOld.size.width/2
    sm.marker.isHidden = (mp1p.z < 0 || mp1p.z > 1)
case .Bottom:
    frameOld.origin.y = CGFloat(mp2p.y) - frameOld.size.height/2
    frameOld.origin.x = CGFloat(mp2p.x) - frameOld.size.width/2
    sm.marker.isHidden = (mp2p.z < 0 || mp2p.z > 1)
case .Left:
    frameOld.origin.y = CGFloat(mp3p.y) - frameOld.size.height/2
    frameOld.origin.x = CGFloat(mp3p.x) - frameOld.size.width/2
    sm.marker.isHidden = (mp3p.z < 0 || mp3p.z > 1)
case .Right:
    frameOld.origin.y = CGFloat(mp4p.y) - frameOld.size.height/2
    frameOld.origin.x = CGFloat(mp4p.x) - frameOld.size.width/2
    sm.marker.isHidden = (mp4p.z < 0 || mp4p.z > 1)
}

It's a cool little sample project!

Update on z-clipping issue

The projectPoint method returns a 3D SCNVector, the x any y coords as we know are the screen coordinates. The z coordinate tells us the location of the point relative to the far and near clipping planes (z = 0 near clipping plane, z = 1 far clipping plane). If you set a negative value for your near clipping plane, objects behind the camera would be rendered. We don't have a negative near clipping plane, but we also don't have any logic to say what happens if those projected point locations fall outside the z far and z near range.

I've updated the code above to include this zNear and zFar check and toggle the UIView visibility accordingly.

tl;dr

The markers visible when the camera was rotated 180deg are behind the camera, but they were still projected onto the view plane. And as we weren't checking if they we behind the camera, they were still displayed.