How to add UIImage to a Scene in AR Kit

Question

I have AR session which adds SCNText and 3D Objects as well. And now, I want to add UIImage from Image Picker and don't know how to do this. Is there any solutions?

SOLUTION

func insertImage(image: UIImage, width: CGFloat = 0.3, height: CGFloat = 0.3) -> SCNNode {
    let plane = SCNPlane(width: width, height: height)
    plane.firstMaterial!.diffuse.contents = image
    let node = SCNNode(geometry: plane)
    node.constraints = [SCNBillboardConstraint()]
    return node
}

let image = insertImage(image: addedImage)
node.addChildNode(image)

Answer 1

As I am sure you are aware an SCNGeometry has a materials property which is simply:

A container for the color or texture of one of a material’s visual properties.

As such you could render a UIImage onto an SCNGeometry using for example the diffuse property.

Here is a fully working and tested example. Which loads a UIImagePickerController after 5 seconds, and then creates an SCNNode with an SCNPlane Geometry which has is contents set to the selected UIImage.

The code is fully commented so it should be easy enough to understand:

//-------------------------------------
//MARK: UIImagePickerControllerDelegate
//-------------------------------------

extension ViewController: UIImagePickerControllerDelegate{

    func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any]) {

        //1. Check We Have A Valid Image
        if let selectedImage = info[UIImagePickerControllerOriginalImage] as? UIImage {

            //2. We Havent Created Our PlaneNode So Create It
            if planeNode == nil{

                //d. Dismiss The Picker
                picker.dismiss(animated: true) {

                    //a. Create An SCNPlane Geometry
                    let planeGeometry = SCNPlane(width: 0.5, height: 0.5)

                    //b. Set's It's Contents To The Picked Image
                    planeGeometry.firstMaterial?.diffuse.contents = self.correctlyOrientated(selectedImage)

                    //c. Set The Geometry & Add It To The Scene
                    self.planeNode = SCNNode()
                    self.planeNode?.geometry = planeGeometry
                    self.augmentedRealityView.scene.rootNode.addChildNode(self.planeNode!)
                    self.planeNode?.position = SCNVector3(0, 0, -1.5)

                }
            }

        }

         picker.dismiss(animated: true, completion: nil)
    }

    func imagePickerControllerDidCancel(_ picker: UIImagePickerController) { picker.dismiss(animated: true, completion: nil) }

}

class ViewController: UIViewController, UINavigationControllerDelegate {

    //1. Create A Reference To Our ARSCNView In Our Storyboard Which Displays The Camera Feed
    @IBOutlet weak var augmentedRealityView: ARSCNView!

    //2. Create Our ARWorld Tracking Configuration & Session
    let configuration = ARWorldTrackingConfiguration()
    let augmentedRealitySession = ARSession()

    //3. Create A Reference To Our PlaneNode
    var planeNode: SCNNode?
    var planeGeomeryImage: UIImage?

    //---------------
    //MARK: LifeCycle
    //---------------

    override func viewDidLoad() {
        super.viewDidLoad()

        //1. Setup The Session
        setupARSession()

        //2. Show The UIImagePicker After 4 Seconds
        DispatchQueue.main.asyncAfter(deadline: .now() + 4) {
            self.selectPhotoFromGallery()
        }

    }

    override func didReceiveMemoryWarning()  { super.didReceiveMemoryWarning() }

    //-------------
    //MARK: ARSetup
    //-------------

    func setupARSession(){

        //1. Run Our Session
        augmentedRealityView.session = augmentedRealitySession
        augmentedRealitySession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
    }

    //---------------------
    //MARK: Image Selection
    //---------------------

    /// Loads The UIImagePicker & Allows Us To Select An Image
    func  selectPhotoFromGallery(){

        if UIImagePickerController.isSourceTypeAvailable(UIImagePickerControllerSourceType.photoLibrary){
            let imagePicker = UIImagePickerController()
            imagePicker.delegate = self
            imagePicker.allowsEditing = true
            imagePicker.sourceType = UIImagePickerControllerSourceType.photoLibrary
            self.present(imagePicker, animated: true, completion: nil)
        }

    }


    /// Correctly Orientates A UIImage
    ///
    /// - Parameter image: UIImage
    /// - Returns: UIImage?
    func correctlyOrientated(_ image: UIImage) -> UIImage {
        if (image.imageOrientation == .up) { return image }

        UIGraphicsBeginImageContextWithOptions(image.size, false, image.scale)
        let rect = CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height)
        image.draw(in: rect)

        let normalizedImage = UIGraphicsGetImageFromCurrentImageContext()!
        UIGraphicsEndImageContext()

        return normalizedImage
    }

}

Don't forget to add the NSPhotoLibraryUsageDescription to your info.plist:

<key>NSPhotoLibraryUsageDescription</key>
<string>For ARkit</string>

This should be more than enough to get you started...