Want to find rotation and position in perspective world of plane from 2d points in image

Question

I have 2D png image

I want to place a 3D plane at a perfect rotation and position of the sofa as shown in figure

I am using three.js and a perspective camera to manually add the rotation and translation to the plane.

I have the center and corner coordinates of the area where the green plane is placed. I have initial 3D coordinates position - (0,0,0) and rotation - (0,0,0). I have initial points of Object (0,0),(0,100),(140,0),(1 like

I want to transform initial frame to final frame, using OpenCV and three.js. How to find the angle of the green plane in the final frame from that 2D image to rotate and translate the green plane to the white plane?

Our end goal is to place a 3D object on a plane like this -

I tried to solve the same using OpenCV solvePNP but didn't get the desired result it's rotation and translation vector are not working and it is giving the wrong result.

Answer 1

An example using OpenCV calculate homography and wrap image to select region in python.

First, load your 2D png image and banner image, the banner image will replace the white region.

import cv2
import numpy as np

img = cv2.imread("2D png image.png")
banner = cv2.imread("replace region img.jpg")

Then, hand selects four corners(sofa corners in your image) to define replace a region.

#point sequence:top_left corner, top right corner, bottom left corner, bottom left corner
pt = np.array([[65,180], [122, 192], [122, 277], [67, 251]]) 

Next, get four corners of our banner image.

#point sequence same as hand select :top_left corner, top right corner, bottom left corner, bottom left corner
pts_banner = np.array([[0, 0], [banner.shape[1] - 1, 0], [banner.shape[1] - 1, banner.shape[0] - 1], [0, banner.shape[0] - 1]])

Next using OpenCV calculate the homography matrix and wrap image to the replace region.

homographyMat, status = cv2.findHomography(pts_banner, pt)
result1 = cv2.warpPerspective(banner, homographyMat, (img.shape[1], img.shape[0]))

Finally, blackout select region and add wrap image to input image.

cv2.fillConvexPoly(img, pt, (0,0,0))
result2 = img + result1
cv2.imwrite("result.png", result2)

banner image:

output:

Reference:

https://anishdubey.com/virtual-billboard-homography-perspective-geometric-transformation-image-opencv