Augmented Reality line that moves with an image in OpenCV

Question

I want to do the following things in openCV. The problem statement that I have is with a bottle, which needs a line on the image and the line needs to rotate as per the movement of bottle.

The first image needs to have red lines as the borders and initiate a green line

The second image needs to have the green line in the middle when the bottle gets rotated. That is the green line has to follow the rotation of the bottle

Finally as per the third image, the application needs to kill itself or save the picture when the green line gets aligned to the red line

I tried doing this in OpenCV using template matching. I tried keeping a template image and then tracking the template image using template matching algorithm. But it does not seem to work properly in this case.

import cv2
from time import sleep
import numpy as np

vid = cv2.VideoCapture(0)
sleep(2)
line_show = False
save_reference = False
template_compare_method = cv2.TM_SQDIFF_NORMED
i = 0

while True:

    check, frame = vid.read()

    print(check)
    frame1 = cv2.line(frame, (500, 0), (500, 720), (255, 0, 0), 7)
    frame1 = cv2.line(frame1, (800, 0), (800, 720), (255, 0, 0), 7)

    if line_show:
        h, w = frame1.shape[:2]
        if not save_reference:
            reference = frame1[200:500, 780:790]
            cv2.imwrite("../../images/white_image.jpg", reference)
            save_reference = True
        if save_reference:
            reference_image = cv2.imread('../../images/white_image.jpg')
            result = cv2.matchTemplate(reference_image, frame1, template_compare_method)
            mn, _, mnLoc, _ = cv2.minMaxLoc(result)
            MPx, MPy = mnLoc
            trows, tcols = reference_image.shape[:2]
            frame1 = cv2.rectangle(frame1, (MPx, MPy), (MPx+tcols, MPy+trows), (0, 0, 255), 2)

    cv2.imshow("image", frame1)
    key = cv2.waitKey(1)
    if key == ord('l'):
        line_show = True
    if key == ord('k'):
        cv2.imwrite("../../images/saved_image_"+str(i)+".jpg", frame1)
        i = i + 1
    if key == ord('s'):
        cv2.imwrite("../../images/saved_image.jpg", frame)
        vid.release()
        print("Image saved")
        break

    elif key == ord('q'):
        vid.release()
        cv2.destroyAllWindows()
        break

Can I use any other algorithms, or am I approaching this problem in a wrong way by looking it as a object tracking task, where I save a small image and track it through template matching ? Can I use some other algorithms like Meanshift, Frame Difference etc. to achieve this ?

Answer 1

If I were you, I would solve this problem using line algorithm. Of course, you can choose any other robust algorithm. My idea is to solve the problem as quickly as possible.

Assume I have the following image with left and right boundaries (blue), and I have the green-line.

When green-line passes the left-border, quit.

• 1. Tracking the green-line

     • First you need to find the features of the frame to track efficiently the green-line.

       while True:
           ret, frm = cap.read()
           frm_gry = cv2.cvtColor(frm, cv2.COLOR_BGR2GRAY)
           frm_cny = cv2.Canny(frm_gry, 50, 200)
       

       Sample output:

       

     • Second, find the approximate length of the green-line:

       • There is no direct way to find the length, do error-trial calculation.

       • Once you are sure, initialize the line algorithm.

         lns = cv2.ximgproc.createFastLineDetector(_length_threshold=400).detect(frm_cny)
         

     • Third, get the coordinates, and check if the green-line is in the border.

       if lns is not None:
           for ln in lns:
               x1 = int(ln[0][0])
               y1 = int(ln[0][1])
               x2 = int(ln[0][2])
               y2 = int(ln[0][3])
       
       if x1 <= 232:
           break 
       

Code:

import cv2

cap = cv2.VideoCapture("sample.mp4")

while True:
    ret, frm = cap.read()

    if ret:
        rgt_bdr = cv2.line(frm, (794, 250), (794, 1250), (255, 0, 0), 7)
        lft_bdr = cv2.line(frm, (232, 250), (232, 1250), (255, 0, 0), 7)

        frm_gry = cv2.cvtColor(frm, cv2.COLOR_BGR2GRAY)
        frm_cny = cv2.Canny(frm_gry, 50, 200)

        lns = cv2.ximgproc.createFastLineDetector(_length_threshold=400).detect(frm_cny)

        if lns is not None:
            for ln in lns:
                x1 = int(ln[0][0])
                y1 = int(ln[0][1])
                x2 = int(ln[0][2])
                y2 = int(ln[0][3])

                cv2.line(frm,
                         pt1=(x1, y1),
                         pt2=(x2, y2),
                         color=(0, 255, 0),
                         thickness=3)

                print("({}, {})-({}, {})".format(x1, y1, x2, y2))

                if x1 <= 232:
                    break

                cv2.imshow("frm", frm)
                cv2.waitKey(1)