CODEWITHSUNDEEP
pythonopencvimage-processingcomputer-vision
Darina
Darina

Reputation: 1632

Detecting overlay in video with python

I am working with frames from a video. The video is overlaid with several semi-transparent boxes and my goal is to find the coordinates of these boxes. These boxes are the only fixed points in the video - the camera is moving, color intensity changes, there is no fixed reference. The problem is that the boxes are semi-transparent, so they also change with the video, albeit not as much. It seems that neither background substraction nor tracking have the right tools for this problem.

Nevertheless, I've tried the background substractors that come with cv2 as well as some homebrewn methods using differences between frames and thresholding. Unfortunately, these don't work due to the box transparency.

For reference, here is what the mean difference between the first 50 frames looks like:

mean difference between frames

And here is what cv2 background subtractor KNN returns:

background subtractor result

I've experimented with thresholds, number of frames taken into account, various contouring algorithms, blurring/sharpening/etc. I've also tried techniques from document layout analysis.

I wonder if maybe there is something I'm missing due to not knowing the right keyword. I don't expect anyone here to give me the perfect solution, but any pointers as to where to look/what approach to try, are appreciated. I'm not bound to cv2 either, anything that works in python will do.

Upvotes: 4

Views: 1100

Answers (2)

Jakub
Jakub

Reputation: 184

Ok here is first step, you can make Canny from that image, from canny you can make countours:

import cv2
import random as rng

image = cv2.imread("c:\stackoverflow\interface.png")


edges = cv2.Canny(image, 100, 240)
contoursext, hierarchy = cv2.findContours(
    edges, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) 
#cv2.RETR_EXTERNAL would work better if the image would not be framed.

for i in range(len(contoursext)):
    color = (rng.randint(0,256), rng.randint(0,256), rng.randint(0,256))
    cv2.drawContours(image, contoursext, i, color, 1, cv2.LINE_8, hierarchy, 0)
    # Show in a window



cv2.imshow("Canny", edges)
cv2.imshow("Contour", image)

cv2.waitKey(0)

Canny And Contours

Then you can test if the contour or combination of 2 contours is rectangles for example...wich would probably detect most of the rectangle overlays...

Or Also you can try to detect canny lines if they are similar to rectangles.

Upvotes: 0

Colim
Colim

Reputation: 1303

If you take a sample of random frames as elements of an array, and calculate the FFT, all the semi-transparent boxes will have a very high signal, and the rest of the pixels would behave as noise, so noise remotion will filter away the semi-transparent boxes. You can add the result of your other methods as additional frames for the fft

You are trying to find something that does not changes on the entire video, so do not use consecutive frames, or if you are forced to use consecutive frames, shuffle them randomly.

To gain speed, you may only take only one color channel from each frame, and pick the color channel randomly. That way the colors becomes noise, and cancel each other.

If the FFT is too expensive, just averaging random frames should filter the noise.

Upvotes: 2

Related Questions