Opencv/Python - how can I get the coordinates of detected areas after image processing

Question

My goal is to write a bot playing minesweeper, but I get stuck at the point, when I want to tell the bot, where the squares are. I tried a lot of different functions. First of all, my tool grabs a screenshot of a predefined area. This picture looks like this:screenshot of game board

after that I want to fill a numpy array in this way:

info_map = np.ones((board_height=9, board_width=9),
                   dtype = np.uint8)*11
>array([[11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11],
        [11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11],
        ...
        [11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11]], dtype=uint8

11 stands in this example for "undiscovered" or a blue, untouched square.

with the help of the opencv square detector from the examples I get a huge list with detected points (nearly 5000points for a 9x9 grid). I don't know how to remove all the noise, to get the right coords of the corners. so I came to the next example showing below:

My last try was the following:

import glob
import cv2
import numpy as np
import sys   

def canny_edge():
"""This function is taken from SentDex from one of his fantastic Python
tutorials.
https://pythonprogramming.net/canny-edge-detection-gradients-python-opencv-tutorial/
"""
for fn in glob('full_snap.png'):
    img = cv2.imread(fn)
    while(1):
        hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
        lower_red = np.array([30,150,50])
        upper_red = np.array([255,255,180])
        mask = cv2.inRange(hsv, lower_red, upper_red)
        #res = cv2.bitwise_and(img, img, mask= mask)
        laplacian = cv2.Laplacian(img, cv2.CV_64F)
        #edges = cv2.Canny(img,100,200)

        cv2.imwrite(os.getcwd() + '\\laplace.png', laplacian)

This is the saved image: laplace.png

here I tried to do a for loop over the Laplacian to check if the pixel is blue or not, and take the first one as an anchor to tell the bot, that from the anchor (starting point) there is a square every x pixel in the x direction and in the y. But is there a better way to do it automatically?

but my main question is how to connect the image processed squares into an numpy array, that I can tell the bot click_square(1,2) and he knows at which pixels is the square from row 1 and column 2.

Hint: My coordinate system starts in the upper left corner of the screen.

Thank you for your help.

Answer 1

Here is the code i used. I have to rearange my Dataframe in the end, but only for better human reading. I think there are a lot of possible improvement in my code, but i´m happy that the behaviour of my code is good enough for my needs.

import cv2
import numpy as np
import pandas as pd

img = cv2.imread('full_snap.png')
"""
some of my blue pixels
R: between 98 and 128
G: between 176 and 211
B: between 255

h: between 210 and 200
s: between 100 and 48
v: between 68 and 100

hsv/2 in opencv and opencv uses BGR not RGB
"""

blue_MIN = np.array([255, 176, 98])
blue_MAX = np.array([255, 211, 128])
"""
https://pythonprogramming.net/color-filter-python-opencv-tutorial/
"""

# find the blue pixels and save it in frame_threshed
frame_threshed = cv2.inRange(img, blue_MIN, blue_MAX)

# find contours in the thresholded image
cnts = cv2.findContours(frame_threshed.copy(), cv2.RETR_EXTERNAL,
    cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if imutils.is_cv2() else cnts[1]

#print("cnts", cnts)
i = len(cnts)
print("i", i)

# we know we're gonna have x rows of data, where x is the product of
# board_width * board_height
numberOfRows = 81

# check if the length of cnts are euqal to the number of rows/ number of tiles
# then go further

# TODO

# create x,y data
# df = pd.DataFrame(index=np.arange(numberOfRows, 0), columns=('x', 'y'))
d = []
print("d", d)
print(type(d))


for c in cnts:
    # compute the center of the contour
    M = cv2.moments(c)
    cX = int(M["m10"] / M["m00"])
    cY = int(M["m01"] / M["m00"])

    # fill the data with the coords
    d.append({'tilenumber': i, 'X-Value': cX, 'Y-Value': cY})

    # decrease i to go backwards, because tile number 81 is the first contour
    i-=1
    # draw the center of the shape on the image
    cv2.circle(img, (cX, cY), 1, (255, 255, 255), -1)

df = pd.DataFrame(d)

# only for debugging
print("x,y dataframe", df)
cv2.imshow("Image_with_contours",img)

# Destroys all of the HighGUI windows.
cv2.destroyAllWindows()

Thank you all for the help! Robinarthur

Answer 2

I don't know what minesweeper is but from what i understood, you want to know which tiles are blue and then make necessary changes to your numpy array (correct me if its something else, i'll edit or remove the answer).

Here's the output after making four tiles blue:

What i did ?

First i thresholded the blue colour, found contours in the range approximately as that of your tiles.

Found their centers to see what pattern they follow -

They were all separated by some 55-57 pixels (both x,y coordinates). Rest is simple.

for i in range(len(coords)):
np_arr[int(coords[i][1]/57)][int(coords[i][0]/57)]=0

The blue tile coordinates are stored in coords, np_arr is my array.