Image Recognition on Browser using Python

Question

I want to implement a software for counting cards in blackjack, using some image recognition to automate the process. But i don't know where to start. I think the problem could be divided into these steps:

1- Take an image from the browser in-game (it's basically an Adobe Flash Game)

2- Process the image, with some image recognition, that recognize all the cards.

3- Updating the counter using the strategy Hi-Lo

4- Showing the result on screen

How can i do this using python? What are the libraries that could help me? This is a totally new field for me. I'll try to implement the question based on your suggestions.

EDIT 1:

Selenium Webdriver works fine, so far i've used this peace of code to get the screenshot of the main page, but i can't get into the game, because i don't have the money to play lol:

from selenium import webdriver

browser = webdriver.Chrome()
browser.get('https://www.888casino.it/giochi-da-casino/')
browser.save_screenshot('screenie.png')
browser.quit()

But basically i need to substitute the browser.get() with something that hooks the browser, rather then something that opens a new page. Then i need to implement a for loop that takes screenshots every second while i'm playing the game, and after that i can start processing those images.

EDIT 2:

I will try TensorFlow API for the image processing, but i didn't find any training model for recognizing cards. So i have to create a brand new model, i found this tutorial that helps me training my own object recognition model. Please, if you know an existing training model, link that.

EDIT 3:

Using Tensorflow i was able to create my own model for object recognition, now i need to use that model in a python script. For now i've used this sample script, that opens an image and draw rectangles around the cards.

import os
import cv2
import numpy as np
import tensorflow as tf
import sys

# This is needed since the notebook is stored in the object_detection folder.
sys.path.append("..")

# Import utilites
from utils import label_map_util
from utils import visualization_utils as vis_util

# Name of the directory containing the object detection module we're using
MODEL_NAME = 'inference_graph'
IMAGE_NAME = 'test1.jpg'

# Grab path to current working directory
CWD_PATH = os.getcwd()

# Path to frozen detection graph .pb file, which contains the model that is used
# for object detection.
PATH_TO_CKPT = os.path.join(CWD_PATH,MODEL_NAME,'frozen_inference_graph.pb')

# Path to label map file
PATH_TO_LABELS = os.path.join(CWD_PATH,'training','labelmap.pbtxt')

# Path to image
PATH_TO_IMAGE = os.path.join(CWD_PATH,IMAGE_NAME)

# Number of classes the object detector can identify
NUM_CLASSES = 13

# Load the label map.
# Label maps map indices to category names, so that when our convolution
# network predicts `5`, we know that this corresponds to `king`.
# Here we use internal utility functions, but anything that returns a
# dictionary mapping integers to appropriate string labels would be fine
label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)

# Load the Tensorflow model into memory.
detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')

    sess = tf.Session(graph=detection_graph)

# Define input and output tensors (i.e. data) for the object detection classifier

# Input tensor is the image
image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')

# Output tensors are the detection boxes, scores, and classes
# Each box represents a part of the image where a particular object was detected
detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')

# Each score represents level of confidence for each of the objects.
# The score is shown on the result image, together with the class label.
detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')
detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')

# Number of objects detected
num_detections = detection_graph.get_tensor_by_name('num_detections:0')

# Load image using OpenCV and
# expand image dimensions to have shape: [1, None, None, 3]
# i.e. a single-column array, where each item in the column has the pixel RGB value
image = cv2.imread(PATH_TO_IMAGE)
image_expanded = np.expand_dims(image, axis=0)

# Perform the actual detection by running the model with the image as input
(boxes, scores, classes, num) = sess.run(
    [detection_boxes, detection_scores, detection_classes, num_detections],
    feed_dict={image_tensor: image_expanded})

# Draw the results of the detection (aka 'visulaize the results')

vis_util.visualize_boxes_and_labels_on_image_array(
    image,
    np.squeeze(boxes),
    np.squeeze(classes).astype(np.int32),
    np.squeeze(scores),
    category_index,
    use_normalized_coordinates=True,
    line_thickness=8,
    min_score_thresh=0.80)

# All the results have been drawn on image. Now display the image.
cv2.imshow('Object detector', image)

# Press any key to close the image
cv2.waitKey(0)

# Clean up
cv2.destroyAllWindows()

Now i need to create my own script that recognise the cards, and for every card updates a counter that must be shown on screen. This is the most tricky part because i don't know where to start. I have several problems with this step, first the script must be able to distinguish the cards that left the deck from the new one, this way it doesn't mess up the counter every time a screenshot is taken. Second, the counter should be updated with -1 for high cards(ten - ace), +1 for low cards(two-six), and 0 for neutral cards(7-8-9), and must be visible on screen.

EDIT 4: I've build the first version of the software, but there are some issues, the counter doesn't update correctly. Here is the code:

import pyscreenshot as ImageGrab
from win32api import GetSystemMetrics
import os
import cv2
import numpy as np
import tensorflow as tf
import sys

import warnings
import h5py

def UpdateCounter(labels, c):
    for i in labels:
        if labels['ace'] > 0:
            c = c - 1
        if labels['king'] > 0:
            c = c - 1
        if labels['queen'] > 0:
            c = c - 1
        if labels['jack'] > 0:
            c = c - 1
        if labels['ten'] > 0:
            c = c - 1
        if labels['six'] > 0:
            c = c + 1
        if labels['five'] > 0:
            c = c + 1
        if labels['four'] > 0:
            c = c + 1
        if labels['three'] > 0:
            c = c + 1
        if labels['two'] > 0:
            c = c + 1
        return c

if __name__ == '__main__':


    sys.path.append("..")
    from utils import label_map_util
    from utils import visualization_utils as vis_util

    MODEL_NAME = 'inference_graph'
    IMAGE_NAME = 'test1.jpg'
    CWD_PATH = os.getcwd()
    PATH_TO_CKPT = os.path.join(CWD_PATH,MODEL_NAME,'frozen_inference_graph.pb')

    PATH_TO_LABELS = os.path.join(CWD_PATH,'training','labelmap.pbtxt')
    PATH_TO_IMAGE = os.path.join(CWD_PATH,IMAGE_NAME)
    os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
    NUM_CLASSES = 13

    label_map = label_map_util.load_labelmap(PATH_TO_LABELS)

    categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)

    category_index = label_map_util.create_category_index(categories)

    detection_graph = tf.Graph()

    with detection_graph.as_default():

        od_graph_def = tf.GraphDef()

        with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
            serialized_graph = fid.read()
            od_graph_def.ParseFromString(serialized_graph)
            tf.import_graph_def(od_graph_def, name='')
            sess = tf.Session(graph=detection_graph)

    image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')

    detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')

    detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')
    detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')

    num_detections = detection_graph.get_tensor_by_name('num_detections:0')

    c = 0


    while True:
        labels = {"ace" : 0, "king": 0, "queen": 0, "jack": 0, "ten": 0, "nine": 0, "eight": 0,"seven": 0, "six": 0, "five": 0, "four":0, "three": 0, "two": 0}

        with warnings.catch_warnings():
            warnings.filterwarnings("ignore",category=FutureWarning)
            screenshot=ImageGrab.grab(bbox=(42,42, GetSystemMetrics(0),GetSystemMetrics(1)))
            screenshot.save(IMAGE_NAME)


        image = cv2.imread(PATH_TO_IMAGE)
        image_expanded = np.expand_dims(image, axis=0)

        (boxes, scores, classes, num) = sess.run(
            [detection_boxes, detection_scores, detection_classes, num_detections],
            feed_dict={image_tensor: image_expanded})

        data = [category_index.get(value) for index,value in enumerate(classes[0]) if scores[0,index] > 0.9]



        for ch in data:
                        if ch['name'] == "ace":
                            labels["ace"] += 1
                        elif ch['name'] == "king":
                            labels["king"] += 1
                        elif ch['name'] == "queen":
                            labels["queen"] += 1
                        elif ch['name'] == "jack":
                            labels["jack"] += 1
                        elif ch['name'] == "ten":
                            labels["ten"] += 1
                        elif ch['name'] == "nine":
                            labels["nine"] += 1
                        elif ch['name'] == "eight":
                            labels["eight"] += 1
                        elif ch['name'] == "seven":
                            labels["seven"] += 1
                        elif ch['name'] == "six":
                            labels["six"] += 1
                        elif ch['name'] == "five":
                            labels["five"] += 1
                        elif ch['name'] == "four":
                            labels["four"] += 1
                        elif ch['name'] == "three":
                            labels["three"] += 1
                        elif ch['name'] == "two":
                            labels["two"] += 1

        print(UpdateCounter(labels, c))

Pls how can i fix this? I need to show the counter only when new cards get recognised, and i need to fix also the bad matchup that the program gets.

Answer 1

I believe you can achieve that by using selenium as you mentioned.

It would be somethig along the lines of:

from selenium import webdriver
import time

browser = webdriver.Chrome()
browser.get('https://www.888casino.it/giochi-da-casino/')

while True:
    browser.save_screenshot('screenie.png')
    #do the image processing...
    time.sleep(1)

browser.quit()

For the image processing itself, you will have the issue of identifying the desired elements on you image, the cards, in your case, to further process each of them individually. So you have a two step task in this regard.

There's a tensorflow object detection API that might come in hand for you: https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API

Good luck!