CODEWITHSUNDEEP
pythonopencvcomputer-visionartificial-intelligencepyautogui
cyb3r_anarch
cyb3r_anarch

Reputation: 35

locate an opencv detection using pyautogui

so I'm making a bot that detects icons on the screen and moves the mouse to the detected icon the user chose. this is what the code looks like:

import numpy as np
import pyautogui
import cv2
from PIL import ImageGrab

fourcc = cv2.VideoWriter_fourcc(*'XVID')

face_csc = cv2.CascadeClassifier('improved_cascade.xml')

out = cv2.VideoWriter("output.avi", fourcc, 5.0, (1366, 768))

while True:

    img = ImageGrab.grab(bbox=None)
    # convert image to numpy array
    img_np = np.array(img)
    # convert color space from BGR to RGB
    frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2RGB)
    # show image on OpenCV frame
    faces = face_csc.detectMultiScale(frame, 1.1 , 15)

    for (x,y,w,h) in faces:
        detected_icon = cv2.rectangle(frame,(x,y),(x+w,y+h), (255,0,0), 2)
        roi_gray = frame[y:y+h, x:x+w]
        roi_color = img_np[y:y+h,x:x+w]
        cv2.putText(frame,'icon',(x,y),cv2.FONT_HERSHEY_TRIPLEX,0.8,(0,0,255),1)
        cv2.imshow("stream", frame)
    # write frame to video writer
    out.write(frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

out.release() 
cv2.destroyAllWindows()

but I'm having trouble making my mouse click on an icon opencv detected. for example: lets say that I set my program so that when it detects chrome on the screen, it hovers the mouse automatically to the icon and click on it. how would I be able to do that? thanks

Upvotes: 2

Views: 2257

Answers (1)

Samuel Henderson
Samuel Henderson

Reputation: 141

I don't have a handy Windows box to run ImageGrab on, but assuming it produces a screenshot with the same width and height (in pixels) as the actual screen, given that both Pyautogui and OpenCV put the origin in the top left, the translation to Pyautogui should be straightforward:

for (x,y,w,h) in faces:
    center_x = x + 0.5 * w
    center_y = y + 0.5 * h
    pyautogui.click(center_x, center_y)

That teleports the mouse pointer to the center of the object rectangle and clicks it. In case you want to simulate more human-like mouse movement and clicking, Pyautogui has a number of tools for that purpose.

Upvotes: 1

Related Questions