Show current cursor position in Selenium

Question

Is there a way to show current cursor position or something like this? I have action chain that should click on the exact point on some object but I guess I've picked up wrong coordinates. I use Firefox webdriver. Here's how the script looks like:

from selenium import webdriver
import time
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Firefox()
driver.get("http://www.mysite/")
elem = driver.find_element_by_xpath('//div[@id="player"][1]/object[@type="application/x-shockwave-flash"]')
action_chains = ActionChains(driver)
action_chains.move_to_element_with_offset(elem, 660, 420).perform()
action_chains.click().perform()
time.sleep(10)
driver.close()

Answer 1

This answer gives the xy coordinates of the cursor:

def main():
    try:
        driver.get(LOGIN_URL)

        # Inject the JavaScript code
        js_code = """
        document.addEventListener('mousemove', updateCursorPosition);
        function updateCursorPosition(event) {
            window.cursorX = event.clientX;
            window.cursorY = event.clientY;
        }
        """
        driver.execute_script(js_code)
        time.sleep(2)  # Give the JavaScript code time to set up

        while True:
            # Retrieve the cursor position values
            cursor_x = driver.execute_script("return window.cursorX || 0;")
            cursor_y = driver.execute_script("return window.cursorY || 0;")
            print(f"Cursor Position: X = {cursor_x}, Y = {cursor_y}")

            time.sleep(0.5)  # Adjust the interval as needed

    except Exception as e:
        logging.error(f"An error occurred: {e}")
        # Optionally log more details only in development/debugging:
        # if __debug__:
        #     logging.error(traceback.format_exc())

    finally:
        driver.quit()```

this below js code can be used to mark the spot on the ui in browser (just replace it with the above js code )  


 document.addEventListener('mousemove', createCircle);

        function createCircle(event) {
            const circle = document.createElement('div');
            circle.style.position = 'absolute';
            circle.style.width = '20px';
            circle.style.height = '20px';
            circle.style.borderRadius = '50%';
            circle.style.backgroundColor = 'red';
            circle.style.left = (event.clientX - 10) + 'px'; // Center the circle
            circle.style.top = (event.clientY - 10) + 'px';  // Center the circle
            circle.style.zIndex = 9999; // Ensure it's on top
            document.body.appendChild(circle);
            setTimeout(() => {
                circle.remove();
            }, 1000); // Remove after 1 second
        }

Answer 2

If all you want to confirm is whether Selenium is clicking on the right element, instead of click()-ing on the element:

actions.move_to_element(my_element).click().perform()

context_click() on it:

actions.move_to_element(my_element).context_click().perform()

Selenium will right click on your element, which will result in the context menu being displayed at the bottom right corner of your cursor. Once you see the position of the context menu, you will know whether Selenium is clicking on the right element.

Answer 3

Yeah, this problem is frustrating.

This method will allow you to see where the mouse is by painting a red dot on it: Start by adding the following CSS to your webpage, temporarily:

// TODO: Remove me after you've figured out where the mouse is moving to.
.dot {
  background: red;
  position: absolute;
  width: 2px;
  height: 2px;
  z-index: 10000;
}

Then, in your code, just before you're calling the *.moveTo() method, add a pause to allow you to inject code into the browser's console. This code is in JavaScript (JS / webdriver.io) but it's easy to add in any language:

// Other test stuff...

console.log("This gives you 10 seconds to inject the code..."); // <-- Add this and the next line
browser.pause(10000);

browser.moveTo(...gawdKnowsWhere);

Now, when you see the test command line console is waiting 10 seconds, open up Chrome/Firefox's developer tools (F12) and paste this snippet into the browser's "Console" tab:

  (function() {
    "use strict";

    document.onmousemove = handleMouseMove;
    function handleMouseMove(event) {
      var dot, eventDoc, doc, body, pageX, pageY;
      
      event = event || window.event; // IE-ism
      
      // If pageX/Y aren't available and clientX/Y
      // are, calculate pageX/Y - logic taken from jQuery
            // Calculate pageX/Y if missing and clientX/Y available
      if (event.pageX == null && event.clientX != null) {
        eventDoc = (event.target && event.target.ownerDocument) || document;
        doc = eventDoc.documentElement;
        body = eventDoc.body;

        event.pageX = event.clientX +
          (doc && doc.scrollLeft || body && body.scrollLeft || 0) -
          (doc && doc.clientLeft || body && body.clientLeft || 0);
        event.pageY = event.clientY +
          (doc && doc.scrollTop  || body && body.scrollTop  || 0) -
          (doc && doc.clientTop  || body && body.clientTop  || 0 );
      }

      // Add a dot to follow the cursor
      dot = document.createElement('div');
      dot.className = "dot";
      dot.style.left = event.pageX + "px";
      dot.style.top = event.pageY + "px";
      document.body.appendChild(dot);
    }
  })();

Now, go hunting for a red dot. Wherever the mouse is moved to, it will not be able to click there because there's a red square in the way now. But at least you can see where it's trying to go.

Can't see the dot?

Some tips:

• Make the dots bigger in the CSS by modifying the width & height.
• Move your mouse across the screen and watch all the red dots. Are they showing up when you move your mouse?
• Make sure you can paint a red dot where you expect it to be. The z-order might not be high enough, etc.

Answer 4

I had an opaque HTML5 canvas where I had to check values of tooltips at various offsets, and what trouble! After lots of banging my head, I learned several important things:

1. move_to_element_by_offset has issues prior to 2.42. See also https://code.google.com/p/selenium/issues/detail?id=4215

2. Even after an update, move_to_element_by_offset still seems persnickety. I switched to a combination of the ActionChains move_to_element hover (defaults to the center of the element) and a second ActionChains with move_by_offset

3. Now that you're in position, re-request the position-sensitive element. In my case, that's a tooltip object. This selenium.webdriver.remote.element will have the .location['x'] and .location['y'] members you need to sanity check your position.

Things that don't work for grabbing cursor position after a perform'd ActionChain to hover, to save everyone some time:

• .location['x'] and .location['y'] on the element you pass into move_to_element. These values are not updated by ActionChains.
• selenium.webdriver.remote.webdriver switch_to_active_element; I'm not clear what elements qualify as "active", but it doesn't acknowledge our tooltips as active elements and gave me back [0, 0] coordinates.
• selenium.webdriver.remote.webdriver get_window_position; the documentation was a bit vague, but this does just what the function suggests: a window rather than a cursor position.

Answer 5

It was a little hard for me as a selenium's newbie to understand that it just can't generate clicks on flash objects. The only possible solution I see here is to use sikuli-script.

I've found the project on GitHub: https://github.com/kevlened/sikuli_cpython. It allows to use sikuli scripts (written in jython) in pure python so I guess I will be able to write some sikuli scripts and integrate them with selenium tests.

Answer 6

I don't see any possible methods for changing the resolution directly as we can not use any locator for this.

For your example of JWPlayer, I am sure that there might be some javascript snippet which will allow changing the resolution. Using the Selenium JavascriptExecutor you can run that javacript code to simulate the resolution change.. although not by clicking the cursor.

WebDriver driver = new ChromeDriver();

if (driver instanceof JavascriptExecutor) {
    ((JavascriptExecutor) driver).executeScript(YOUR_JAVA_SCRIPT_STRING);
}

I noticed you mentioned you don't want to use any native JWPlayer API. I see this as the only possible solution.