CODEWITHSUNDEEP
javaartificial-intelligenceawtrobot
Michael Libman
Michael Libman

Reputation: 73

Using Java Robots class to locate things on a screen

I was wondering if it was possible to locate files or programs with this class. I know that there are ways to use mouse coordinates or simple find an application and run it but that's not what I want to do.

And example to be more clear is: Let's say I have Google Chrome, Firefox and Safari on my desktop and I shuffle them around all over. How can I get my mouse to find the browser that I'm looking for without simply using a 'run application' command.

I hope that clarifies it! Thanks for your help =)

Upvotes: 4

Views: 5827

Answers (3)

Nelson Macieira
Nelson Macieira

Reputation: 1

I'm doing something like that for web-crawling, and I can tell you that the tools on the post below will not work that simple. I'm using Selenium for other purposes but it will not work on images where you've got a canvas on the top of all the visible area. To achieve that, you should use the robot class, but previously you have to know the coordinates where you expect it to be. To use some sort of GUI automated testing tool you have to have the fields simply available to be selected. I don't think that you will find it easily on some webpage. Probably to achieve a working solution you'll have to do digital image processing, which is quite good and not that difficult. There are lots of Java libs to do that. Try OpenCV, I think it's JavaCV the lib name, it's a powerful one.

Upvotes: 0

Tom
Tom

Reputation: 16188

No I don't think this is easily achievable using the Robot class. This is because Java is a cross platform language and cannot/does not have the libraries available to integrate into the native environment and window manager.

The Java code needs to run on any platform with a JDK, adding these features to every JDK would be quite a task. I would look into other Java libraries that can provide this functionality, Robots will not provide what you are looking for.

For Windows I would recommend using AutoHotkey, which is a scripting language for desktop automation and is very powerful. There is also a port for Mac OSX as IronAHK.

As for Java libraries depending on what you wish to accomplish you could have a look at some of the GUI automated testing solutions such as TestComplete or even Selenium. Have a look for test automation tools for Java and you can find even more!

Upvotes: 0

Bernhard Barker
Bernhard Barker

Reputation: 55609

The simplest way (that I know of, using Robot) is to record the pixels that make up something you want to find and look for it pixel by pixel (you can compare the current screenshot to an image stored in a file). To do this:

Take a screenshot of the icon you are looking for and save it to a file (just save the icon) (not .jpg, since this is not lossless, probably just .bmp).

Then you can do something like:

Robot robot = new Robot();
BufferedImage current = robot.createScreenCapture(
  new Rectangle(Toolkit.getDefaultToolkit().getScreenSize()));

static void getScreenStart()
{
  BufferedImage icon = ImageIO.read(new File("chromeIcon.bmp"));
  for (int x = 0; x < current.getWidth (); x++)
  for (int y = 0; y < current.getHeight(); y++)
  {
     boolean matches = true;
     for (int x2 = 0; x2 < icon.getWidth () && matches; x2++)
     for (int y2 = 0; y2 < icon.getHeight() && matches; y2++)
        if (icon.getRGB(x2, y2) != current.getRGB(x+x2, y+y2))
           matches = false;
     if (matches)
     {
        X_START = x;
        Y_START = y;
        return;
     }
  }
}

But do generally try to avoid doing this when possible.

Upvotes: 1

Related Questions