CODEWITHSUNDEEP
image-processingcomputer-visiontrackingvideo-processingobject-recognition
zergylord
zergylord

Reputation: 4436

Algorithm for real-time tracking of several simple objects

I'm trying to write a program to track relative position of certain objects while I'm playing the popular game, League of Legends. Specifically, I want to track the x,y screen coordinates of any "minions" currently on the screen (The "minions" are the little guys in the center of the picture with little red and green bars over their heads).

I'm currently using the Java Robot class to send screen captures to my program while I'm playing, and am trying to figure out the best algorithm for locate the minions and track them so long as they stay on the screen.

My current thinking is to use a convolutional neural network to identify and locate the minions by the the colored bars over there heads. However, I'd have to re-identify and locate the minions on every new frame, and this seems like it'd be computationally expensive if I want to do this in real time (~10-60 fps).

These sorts of computer vision algorithms aren't really my specialization, but it seems reasonable that algorithms exist that exploit the fact objects in videos move in a continuous manner (i.e. they don't jump around from frame to frame).

So, is there an easily implementable algorithm for accomplishing this task?

enter image description here

Upvotes: 3

Views: 2406

Answers (2)

Andrey Rubshtein
Andrey Rubshtein

Reputation: 20915

Since this is a computer game, I think that the color of the bars should be constant. That might not be true only if the dynamic illumination affects the health bar, which is highly unlikely.

Thus, just find all of the pixels with this specific colors. Then you do some morphological operations and segment the image into blobs. By selecting only the blobs that fit some criteria, you can find the location of the units.

I know that my answer does not involve video, but the operations should be so simple, that it should be very quick.

As for the tracking, just find per each point the closest in the next frame.

Since the HUD location is constant, there should be no problem removing it. Image without HUD

Here is mine quick and not-so-robust implementation in Matlab that has a few limitations:

  1. Units must be quite healthy (At least 40 pixels wide)
  2. The bars do not overlap.
 function FindUnits()
    x = double(imread('c:\1.jpg'));
    green = cat(3,149,194,151); 

    diff = abs(x - repmat(green,[size(x,1) size(x,2)]));
    diff =  mean(diff,3);
    diff = logical(diff < 30);
    diff = imopen(diff,strel('square',1));

    rp = regionprops(diff,'Centroid','MajorAxisLength','MinorAxisLength','Orientation');
    long = [rp.MajorAxisLength]./[rp.MinorAxisLength];
    rp( long < 20) = [];

    xy = [rp.Centroid];
    x = xy(1:2:end);
    y = xy(2:2:end);
    figure;imshow('c:\1.jpg');hold on ;scatter(x,y,'g');
end

And the results:

enter image description here

Upvotes: 4

jkt
jkt

Reputation: 2578

You should use a model which includes a dynamic structure in it. For your object tracking purpose Hidden Markov Models (HMMs) (or in general Dynamic Bayesian Networks) are very well suitable. You can find a lot of resources on HMMs online. The issues you are going to face however, depends on your system model. If your system dynamics can easily be represented as a linear Gauss-Markov model then a simple Kalman Filter will do fine. However, in the case of nonlinear non-gaussian dynamics you should use Particle Filtering which is a Sequential Monte Carlo Method. Both Kalman Filter and Particle Filter are sequential methods so you will use the results you have at the current step to have a result at the next time step. I suggest you to check some online tutorials and papers on Multiple Object Tracking via Particle Filters. As far as I am concerned the main difficulty you will have is however, the number of objects you may want to track since you won't know the number of the objects you want to track and also a object you are tracking can just disappear as well (you may kill those little guys or they may just leave the screen) or some other guy can just enter the screen. Hope this helps.

Upvotes: 2

Related Questions