What OpenCV tracking API should I use to track running cells?

Question

I am working on a project to track multiple cells running through a fluid channel. I need to do track the cells from when they come in the channel until they exit the channel.

I was working with LabVIEW for this tracking task in which I use the X coordinator between frames. This methods is only good when the flow is slow. It becomes less accurate when there are large number of cells coming in.

Could you please give me some advice on what OpenCV tracking methods for my task? I am using Python.

Answer 1

First, before flowing any cells, I would acquire an averaged background image by accumulating frames. This will allow you tu use background subtraction later to more easily find the cells you care about. I use a function that looks like this to do it, where images is the series of frames you want to average.

# returns a frame that is the average of all the provided frames
def average(images):
    average = np.zeros(images[0].shape, np.float32)
    for image in images:
        cv2.accumulate(image, average)
    average = average / len(images)
    average = np.uint8(average)
    return average

Once you have a good background (using an appropriate number of frames, if you see noise after subtraction, up the number of frames used for averaging), you can use a series of background subtraction, erosion and dilation (if necessary) and contour finding to locate cells.

This function has all of these parts, you can use it as an example. It will subtract out the background, threshold it, then erode and dilate to get rid of any remaining noise, and look for the contours that are left which would ideally be the cells you care about. You will likely have to change the erosion and dilation sizes (they are in pixels) to get a good picture.

# detect a moving ball with countours and background subtraction
def detectBall(frame, background, roi):
    # background subtraction, thresholding, erosion and dialation
    motion = cv2.absdiff(background, frame[roi[0][1]:roi[1][1], roi[0][0]:roi[1][0]])
    _, thresh1 = cv2.threshold(motion, 10, 255, cv2.THRESH_BINARY)
    gray = cv2.cvtColor(thresh1, cv2.COLOR_BGR2GRAY)
    blurred = cv2.GaussianBlur(gray, (5, 5), 0)
    thresh = cv2.threshold(blurred, 30, 255, cv2.THRESH_BINARY)[1]
    erosion_size = 10
    dilate_size = 8
    thresh = cv2.erode(thresh, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (erosion_size, erosion_size)))
    thresh = cv2.dilate(thresh, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (dilate_size, dilate_size)))

    # find countours
    cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    cnts = imutils.grab_contours(cnts)
    center = None
    for c in cnts:
        # compute the center of the contour
        M = cv2.moments(c)
        cX = int((M["m10"] / (M["m00"] + 1e-7)))
        cY = int((M["m01"] / (M["m00"] + 1e-7)))

        # do some color filtering on each contour to eliminate extra contours
        hsv = cv2.cvtColor(motion, cv2.COLOR_BGR2HSV)
        (_, _, h) = hsv[cY, cX]
        if 80 < h < 140:
            # calculate the center and draw the detected ball
            c = c.astype("float")
            c = c.astype("int")
            area = int(cv2.contourArea(c))
            diam = int(math.sqrt(area / math.pi))
            center = (cX + roi[0][0], cY + roi[0][1])
            cv2.circle(frame, center, 1, RED, 5)
            cv2.circle(frame, center, diam, GREEN, 2)
            cv2.putText(frame, str(center), (center[0] + 10, center[1] + 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, BLUE, 2) # draw center text
    return center, frame

This will give you the locations of any object that differs from the background and on my system runs quite quickly.

This may be enough for you, but if after identifying you want to try to track specific cells, you can use template matching and slicing around centers of known cell locations to see where they move to. If cells all look pretty similar though, this may or may not work well but you can try it out.

For this, I use functions like this:

def findMatchingPoints(old_frame, new_frame, pts, templateSize=5, searchWindowSize=20):
    output_points = []
    strengths = []
    for pt in pts:
        # make template and window images
        l_template = int(templateSize / 2)      # calculate half template size
        l_window = int(searchWindowSize /2)     # calculate half window size
        x = int(pt[0][0])                       # get x coordinate
        y = int(pt[0][1])                       # get y coordinate
        template = old_frame[y-l_template:y+l_template, x-l_template:x+l_template]      # templeate comes from old
        window = new_frame[y-l_window:y+l_window, x-l_window:x+l_window]                # look in the new
        # skip poorly formed windows or templates
        if 0 in window.shape or 0 in template.shape:
            strengths.append(float(10))
            output_points.append([[np.float32(0), np.float32(0)]])
            continue
        # Apply template matching and save results
        res = cv2.matchTemplate(window, template, cv2.TM_SQDIFF_NORMED)
        strength, _, top_left, _ = cv2.minMaxLoc(res)                    # returns (min_val, max_val, min_loc, max_loc), for SSD we want min location, is top left of template
        center = [[np.float32(top_left[0]+x-l_window), np.float32(top_left[1]+y-l_window)]]
        strengths.append(strength)
        output_points.append(center)

    # create a boolean mask to keep good points
    output_points = np.asarray(output_points)
    _, mask = cv2.findFundamentalMat(pts, output_points, cv2.FM_RANSAC)

    return output_points, strengths, mask

# see which matches are good
def filter_matches(p0, p1, st, good):
    good_old, good_new = [], []
    for old, new, s, g in zip(p0, p1, st, good):
        s = int(s*100)
        if s < 1 and g == 1:
            good_new.append(new)
            good_old.append(old)
    return good_old, good_new

Hopefully this helps. The key functions/ideas here are background averaging and subtraction, contour finding, and template matching.