OpenCV Python Bindings for GrabCut Algorithm

Question

I've been trying to use the OpenCV implementation of the grab cut method via the Python bindings. I have tried using the version in both cv and cv2 but I am having trouble finding out the correct parameters to use to get the method to run correctly. I have tried several permutations of the parameters and nothing seems to work (basically every example I've seen on Github). Here are a couple examples I have tried to follow:

Example 1

Example 2

And here is the method's documentation and a known bug report:

Documentation

Known Grabcut Bug

I can get the code to execute using the example below, but it returns a blank (all black) image mask.

img = Image("pills.png")
mask = img.getEmpty(1)
bgModel = cv.CreateMat(1, 13*5, cv.CV_64FC1)
fgModel = cv.CreateMat(1, 13*5, cv.CV_64FC1)
for i in range(0, 13*5):
    cv.SetReal2D(fgModel, 0, i, 0)
    cv.SetReal2D(bgModel, 0, i, 0)

rect = (150,70,170,220)
tmp1 = np.zeros((1, 13 * 5))
tmp2 = np.zeros((1, 13 * 5))
cv.GrabCut(img.getBitmap(),mask,rect,tmp1,tmp2,5,cv.GC_INIT_WITH_RECT)

I am using SimpleCV to load the images. The mask type and return type from img.getBitmap() are:

iplimage(nChannels=1 width=730 height=530 widthStep=732 )
iplimage(nChannels=3 width=730 height=530 widthStep=2192 )

If someone has a working example of this code I would love to see it. For what it is worth I am running on OSX Snow Leopard, and my version of OpenCV was installed from the SVN repository (as of a few weeks ago). For reference my input image is this:

I've tried changing the result mask enum values to something more visible. It is not the return values that are the problem. This returns a completely black image. I will try a couple more values.

img = Image("pills.png")
mask = img.getEmpty(1)
bgModel = cv.CreateMat(1, 13*5, cv.CV_64FC1)
fgModel = cv.CreateMat(1, 13*5, cv.CV_64FC1)
for i in range(0, 13*5):
    cv.SetReal2D(fgModel, 0, i, 0)
    cv.SetReal2D(bgModel, 0, i, 0)

rect = (150,70,170,220)
tmp1 = np.zeros((1, 13 * 5))
tmp2 = np.zeros((1, 13 * 5))
cv.GrabCut(img.getBitmap(), mask, rect, tmp1, tmp2, 5, cv.GC_INIT_WITH_MASK)
mask[mask == cv.GC_BGD] = 0
mask[mask == cv.GC_PR_BGD] = 0
mask[mask == cv.GC_FGD] = 255
mask[mask == cv.GC_PR_FGD] = 255
result = Image(mask)
result.show()
result.save("result.png")

Answer 1

Kat, this version of your code seems to work for me.

import numpy as np
import matplotlib.pyplot as plt
import cv2


filename = "pills.png"
im = cv2.imread(filename)

h,w = im.shape[:2]

mask = np.zeros((h,w),dtype='uint8')
rect = (150,70,170,220)
tmp1 = np.zeros((1, 13 * 5))
tmp2 = np.zeros((1, 13 * 5))

cv2.grabCut(im,mask,rect,tmp1,tmp2,10,mode=cv2.GC_INIT_WITH_RECT)

plt.figure()
plt.imshow(mask)
plt.colorbar()
plt.show()

Produces a figure like this, with labels 0,2 and 3.

Answer 2

Your mask is filled with the following values:

• GC_BGD defines an obvious background pixels.
• GC_FGD defines an obvious foreground (object) pixel.
• GC_PR_BGD defines a possible background pixel.
• GC_PR_FGD defines a possible foreground pixel.

Which are all part of an enum:

enum { GC_BGD    = 0,  // background
       GC_FGD    = 1,  // foreground
       GC_PR_BGD = 2,  // most probably background
       GC_PR_FGD = 3   // most probably foreground
     };

Which translates to the colors: completely black, very black, dark black, and black. I think you'll find that if you add the following code (taken from your example 1 and slightly modified) your mask will look nicer:

mask[mask == cv.GC_BGD] = 0 //certain background is black
mask[mask == cv.GC_PR_BGD] = 63 //possible background is dark grey
mask[mask == cv.GC_FGD] = 255  //foreground is white
mask[mask == cv.GC_PR_FGD] = 192 //possible foreground is light grey