Image Remap without cropping

Question

This is a continuation of this question : Finding Squares in Image

I followed the steps in my answer there : https://dsp.stackexchange.com/a/7526/818, And I got the answer as given below :

But at the end of that answer, I have explained a problem, and that is my question.

Explanation :

I already have the centroids of detected squares from step 1 in the previous link (and those detected squares are marked in mask_image below):

I created a grid image as below ( I know their centroid values also) :

I also found which point in grid image to be mapped to corresponding point in mask_image.

With that information, I applied scipy.interpolate.griddata(), and then OpenCV's cv2.remap() function.

And its result is given below :

As you can see, all the squares except the two at center are clipped. It is like, output contain only the region inside a boundary drawn connecting all the centroids of the mask_image.

Scene becomes more worse below :

The case is more worse when the last square (yellow) or any other square in four corners is not detected in first step. Consider last one is not detected. Then below is the result I get and you can see a slant cut at the bottom (marked with yellow color):

Question :

Why remap function not working beyond the points I have given ? And what should I do to remap it without clipping ?

I thought it would work for the full image even if I give some points, which are not at edges.

Expected Output :

Below is the output I expected at the end of my operation. (Region inside red boundary is what I got actually got now )

Looking for some good suggestions ...

UPDATE :

I also add the code here. Only remapping part code is added. Full code is too big to be added here :

# ideal - the grid image - https://i.sstatic.net/3QudG.png
# centroids - list of centroids of the squares in mask_image - https://i.sstatic.net/jh6bQ.png
# match_pts - list of centroids of the squares in grid image corresponding to squares in mask_image
# warped - the final image obtained after remap - https://i.sstatic.net/O26ZA.png

grid_x,grid_y = np.meshgrid(np.arange(ideal.shape[1]),np.arange(ideal.shape[0]))
dst = np.array(centroids) 
src = np.array(match_pts) 
grid_z = griddata(dst,src,(grid_x,grid_y),method='cubic')
map_x_32 = grid_z[:,:,0].astype('float32')
map_y_32 = grid_z[:,:,1].astype('float32')
warped = cv2.remap(ideal, map_x_32, map_y_32, cv2.INTER_CUBIC)

Also added the datas like centroids, match_pts etc, so that if someone want to try it out, they can directly use the data instead of finding it from image : gist.github.com/4540887

Answer 1

Step 1: Whatever final binary image you are getting from analyzing in B,G,R,H,S,V plane, in that image do a blob counting algorithm.

Step 2: Find the largest blob on basis of area or contour length. Since your blobs will be mostly parallelogram types so area or contour, any one will do.

Step 3: With the largest blob (since largest blob is the best blob resembling your real world squares) try to find the orientation of the blob...this you can get by a fitting a best fit rectangle OR you can get the corner points...get the slope of the lines joining them (in both horizon and vertical direction).

Step 4: Once you get the two slopes draw two lines running through the axis of the blob. for axis you can average the corner points or you can use the centroid (center of mass)...I would go with average of corner points...

Step 5: Since in each horizontal and vertical direction, spacing is equal (ideally horizontal and vertical spacing are also equal as it comes from your ideal square picture but we will not assume it..) just need to locate the possible centroids of the other parallelograms

BOTTOM LINE: If any one square gets detected perfectly you can make the whole grid. Just keep marking centers at an interval of 2H (H = horizontal width of biggest blob) along the horizontal axis of the biggest blob and at an interval of 2V (V = vertical height of biggest blob) vertically along the vertical axis of the blob.

Some pics to support