How to get coordinates of the overall bounding box of a text image?

Question

original image

img = cv2.imread('eng2.png')

d = pytesseract.image_to_data(img, output_type=Output.DICT)
n_boxes = len(d['level'])
for i in range(n_boxes):
    (x, y, w, h) = (d['left'][i], d['top'][i], d['width'][i], d['height'][i])
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 2)

plt.figure(figsize=(10,10))
plt.imshow(img)

The above code produces this image. Now in the image there are two coordinates one for each word and other for the whole text. I would like to get the coordinates for the whole text (sentences in each line or the whole paragraph

This is what I have tried

box = pd.DataFrame(d) #dict to dataframe
box['text'].replace('', np.nan, inplace=True) #replace empty values by NaN
box= box.dropna(subset = ['text']) #delete rows with NaN 

print(box)


def lineup(boxes):
    linebox = None
    for _, box in boxes.iterrows():
        if linebox is None: linebox = box           # first line begins
        elif box.top <= linebox.top+linebox.height: # box in same line
            linebox.top = min(linebox.top, box.top)
            linebox.width = box.left+box.width-linebox.left
            linebox.heigth = max(linebox.top+linebox.height, box.top+box.height)-linebox.top
            linebox.text += ' '+box.text
        else:                                       # box in new line
            yield linebox
            linebox = box                           # new line begins
    yield linebox                                   # return last line

lineboxes = pd.DataFrame.from_records(lineup(box))

Output dataframe

n_boxes = len(lineboxes['level'])
for i in range(n_boxes):
    (x, y, w, h) = (lineboxes['left'][i], lineboxes['top'][i], lineboxes['width'][i], lineboxes['height'][i])
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 2)

plt.figure(figsize=(10,10))
plt.imshow(img)

There seems to be no difference between the original coordinates and after joining all the coordinates

How can i get the coordinates of the whole text (sentences in each line or the whole paragraph) using pytesseract library?

Answer 1

You faced a similar issue in one of your previous questions linked here. I failed to elaborate what I meant in the comments. Here is a more visual explanation.

By horizontal kernel I meant an array with single row [1, 1, 1, 1, 1]. The number of columns can be determined based on the font size and space between characters/words. Using the kernel with a morphological dilation operation you can connect individual entities that are present horizontally as a single entity.

In your case, we would like to extract each line as an individual entity. Let's go through the code:

Code:

img = cv2.imread('letter.png')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# inverse binary image, to ensure text region is in white
# because contours are found for objects in white
th = cv2.threshold(gray,0,255,cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

Now there is a black border surrounding the original image. In th it becomes are white border. Since it is unwanted we will remove it using cv2.floodFill()

black = np.zeros([img.shape[0] + 2, img.shape[1] + 2], np.uint8)
mask = cv2.floodFill(th.copy(), black, (0,0), 0, 0, 0, flags=8)[1]

# dilation using horizontal kernel
kernel_length = 30
horizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (kernel_length, 1))
dilate = cv2.dilate(mask, horizontal_kernel, iterations=1)

img2 = img.copy()
contours = cv2.findContours(dilate, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
contours = contours[0] if len(contours) == 2 else contours[1]
for c in contours:
  x, y, w, h = cv2.boundingRect(c)
  img2 = cv2.rectangle(img, (x, y), (x + w, y + h), (0,255,0), 2)

You can get the coordinates for each line from cv2.boundingRect(). This can be seen in the image above. Using those coordinates you can crop each line in the document and feed it to pytesseract library.