opencv, find a letter located at a specific location of a picture?

Question

friends, could you please help with my questions?

I am using opencv in c++.
I am randomly cropping a small picture from a camera view. I want to find the word located at the bottom of this cropped picture, and this word should also be penetrated by the vertical center line (imaginary) of this cropped picture. please see the following code :

char* my_word = do_ocr(my_cropped_image);

and the do_ocr function is like this:

char* do_ocr(cv::Mat im)
{
cv::Mat gray;
cv::cvtColor(im, gray, CV_BGR2GRAY);
// ...other image pre-processing here...

// Pass it to Tesseract API
tesseract::TessBaseAPI tess;
tess.Init(NULL, "eng", tesseract::OEM_DEFAULT);
tess.SetPageSegMode(tesseract::PSM_SINGLE_BLOCK);
tess.SetImage((uchar*)gray.data, gray.cols, gray.rows, 1, gray.cols);

// Get the text
char* out = tess.GetUTF8Text();
std::cout << out << std::endl;
return out;
}

The following is the schematic diagram and some samples of my_cropped_image :

my_cropped_image sample # 1, the letter "preceding" should be detected:

my_cropped_image sample # 2, the letter "advantageous" should be detected:

my_cropped_image sample # 3, the letter "Correlation" should be detected:

my_cropped_image sample # 4, the letter "density" should be detected:

my_cropped_image sample # 5, the letter "time" should be detected:

I'll appreciate the helps from you to update my do_ocr function.

Thank you and have a great day!

Answer 1

Are these the results you were looking for?

---

Methodology:

1) Binaryze the image, white is foreground. Here is simply done with img = img < 150;. You can use more sophisticated methods, like adaptiveThreshold. You get something like:

2) Apply a open morphological operation, so that all the letters in a single word for a single blob:

3) Find the rectangle of each connected component:

4) Take the bottom one, in the center.

Here the full code:

#include <opencv2\opencv.hpp>
#include <vector>
using namespace std;
using namespace cv;

Mat3b dbg;

int main()
{
    Mat1b img = imread("path_to_image", IMREAD_GRAYSCALE);
    cvtColor(img, dbg, COLOR_GRAY2BGR);

    Mat3b result;
    cvtColor(img, result, COLOR_GRAY2BGR);

    Mat1b img2;
    img2 = img < 150; 

    Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(5,3));
    morphologyEx(img2, img2, MORPH_DILATE, kernel);

    // Apply a small border
    copyMakeBorder(img2, img2, 5, 5, 5, 5, BORDER_CONSTANT, Scalar(0));

    // Take the bounding boxes of all connected components
    vector<vector<Point>> contours;
    findContours(img2.clone(), contours, CV_RETR_LIST, CHAIN_APPROX_NONE);

    int minArea = 60;
    vector<Rect> rects;
    for (int i = 0; i < contours.size(); ++i)
    {
        Rect r = boundingRect(contours[i]);
        if (r.area() >= minArea)
        {
            // Account for border
            r -= Point(5,5);
            rects.push_back(r);
        }
    }

    int middle = img.cols / 2;

    // Keep bottom rect, containig middle point
    if (rects.empty()) return -1;

    Rect word;
    for (int i = 1; i < rects.size(); ++i)
    {
        Point pt(middle, rects[i].y + rects[i].height/2);
        if (rects[i].contains(pt))
        {
            if (rects[i].y > word.y)
            {
                word = rects[i];
            }
        }
    }

    // Show results
    Mat3b res;
    cvtColor(img, res, COLOR_GRAY2BGR);
    for (int i = 0; i < rects.size(); ++i)
    {
        rectangle(res, rects[i], Scalar(0, 255, 0));
    }

    rectangle(result, word, Scalar(0, 0, 255), 2);

    imshow("Rects", res);
    imshow("Result", result);
    waitKey();

    return 0;
}