Issue with OCR Scan on Android

Question

I'm trying to create an app with a OCR Scanner by using the tessract/tess-two library, I've successful access to the Phone camera, I can do the manual focus but when I take a picture the following error:

07-18 19:07:06.335 2585-2585/com.fastnetserv.myapp D/DBG_com.fastnetserv.myapp.MainActivity: Picture taken
07-18 19:07:06.335 2585-2585/com.fastnetserv.myapp D/DBG_com.fastnetserv.myapp.MainActivity: Got null data
07-18 19:07:06.405 2585-2585/com.fastnetserv.myapp D/DBG_com.fastnetserv.myapp.MainActivity: Picture taken
07-18 19:07:06.426 2585-2585/com.fastnetserv.myapp D/DBG_com.fastnetserv.myapp.MainActivity: Got bitmap
07-18 19:07:06.427 2585-11599/com.fastnetserv.myapp E/DBG_com.fastnetserv.myapp.TessAsyncEngine: Error passing parameter to execute(context, bitmap)
07-18 19:07:14.111 2585-2585/com.fastnetserv.myapp D/DBG_com.fastnetserv.myapp.CameraUtils: CameraEngine Stopped

Here the CameraFragment code:

package com.fastnetserv.myapp;

import android.content.Context;
import android.graphics.Bitmap;
import android.hardware.Camera;
import android.net.Uri;
import android.os.AsyncTask;
import android.os.Bundle;
import android.support.v4.app.Fragment;
import android.util.Log;
import android.view.LayoutInflater;
import android.view.SurfaceHolder;
import android.view.SurfaceView;
import android.view.View;
import android.view.ViewGroup;
import android.widget.Button;

import com.googlecode.tesseract.android.TessBaseAPI;


/**
 * A simple {@link Fragment} subclass.
 * Activities that contain this fragment must implement the
 * {@link //CameraFragment.//OnFragmentInteractionListener} interface
 * to handle interaction events.
 * Use the {@link CameraFragment#newInstance} factory method to
 * create an instance of this fragment.
 */
public class CameraFragment extends Fragment implements SurfaceHolder.Callback, View.OnClickListener,
        Camera.PictureCallback, Camera.ShutterCallback {

    static final String TAG = "DBG_" + MainActivity.class.getName();

    Button shutterButton;
    Button focusButton;
    FocusBoxView focusBox;
    SurfaceView cameraFrame;
    CameraEngine cameraEngine;

    // TODO: Rename parameter arguments, choose names that match
    // the fragment initialization parameters, e.g. ARG_ITEM_NUMBER
    private static final String ARG_PARAM1 = "param1";
    private static final String ARG_PARAM2 = "param2";

    // TODO: Rename and change types of parameters
    private String mParam1;
    private String mParam2;

    private OnFragmentInteractionListener mListener;

    public CameraFragment() {
        // Required empty public constructor
    }

    /**
     * Use this factory method to create a new instance of
     * this fragment using the provided parameters.
     *
     * @param param1 Parameter 1.
     * @param param2 Parameter 2.
     * @return A new instance of fragment CameraFragment.
     */
    // TODO: Rename and change types and number of parameters
    public static CameraFragment newInstance(String param1, String param2) {
        CameraFragment fragment = new CameraFragment();
        Bundle args = new Bundle();
        args.putString(ARG_PARAM1, param1);
        args.putString(ARG_PARAM2, param2);
        fragment.setArguments(args);
        return fragment;
    }

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        if (getArguments() != null) {
            mParam1 = getArguments().getString(ARG_PARAM1);
            mParam2 = getArguments().getString(ARG_PARAM2);
        }
    }

    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
                             Bundle savedInstanceState) {
        // Inflate the layout for this fragment
        return inflater.inflate(R.layout.fragment_camera, container, false);
    }

    // TODO: Rename method, update argument and hook method into UI event
    public void onButtonPressed(Uri uri) {
        if (mListener != null) {
            mListener.onFragmentInteraction(uri);
        }
    }

    @Override
    public void onAttach(Context context) {
        super.onAttach(context);
        try {
            mListener = (OnFragmentInteractionListener) context;
        } catch (ClassCastException e) {
            throw new ClassCastException(context.toString()
                    + " must implement OnFragmentInteractionListener");
        }
    }

    @Override
    public void onDetach() {
        super.onDetach();
        mListener = null;
    }

    // Camera Code

    public String detectText(Bitmap bitmap) {

        TessDataManager.initTessTrainedData(getActivity());
        TessBaseAPI tessBaseAPI = new TessBaseAPI();

        String path = "/mnt/sdcard/com.fastnetserv.myapp/tessdata/ita.traineddata";

        Log.d(TAG, "Check data path: " + path);

        tessBaseAPI.setDebug(true);
        tessBaseAPI.init(path, "ita"); //Init the Tess with the trained data file, with english language

        //For example if we want to only detect numbers
        tessBaseAPI.setVariable(TessBaseAPI.VAR_CHAR_WHITELIST, "1234567890");
        tessBaseAPI.setVariable(TessBaseAPI.VAR_CHAR_BLACKLIST, "!@#$%^&*()_+=-qwertyuiop[]}{POIU" +
                "YTREWQasdASDfghFGHjklJKLl;L:'\"\\|~`xcvXCVbnmBNM,./<>?");


        tessBaseAPI.setImage(bitmap);

        String text = tessBaseAPI.getUTF8Text();

        //Log.d(TAG, "Got data: " + result);
        tessBaseAPI.end();

        return text;
    }

    @Override
    public void surfaceCreated(SurfaceHolder holder) {

        Log.d(TAG, "Surface Created - starting camera");

        if (cameraEngine != null && !cameraEngine.isOn()) {
            cameraEngine.start();
        }

        if (cameraEngine != null && cameraEngine.isOn()) {
            Log.d(TAG, "Camera engine already on");
            return;
        }

        cameraEngine = CameraEngine.New(holder);
        cameraEngine.start();

        Log.d(TAG, "Camera engine started");
    }

    @Override
    public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {

    }

    @Override
    public void surfaceDestroyed(SurfaceHolder holder) {

    }


    @Override
    public void onResume() {
        super.onResume();

        cameraFrame = (SurfaceView) getActivity().findViewById(R.id.camera_frame);
        shutterButton = (Button) getActivity().findViewById(R.id.shutter_button);
        focusBox = (FocusBoxView) getActivity().findViewById(R.id.focus_box);
        focusButton = (Button) getActivity().findViewById(R.id.focus_button);

        shutterButton.setOnClickListener(this);
        focusButton.setOnClickListener(this);

        SurfaceHolder surfaceHolder = cameraFrame.getHolder();
        surfaceHolder.addCallback(this);
        surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);

        cameraFrame.setOnClickListener(this);
    }

    @Override
    public void onPause() {
        super.onPause();

        if (cameraEngine != null && cameraEngine.isOn()) {
            cameraEngine.stop();
        }

        SurfaceHolder surfaceHolder = cameraFrame.getHolder();
        surfaceHolder.removeCallback(this);
    }

    @Override
    public void onClick(View v) {
        if(v == shutterButton){
            if(cameraEngine != null && cameraEngine.isOn()){
                cameraEngine.takeShot(this, this, this);
            }
        }

        if(v == focusButton){
            if(cameraEngine!=null && cameraEngine.isOn()){
                cameraEngine.requestFocus();
            }
        }
    }

    @Override
    public void onPictureTaken(byte[] data, Camera camera) {

        Log.d(TAG, "Picture taken");

        if (data == null) {
            Log.d(TAG, "Got null data");
            return;
        }

        Bitmap bmp = Tools.getFocusedBitmap(getActivity(), camera, data, focusBox.getBox());

        Log.d(TAG, "Got bitmap");

        new TessAsyncEngine().executeOnExecutor(AsyncTask.SERIAL_EXECUTOR, this, bmp);

    }

    @Override
    public void onShutter() {

    }

}

And here the TessAsyncEngine:

package com.fastnetserv.myapp;

import android.app.Activity;
import android.graphics.Bitmap;
import android.os.AsyncTask;
import android.util.Log;

import com.fastnetserv.myapp.ImageDialog;
import com.fastnetserv.myapp.Tools;

/**
 * Created by Fadi on 6/11/2014.
 */
public class TessAsyncEngine extends AsyncTask<Object, Void, String> {

    static final String TAG = "DBG_" + TessAsyncEngine.class.getName();

    private Bitmap bmp;

    private Activity context;


    @Override
    protected String doInBackground(Object... params) {

        try {

            if(params.length < 2) {
                Log.e(TAG, "Error passing parameter to execute - missing params");
                return null;
            }

            if(!(params[0] instanceof Activity) || !(params[1] instanceof Bitmap)) {
                Log.e(TAG, "Error passing parameter to execute(context, bitmap)");

                return null;
            }

            context = (Activity)params[0];

            bmp = (Bitmap)params[1];

            if(context == null || bmp == null) {
                Log.e(TAG, "Error passed null parameter to execute(context, bitmap)");
                return null;
            }

            int rotate = 0;

            if(params.length == 3 && params[2]!= null && params[2] instanceof Integer){
                rotate = (Integer) params[2];
            }

            if(rotate >= -180 && rotate <= 180 && rotate != 0)
            {
                bmp = Tools.preRotateBitmap(bmp, rotate);
                Log.d(TAG, "Rotated OCR bitmap " + rotate + " degrees");
            }

            TessEngine tessEngine =  TessEngine.Generate(context);

            bmp = bmp.copy(Bitmap.Config.ARGB_8888, true);

            String result = tessEngine.detectText(bmp);

            Log.d(TAG, result);

            return result;

        } catch (Exception ex) {
            Log.d(TAG, "Error: " + ex + "\n" + ex.getMessage());
        }

        return null;
    }

    @Override
    protected void onPostExecute(String s) {

        if(s == null || bmp == null || context == null)
            return;

        ImageDialog.New()
                .addBitmap(bmp)
                .addTitle(s)
                .show(context.getFragmentManager(), TAG);

        super.onPostExecute(s);
    }
}

I have followed this tutorial (http://www.codeproject.com/Tips/840623/Android-Character-Recognition) but probably I forgot something due to the lack of my knowledge with Android

Answer 1

That if(context == null || bmp == null) is not needed, as you already tested those values with instanceof.

But I'm guessing your main problem is passing this from Fragment as Activity parameter, which is not.

To fix.. I overall would try to not toss Activity pointer around wildly, as those have quite limited life cycle on android. I have an app with tess-two and I don't recall ever needing Activity to init it (although usually I init it from native C++, so YMMV).

Isn't just the Context needed for that call? If yes, I would suggest to move to getApplicationContext() value instead. I think this is directly or indirectly accessible from Fragment too.

Sorry for not trying your code, but this is something you can debug quite easily.

One more note to android and tesseract usage. What is Tools.getFocusedBitmap? Will it cut down the pic reasonably? If it keeps full size, and your Camera is set to full size, you are tossing around 5-10+MP Bitmaps around, which in Android means to hit Out-Of-Memory (OOM) almost instantly. Either set Camera to reasonably low resolution, or cut-out designed part of photo ASAP and drop the whole Image, ideally as first step of processing.

---

Also you may want to reconsider whole tess-two thing, and try the official Google Text API from Google play services.

https://developers.google.com/android/reference/com/google/android/gms/vision/text/Text

It's brand new addition, inside I guess it will use the second generation of Tesseract engine with latest improvements, so very likely to get better results and better speed, than from tess-two.

I think it's accessible only from Android 4.4 and only on devices with Google Play Services, and cross-platform sucks, so I'm staying with tess-two in my projects - as I have to support also iOS and Windows Phone.

And generally I don't believe things which don't come with source along, SW without source is zombie, already dead while you are using it (will take at most 30-50y to die), and it's a waste of time and skill of those programmers.