Android + OpenCV + Face detection + Custom Layout

Question

I am using:

• Android 4.0.3
• OpenCV 2.4.2
• Samsung Galaxy S2

The face-detection example (from the opencv 2.4.2) is working perfectly. But now, I would like to create a custom layout and actually work with just the data extracted from face detection and build a game on it. Not necessarily having the FdView surface taking the entire screen.

I have done these modifications below, but just a black screen is displayed. Nothing appears on the screen.

Added a fd.xml layout:

<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="horizontal">

    <org.opencv.samples.fd.FdView android:id="@+id/FdView" 
        android:layout_width="640dp" 
        android:layout_height="480dp"
        android:visibility="visible"
        />

    <TextView
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:textColor="#FF0000"
        android:text="hi"/>

Modified the baseLoaderCallback of FdActivity.java:

    private BaseLoaderCallback  mOpenCVCallBack = new BaseLoaderCallback(this) {
    @Override
    public void onManagerConnected(int status) {
        switch (status) {
            case LoaderCallbackInterface.SUCCESS:
            {
                Log.i(TAG, "OpenCV loaded successfully");

                // Load native libs after OpenCV initialization
                System.loadLibrary("detection_based_tracker");

                //EXPERIMENT
                setContentView(R.layout.fd);
                FdView surface = (FdView) (findViewById(R.id.FdView));

                surface = mView;
                // Create and set View
                mView = new FdView(mAppContext);
                mView.setDetectorType(mDetectorType);
                mView.setMinFaceSize(0.2f);
                //setContentView(mView);


                // Check native OpenCV camera
                if( !mView.openCamera() ) {
                    AlertDialog ad = new AlertDialog.Builder(mAppContext).create();
                    ad.setCancelable(false); // This blocks the 'BACK' button
                    ad.setMessage("Fatal error: can't open camera!");
                    ad.setButton("OK", new DialogInterface.OnClickListener() {
                        public void onClick(DialogInterface dialog, int which) {
                        dialog.dismiss();
                        finish();
                        }
                    });
                    ad.show();
                }
            } break;
            default:
            {
                super.onManagerConnected(status);
            } break;
        }
    }
};

Added constructors in FdView.java:

    public FdView(Context context, AttributeSet attrs, int defStyle) {
    super(context, attrs, defStyle);
    // TODO Auto-generated constructor stub
}

public FdView(Context context, AttributeSet attrs) {
    super(context, attrs);
    // TODO Auto-generated constructor stub
}

Added constructors in SampleCvViewBase.java:

    public SampleCvViewBase(Context context, AttributeSet attrs, int defStyle) {
    super(context, attrs, defStyle);
    // TODO Auto-generated constructor stub
}

public SampleCvViewBase(Context context, AttributeSet attrs) {
    super(context, attrs);
    // TODO Auto-generated constructor stub
}

Answer 1

Hah, I figured out one way. You could just simply separate the OpenCV Loader and the custom layout.

Define BaseLoaderCallback mOpenCVCallBack.

    private BaseLoaderCallback mOpenCVCallBack = new BaseLoaderCallback(this) {

    @Override
    public void onManagerConnected(int status) {
        switch (status) {
        case LoaderCallbackInterface.SUCCESS: {
            Log.i(TAG, "OpenCV loaded successfully");

            // Load native library after(!) OpenCV initialization
            System.loadLibrary("native_sample");    
        }
            break;
        default: {
            super.onManagerConnected(status);
        }
            break;
        }
    }
};

In OnCreat, build your custom layout, load the OpenCv Loader,

    public void onCreate(Bundle savedInstanceState) {
    Log.i(TAG, "onCreate");
    super.onCreate(savedInstanceState);
    requestWindowFeature(Window.FEATURE_NO_TITLE);

    // /////////////////////////////////////////////////////////////////////
    // // begin:
    // // Create and set View
    setContentView(R.layout.main);
    mView = (Sample3View) findViewById(R.id.sample3view);

    mcameraButton = (ImageView) findViewById(R.id.cameraButton);

    if (!OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_2, this, mOpenCVCallBack)) {
        Log.e(TAG, "Cannot connect to OpenCV Manager");
    }

}

Just that! I did that, and it worked very well.

Answer 2

I met the same problem that I wanted to create a custom view using layout. OpenCV 2.4.2 seems not to offer this function. OpenCV 2.4.3 has the function, but its tutorial doesn't say so (it uses the old example from OpenCV2.4.2). Its Android samples provide some insights. Finally I found the instruction in OpenCV 2.4.9 documentation.

Hope it helps.

Answer 3

Sry not a real answer (yet) but also tried to make a custom layout in opencv 2.4.2

i have this perfectly working solution for 2.4.0 if i remember it right it was enough to add the instructors.. but it doesn't work with 2.4.2

i'll try to figure smthg out and let you guys know.

Answer 4

I have precisely the same issue. Also trying to figure it out. I'm trying to display the image on a SurfaceView that doesn't take the whole screen. And with that I read that you can't have your Camera handler class and linked SurfaceView in different classes. So smashed everything into one.

So, at the moment I have the camera displaying on the SurfaceView, and copying the frame data to a mFrame variable. Basically just struggling to get the mFrame processed (in the multi-threading, Run(), function) and showing the result on the SurfaceView.

This is the code I have, if you think it would help: (excuse the formatting as my code is also a work in progress)

package org.opencv.samples.tutorial3;

import java.io.IOException;
import java.util.List;

import org.opencv.android.BaseLoaderCallback;
import org.opencv.android.LoaderCallbackInterface;
import org.opencv.android.OpenCVLoader;

import android.app.Activity;
import android.app.AlertDialog;
import android.content.Context;
import android.content.DialogInterface;
import android.graphics.Bitmap;
import android.graphics.Canvas;
import android.graphics.Color;
import android.graphics.ImageFormat;
import android.graphics.Paint;
import android.graphics.Rect;
import android.graphics.RectF;
import android.hardware.Camera;
import android.hardware.Camera.PreviewCallback;
import android.os.Bundle;
import android.util.Log;
import android.view.SurfaceHolder;
import android.view.SurfaceView;
import android.view.Window;
import android.widget.TextView;

public class Sample3Native extends Activity implements SurfaceHolder.Callback,Runnable{


    //Camera variables
    private Camera              cam;
    private boolean             previewing = false;
    private SurfaceHolder       mHolder;
    private SurfaceView         mViewer;
    private int                 mFrameWidth;
    private int                 mFrameHeight;
    private byte[]              mFrame;
    private boolean             mThreadRun;
    private byte[]              mBuffer;
    Sample3View viewclass;
    TextView text;
    int value = 0;
    //==========

    int framecount = 0;

    private BaseLoaderCallback  mOpenCVCallBack = new BaseLoaderCallback(this) {
        @Override
        public void onManagerConnected(int status) {
            switch (status) {
                case LoaderCallbackInterface.SUCCESS:
                {

                    // Load native library after(!) OpenCV initialization
                    System.loadLibrary("native_sample");

                    //constructor for viewclass that works on frames
                    viewclass = new Sample3View();

                    //setContentView(mView);
                    //OpenCam();
                    //setContentView(R.layout.main);

                    // Create and set View
                    CameraConstruct();
                    Camopen();

                } break;
                default:
                {
                    super.onManagerConnected(status);
                } break;
            }
        }
    };

    public Sample3Native()
    {}

    @Override
    public void onCreate(Bundle savedInstanceState) 
    {
        super.onCreate(savedInstanceState);
        requestWindowFeature(Window.FEATURE_NO_TITLE);

        setContentView(R.layout.main);

        OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_2, this, mOpenCVCallBack);
    }

    //Camera construction
    public void CameraConstruct()
    {
        mViewer = (SurfaceView)findViewById(R.id.camera_view);
        text = (TextView)findViewById(R.id.text);
        mHolder = mViewer.getHolder();
        mHolder.addCallback(this);
        mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
    }


    //calls camera screen setup when screen surface changes
    public void surfaceChanged(SurfaceHolder holder, int format, int width,int height) 
    {
        CamStartDisplay();  
    }

    public void Camclose()
    {
        if(cam != null && previewing)
        {
            cam.setPreviewCallback(null);
            cam.stopPreview();
            cam.release();
            cam = null;

            previewing = false;
        }

        mThreadRun = false;
        viewclass.PreviewStopped();
    }

    //only open camera, and get frame data
    public void Camopen()
    {
        if(!previewing){
            cam = Camera.open();
            //rotate display
            cam.setDisplayOrientation(90);
            if (cam != null)
            {
                //copy viewed frame
                cam.setPreviewCallbackWithBuffer(new PreviewCallback() 
                {

                    public void onPreviewFrame(byte[] data, Camera camera) 
                    {
                        synchronized (this) 
                        {
                            System.arraycopy(data, 0, mFrame, 0, data.length);

                            this.notify(); 
                        }
                        //text.setText(Integer.toString(value++));
                        camera.addCallbackBuffer(mBuffer);
                    }
                });

            }

        }//if not previewing
    }

    //start preview
    public void CamStartDisplay()
    {
        synchronized (this) 
        {
            if(cam != null)
            {
                //stop previewing till after settings is changed
                if(previewing == true)
                {
                    cam.stopPreview();
                    previewing = false;
                }

                Camera.Parameters p = cam.getParameters();
                for(Camera.Size s : p.getSupportedPreviewSizes())
                {
                    p.setPreviewSize(s.width, s.height);
                    mFrameWidth = s.width;
                    mFrameHeight = s.height;
                    break;
                }

                p.setPreviewSize(mFrameWidth, mFrameHeight);

                List<String> FocusModes = p.getSupportedFocusModes();
                if (FocusModes.contains(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO))
                {
                    p.setFocusMode(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO);
                }
                cam.setParameters(p);

                //set the width and height for processing
                viewclass.setFrame(mFrameWidth, mFrameHeight);

                int size = mFrameWidth*mFrameHeight;
                size  = size * ImageFormat.getBitsPerPixel(p.getPreviewFormat()) / 8;
                mBuffer = new byte[size];
                mFrame = new byte [size];
                cam.addCallbackBuffer(mBuffer);

                viewclass.PreviewStarted(mFrameWidth, mFrameHeight);

                //start display streaming
                try 
                {
                    //cam.setPreviewDisplay(null);
                    cam.setPreviewDisplay(mHolder);
                    cam.startPreview();
                    previewing = true;
                } 
                catch (IOException e) 
                {
                    e.printStackTrace();
                }

            }//end of if cam != null
        }//synchronising
    }

    //thread gets started when the screen surface is created
    public void surfaceCreated(SurfaceHolder holder) {
        //Camopen();
        //CamStartDisplay();
        (new Thread(this)).start(); 
    }

    //called when the screen surface is stopped
    public void surfaceDestroyed(SurfaceHolder holder) 
    {
        Camclose();
    }

    //this is function that is run by thread
    public void run() 
    {

        mThreadRun = true;
        while (mThreadRun) 
        {
            //text.setText(Integer.toString(value++));
            Bitmap bmp = null;

            synchronized (this) 
            {
                try 
                {
                    this.wait();

                    bmp = viewclass.processFrame(mFrame);
                } 
                catch (InterruptedException e) {}
            }

            if (bmp != null) 
            {
                Canvas canvas = mHolder.lockCanvas();

                if (canvas != null) 
                {
                    canvas.drawBitmap(bmp, (canvas.getWidth() - mFrameWidth) / 2, (canvas.getHeight() - mFrameHeight) / 2, null);
                    mHolder.unlockCanvasAndPost(canvas);
                }
            }//if bmp != null
        }// while thread in run
    }


}//end class

Sample3View as used in this class just includes the processFrame function as such:

package org.opencv.samples.tutorial3;

import android.content.Context;
import android.graphics.Bitmap;
import android.widget.TextView;

class Sample3View {

    private int mFrameSize;
    private Bitmap mBitmap;
    private int[] mRGBA;

    private int frameWidth;
    private int frameHeight;
    private int count = 0;

    Sample3Native samp;

    //constructor
    public Sample3View() 
    {
    }

    public void setFrame(int width,int height)
    {
        frameWidth = width;
        frameHeight = height;
    }

    public void PreviewStarted(int previewWidtd, int previewHeight) {
        mFrameSize = previewWidtd * previewHeight;
        mRGBA = new int[mFrameSize];
        mBitmap = Bitmap.createBitmap(previewWidtd, previewHeight, Bitmap.Config.ARGB_8888);
    }

    public void PreviewStopped() {
        if(mBitmap != null) {
            mBitmap.recycle();
            mBitmap = null;
        }
        mRGBA = null;   
    }

    public Bitmap processFrame(byte[] data) {
        int[] rgba = mRGBA;

        FindFeatures(frameWidth, frameHeight, data, rgba);

        Bitmap bmp = mBitmap; 
        bmp.setPixels(rgba, 0, frameWidth, 0, 0, frameWidth, frameHeight);


        //samp.setValue(count++);
        return bmp;
    }

    public native void FindFeatures(int width, int height, byte yuv[], int[] rgba);
}

So yeah, hope this helps. If I get the complete solution working, I'll post that also. Also post your stuff if you get the solution first please! Enjoy