real time video processing with android openCV

Question

I am a first time android programmer. The project I am working on requires me to do (simple?) real time video processing. The app, once finished needs to do this: When we click on the inbuilt camera application, it opens. I then proceed to choose the video recording option. Using that I can see the surroundings without needing to record. What I am trying to accomplish is to delay that display by a few hundred milliseconds. A colleague of mine could do this pretty easily with the delay option using the laptop webcam and openCV (for computers). I am trying to accomplish the same with an android phone.

Perhaps I am doing a poor job of explaining the situation. Kindly reply at the earliest. I am working on the code now and being a first time programmer taking some time. Excited to start with Android programming!

Answer 1

no idea if this task actually needs opencv ( might be a bit of overkill ) but if you opt for that, its fairly easy.

see all we do here is record frames continuously, and toggle between realtime/playback mode on some event (onTouch for simplicity here):

package com.berak.echo;

import java.util.ArrayList;
import java.util.List;

import android.os.Bundle;
import android.view.Menu;
import android.view.MotionEvent;
import android.view.View;
import android.view.View.OnTouchListener;
import android.app.Activity;

import org.opencv.android.BaseLoaderCallback;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.LoaderCallbackInterface;
import org.opencv.android.OpenCVLoader;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.Point;
import org.opencv.core.Scalar;

import com.berak.echo.R;

public class EchoActivity extends Activity implements CvCameraViewListener2, OnTouchListener {
    CameraBridgeViewBase mOpenCvCameraView;
    List<Mat> ring = new ArrayList<Mat>(); // recording buffer
    int delay = 100;                       // delay == length of buffer
    boolean delayed = false;               // state

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_echo);
        mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.cam3_surface_view);
        mOpenCvCameraView.setCvCameraViewListener(this);
        mOpenCvCameraView.setOnTouchListener(this);    // setup as touchlistener
    }

    // lots of boilerplate, ugly, but needed.
    private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
        @Override
        public void onManagerConnected(int status) {
            switch (status) {
            case LoaderCallbackInterface.SUCCESS:
                mOpenCvCameraView.enableView();
                break;
            default:
                super.onManagerConnected(status);
                break;
            }
        }
    };   
    @Override
    public void onResume() {;
        super.onResume();
        OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_5,this, mLoaderCallback);
    }
    @Override
    public void onPause() {
        super.onPause();
        if (mOpenCvCameraView != null)
            mOpenCvCameraView.disableView();
    }
    @Override
    public void onDestroy() {
        super.onDestroy();
    }   
    @Override
    public void onCameraViewStarted(int width, int height) { }   
    @Override
    public void onCameraViewStopped() { }


    // here's the bread & butter stuff:
    @Override
    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
        Mat mRgba = inputFrame.rgba();
        ring.add(mRgba.clone());            // add one at the end
        if ( ring.size() >= delay ) {       // pop one from the front
            ring.get(0).release();
            ring.remove(0);
        }

        Mat ret;
        String txt;
        if ( delayed && ring.size()>0 ) {   // depending on 'delayed' return either playback
            ret = ring.get(0);              // return the 'oldest'
            txt = "playback";
        } else {
            ret = mRgba;                    // or realtime frame
            txt = "realtime";
        }
        Core.putText(ret, txt, new Point(20,20), Core.FONT_HERSHEY_PLAIN, 1.2, new Scalar(200,0,0));
        return ret;
    }

    @Override
    public boolean onTouch(View v, MotionEvent event) {
        // just toggle between delayed an realtime view:
        delayed = ! delayed;
        return false;
    }    
}