How to convert OpenCV Mat input frame to Tensorflow tensor in Android Studio?

Question

I've been trying to run a Tensorflow model on android. The solution to do this was to create a tensorflow model(I used a pretrained Mobilenetv2 model) first. After training it on my own dataset, I converted it to a .tflite model which is supported by Android. Since I want to work with realtime video analysis, I am also using OpenCV library built for Android SDK.

Now the part where I'm currently stuck is - how to convert the inputframe received by opencv JavaCameraView and feed it to the tflite model for inference? I found few solutions to convert Mat datatype to an Input Tensor but nothing seems clear. Can someone help me out with this?

edit : Here's the code(need help with onCameraFrame Method below)

public class MainActivity extends AppCompatActivity implements CameraBridgeViewBase.CvCameraViewListener2 {`enter code here`

    CameraBridgeViewBase cameraBridgeViewBase;
    BaseLoaderCallback baseLoaderCallback;
  //  int counter = 0;
    Interpreter it;


    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);


        cameraBridgeViewBase = (JavaCameraView)findViewById(R.id.CameraView);
        cameraBridgeViewBase.setVisibility(SurfaceView.VISIBLE);
        cameraBridgeViewBase.setCvCameraViewListener(this);
        try{
            it=new Interpreter(loadModelFile(this));
        }
        catch(Exception e){
            Toast.makeText(this,"Tf model didn't load",Toast.LENGTH_LONG).show();
        }

        //System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
        baseLoaderCallback = new BaseLoaderCallback(this) {
            @Override
            public void onManagerConnected(int status) {
                super.onManagerConnected(status);

                switch(status){

                    case BaseLoaderCallback.SUCCESS:
                        cameraBridgeViewBase.enableView();
                        break;
                    default:
                        super.onManagerConnected(status);
                        break;
                }


            }

        };




    }
    private MappedByteBuffer loadModelFile(Activity activity) throws IOException {
        AssetFileDescriptor fileDescriptor = activity.getAssets().openFd("model.tflite");
        FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
        FileChannel fileChannel = inputStream.getChannel();
        long startOffset = fileDescriptor.getStartOffset();
        long declaredLength = fileDescriptor.getDeclaredLength();
        return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
    }
    @Override
    public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {

       //how to convert inputFrame to Input Tensor???



        
    }


    @Override
    public void onCameraViewStarted(int width, int height) {

    }


    @Override
    public void onCameraViewStopped() {

    }


    @Override
    protected void onResume() {
        super.onResume();

        if (!OpenCVLoader.initDebug()){
            Toast.makeText(getApplicationContext(),"There's a problem, yo!", Toast.LENGTH_SHORT).show();
        }

        else
        {
            baseLoaderCallback.onManagerConnected(baseLoaderCallback.SUCCESS);
        }



    }

    @Override
    protected void onPause() {
        super.onPause();
        if(cameraBridgeViewBase!=null){

            cameraBridgeViewBase.disableView();
        }

    }


    @Override
    protected void onDestroy() {
        super.onDestroy();
        if (cameraBridgeViewBase!=null){
            cameraBridgeViewBase.disableView();
        }
    }
}

Answer 1

I suggest that you convert Mat into FloatBuffer as follows:

Mat floatMat = new Mat();
mat.convertTo(floatMat, CV_32F);
FloatBuffer floatBuffer = floatMat.createBuffer();

Note that the createBuffer method is found within the Mat class of the import org.bytedeco.opencv.opencv_core.Mat not the import org.opencv.core.

Then you can create a tensor from the floatBuffer variable:

Tensor.create(new long[]{1, image_height, image_width, 3}, floatBuffer)

This creates a tensor that contains a batch of one image (as indicated by the number 1 on the far left), with an image of dimensions (image_height, image_width, 3) which you should know and replace. Most image processing and machine learning libraries use the first dimension for the height of the image or the "rows" and the second for the width or the "columns" and the third for the number of channels (RGB = 3 channels). If you have a grayscale image, then replace 3 by 1.

Please check whether you can directly feed this tensor to your model or you have to perform some pre-processing steps first such as normalization.