CODEWITHSUNDEEP
javaandroidandroid-camera
insta catering
insta catering

Reputation: 160

Using onPreviewFrame To run ML models

So i'm using the legacy Camera API (as far as I can tell) to get previewFrame call backs to then run a few machine learning models I have. I have confirmed that the machine learning models work when given a bitmap decoded when I take a picture via the onPictureTaken callback. Right now in the samples below, I am just simply testing on ML Kit's barcode scanner as a base case, but my custom models seemed to work fine with the onPictureTaken callback as well.

From what i've gathered, using onPreviewFrame isn't necessarily the best way to do this, but for the sake of having a quick sample play-around (and learning experience) I decided to just go this route. Based on everything i've tried from others having solutions online, I can't seem to get anything to work properly. The below code returns null:

@Override
public void onPreviewFrame(byte[] data, Camera camera) {
//        Log.d("onPreviewFrame bytes.length", String.valueOf(bytes.length));
//        final Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
//        Log.d("onPreviewFrame bmp.getHeight()", String.valueOf(bmp.getHeight()));

    Camera.Parameters parameters = camera.getParameters();
    int width = parameters.getPreviewSize().width;
    int height = parameters.getPreviewSize().height;


    Log.d("onPreviewFrame - width", String.valueOf(width));
    Log.d("onPreviewFrame - height", String.valueOf(height));
    Log.d("onPreviewFrame - parameters.getPreviewFormat()", String.valueOf(parameters.getPreviewFormat()));

    YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);

    ByteArrayOutputStream out = new ByteArrayOutputStream();
    yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);

//
//        byte[] bytes = out.toByteArray();
//        final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);


    byte[] bytes = yuv.getYuvData();
    final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);

    extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
}

Here's something else I tried:

@Override
public void onPreviewFrame(byte[] data, Camera camera) {
//        Log.d("onPreviewFrame bytes.length", String.valueOf(bytes.length));
//        final Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
//        Log.d("onPreviewFrame bmp.getHeight()", String.valueOf(bmp.getHeight()));

    Camera.Parameters parameters = camera.getParameters();
    int width = parameters.getPreviewSize().width;
    int height = parameters.getPreviewSize().height;


    Log.d("onPreviewFrame - width", String.valueOf(width));
    Log.d("onPreviewFrame - height", String.valueOf(height));

    YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);

    ByteArrayOutputStream out = new ByteArrayOutputStream();
    yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);


    byte[] bytes = out.toByteArray();
    final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);

    extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
}

Unfortunately I got this error:

ML Kit has detected that you seem to pass camera frames to the detector as a Bitmap object. This is inefficient. Please use YUV_420_888 format for camera2 API or NV21 format for (legacy) camera API and directly pass down the byte array to ML Kit.

with parameters.getPreviewFormat() returning 17 which is NV21. I also tried simply by changing that to ImageFormat.YUV_420_888 but that resulted in the below illegal argument exception:

only support ImageFormat.NV21 and ImageFormat.YUY2 for now

Upvotes: 4

Views: 446

Answers (2)

M Ajmal Shah
M Ajmal Shah

Reputation: 11

Insted of directly passing FirebaseVisionImage 

extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);

you can do it like this 

var bitmap = toARGBBitmap(ocrBitmap)
  extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);

 private fun toARGBBitmap(img: Bitmap): Bitmap {
        return img.copy(Bitmap.Config.ARGB_8888, true)
    }

You can try this:)

Upvotes: 0

Shubham Panchal
Shubham Panchal

Reputation: 4299

Instead of using the Camera API, try using CameraX. It's easy to use and you can execute your code whenever a frame is received from the camera. While trying to integrate an ML model with the camera, I faced a similar error and then turned to CameraX.

Basically, we'll create an ImageAnalysis.Analyser class through which we would get the Image object ( frames ). Using an extension function, we will convert this Image object to a YuvImage.

You can follow this codelab to use CameraX to analyze frames. You will create a class that extends ImageAnalysis.Analyser class.

class FrameAnalyser() : ImageAnalysis.Analyzer {

    override fun analyze(image: ImageProxy?, rotationDegrees: Int) {
        val yuvImage = image?.image?.toYuv() // The extension function
    }
}

Create the extension function which transforms the Image to a YuvImage. 

private fun Image.toYuv(): YuvImage {
    val yBuffer = planes[0].buffer
    val uBuffer = planes[1].buffer
    val vBuffer = planes[2].buffer
    val ySize = yBuffer.remaining()
    val uSize = uBuffer.remaining()
    val vSize = vBuffer.remaining()
    val nv21 = ByteArray(ySize + uSize + vSize)
    yBuffer.get(nv21, 0, ySize)
    vBuffer.get(nv21, ySize, vSize)
    uBuffer.get(nv21, ySize + vSize, uSize)
    val yuvImage = YuvImage(nv21, ImageFormat.NV21, this.width, this.height, null)
    return yuvImage
}

You can change the YUV Image format as required. Refer to these docs.

Upvotes: 4

Related Questions