CODEWITHSUNDEEP
flutterdarttensorflow-litetfliteimage-preprocessing
yousef sultan
yousef sultan

Reputation: 21

How to preprocess an image to match TFLite model input shape [1, 640, 640, 3] in Flutter?

Problem:

I am working on a Flutter project where I need to run a TensorFlow Lite (TFLite) object detection model. The model has specific input and output shape requirements, and I am encountering difficulties preprocessing the input image to match the required input format.

I have the following model details:

The main challenge is correctly preprocessing the input image. I need to resize the image to 640x640 pixels and convert it into the correct data format (float32) that the model expects.

I have tried multiple approaches, but none of them seem to work as expected. The image is not being processed correctly, leading to errors and incorrect predictions.

Specific Issues:

What I Tried:

I tried multiple approaches to preprocess the input image for my TensorFlow Lite model. Here's what I did:

  1. First Attempt: I used the following code to load the image and convert it to float32:
import 'dart:io';

import 'package:camera/camera.dart';
import 'package:get/get.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:tflite/tflite.dart';
import 'dart:developer' as developer;
import 'package:flutter/material.dart';
import 'package:tflite_flutter/tflite_flutter.dart';


class ScanController extends GetxController {

  @override
  void onInit() {
    super.onInit();
    loadModel();
  }

  void loadModel() async{
    final interpreter = await Interpreter.fromAsset('assets/model.tflite');

    var image = File('assets/test.jpg');
    var imageBytes = await image.readAsBytes();
    // conver the image to float32
    final input = imageBytes.buffer.asFloat32List();

    // output size is 1 x 25200 x 7
    final output = List.filled(1 * 25200 * 7, 0.0).reshape([1, 25200, 7]);

    interpreter.run(input, output);

    developer.log(output.toString());

  }

}

but this didn't work

i also tried to use TensorImage and ImageProcessorBuilder from tflite_flutter_helper in this code


import 'package:get/get.dart';
import 'package:get/get_connect/http/src/utils/utils.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:tflite_flutter/tflite_flutter.dart' as tflite;
import 'dart:developer' as developer;
import 'dart:io';
import 'package:image/image.dart';
import 'package:image/image.dart' as img;
import 'package:tflite_flutter_helper/tflite_flutter_helper.dart';
import 'package:tflite_flutter/tflite_flutter.dart';
import 'package:tflite_flutter_helper/tflite_flutter_helper.dart';



class ScanController extends GetxController {

  @override
  void onInit() {
    super.onInit();
    loadModel();
  }



void loadModel() async {
  try {
    final interpreter = await tflite.Interpreter.fromAsset('assets/model.tflite');
    developer.log('Model loaded successfully.', name: 'scan_controller.dart');
    
    developer.log('input info:');
    var inputShape = interpreter.getInputTensor(0).shape;
    developer.log('shape: $inputShape');
    var inputType = interpreter.getInputTensor(0).type;
    developer.log('type: $inputType');
    var outputShape = interpreter.getOutputTensor(0).shape;
    developer.log('output shape: $outputShape');
    var outputType = interpreter.getOutputTensor(0).type;
    developer.log('output type: $outputType');

    var file = File('assets/test.jpg');

    // Input shape: [1, 640, 640, 3]
    ImageProcessor imageProcessor = ImageProcessorBuilder()
        .add(ResizeOp(640, 640, ResizeMethod.BILINEAR))
        .add(NormalizeOp(0, 255))
        .build();

    TensorImage tensorImage = TensorImage.fromFile(file);
    tensorImage = imageProcessor.process(tensorImage);

    // Output shape: [1, 25200, 7]
    TensorBuffer outputBuffer = TensorBuffer.createFixedSize(outputShape, tflite.TfLiteType.float32);

    // Run inference
    interpreter.run(tensorImage.buffer, outputBuffer.buffer);

    // print the result
    var output = outputBuffer.getDoubleList();
    developer.log('output: $output');
    

  } catch (e) {
    developer.log('Failed to load model: $e', name: 'scan_controller.dart');
  }
}


    
  }

but it also didn't work

Can someone please guide me on how to properly preprocess the input image to match the input shape [1, 640, 640, 3] and convert it to the required float32 format so that it can be passed to the model?

Upvotes: 2

Views: 90

Answers (1)

yousef sultan
yousef sultan

Reputation: 21

After a while i found the solution.

Use this if you take the frames from the camera, after the execution of this method the data will be filled to the input parameter(float32list), image format is YUV420.

    void _processCameraImage(CameraImage image, List<List<List<double>>> input) {
    final int width = image.width;
    final int height = image.height;
    final int uvRowStride = image.planes[1].bytesPerRow;
    final int uvPixelStride = image.planes[1].bytesPerPixel!;
    
    final bytes = image.planes[0].bytes;
    final uvBytes1 = image.planes[1].bytes;
    final uvBytes2 = image.planes[2].bytes;

    // Calculate scaling factors
    final double scaleX = width / 320;
    final double scaleY = height / 320;

    for (int y = 0; y < 640; y++) {
      for (int x = 0; x < 640; x++) {
        final int srcX = (x * scaleX).floor();
        final int srcY = (y * scaleY).floor();
        
        final int uvIndex = 
          uvPixelStride * (srcX >> 1) + 
          uvRowStride * (srcY >> 1);
        
        final int index = srcY * width + srcX;

        final yp = bytes[index];
        final up = uvBytes1[uvIndex];
        final vp = uvBytes2[uvIndex];

        // Optimized YUV to RGB conversion
        int r = yp + ((1436 * (vp - 128)) >> 10);
        int g = yp - ((46549 * (up - 128)) >> 17) - ((93604 * (vp - 128)) >> 17);
        int b = yp + ((1814 * (up - 128)) >> 10);

        // Clamp and normalize
        input[y][x][0] = (r < 0 ? 0 : (r > 255 ? 255 : r)) / 255.0;
        input[y][x][1] = (g < 0 ? 0 : (g > 255 ? 255 : g)) / 255.0;
        input[y][x][2] = (b < 0 ? 0 : (b > 255 ? 255 : b)) / 255.0;
      }
    }}

Upvotes: 0

Related Questions