Reputation: 21
I am working on a Flutter project where I need to run a TensorFlow Lite (TFLite) object detection model. The model has specific input and output shape requirements, and I am encountering difficulties preprocessing the input image to match the required input format.
I have the following model details:
[1, 640, 640, 3]
(a single image with 640x640 resolution and 3 color channels)[1, 25200, 7]
(model will output a list of detection results)The main challenge is correctly preprocessing the input image. I need to resize the image to 640x640 pixels and convert it into the correct data format (float32
) that the model expects.
I have tried multiple approaches, but none of them seem to work as expected. The image is not being processed correctly, leading to errors and incorrect predictions.
tflite_flutter_helper
package, but it depends on outdated dependencies, which causes compatibility issues.float32
array, but I am unsure how to correctly convert the image and feed it into the model.I tried multiple approaches to preprocess the input image for my TensorFlow Lite model. Here's what I did:
float32
:import 'dart:io';
import 'package:camera/camera.dart';
import 'package:get/get.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:tflite/tflite.dart';
import 'dart:developer' as developer;
import 'package:flutter/material.dart';
import 'package:tflite_flutter/tflite_flutter.dart';
class ScanController extends GetxController {
@override
void onInit() {
super.onInit();
loadModel();
}
void loadModel() async{
final interpreter = await Interpreter.fromAsset('assets/model.tflite');
var image = File('assets/test.jpg');
var imageBytes = await image.readAsBytes();
// conver the image to float32
final input = imageBytes.buffer.asFloat32List();
// output size is 1 x 25200 x 7
final output = List.filled(1 * 25200 * 7, 0.0).reshape([1, 25200, 7]);
interpreter.run(input, output);
developer.log(output.toString());
}
}
but this didn't work
i also tried to use TensorImage and ImageProcessorBuilder from tflite_flutter_helper in this code
import 'package:get/get.dart';
import 'package:get/get_connect/http/src/utils/utils.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:tflite_flutter/tflite_flutter.dart' as tflite;
import 'dart:developer' as developer;
import 'dart:io';
import 'package:image/image.dart';
import 'package:image/image.dart' as img;
import 'package:tflite_flutter_helper/tflite_flutter_helper.dart';
import 'package:tflite_flutter/tflite_flutter.dart';
import 'package:tflite_flutter_helper/tflite_flutter_helper.dart';
class ScanController extends GetxController {
@override
void onInit() {
super.onInit();
loadModel();
}
void loadModel() async {
try {
final interpreter = await tflite.Interpreter.fromAsset('assets/model.tflite');
developer.log('Model loaded successfully.', name: 'scan_controller.dart');
developer.log('input info:');
var inputShape = interpreter.getInputTensor(0).shape;
developer.log('shape: $inputShape');
var inputType = interpreter.getInputTensor(0).type;
developer.log('type: $inputType');
var outputShape = interpreter.getOutputTensor(0).shape;
developer.log('output shape: $outputShape');
var outputType = interpreter.getOutputTensor(0).type;
developer.log('output type: $outputType');
var file = File('assets/test.jpg');
// Input shape: [1, 640, 640, 3]
ImageProcessor imageProcessor = ImageProcessorBuilder()
.add(ResizeOp(640, 640, ResizeMethod.BILINEAR))
.add(NormalizeOp(0, 255))
.build();
TensorImage tensorImage = TensorImage.fromFile(file);
tensorImage = imageProcessor.process(tensorImage);
// Output shape: [1, 25200, 7]
TensorBuffer outputBuffer = TensorBuffer.createFixedSize(outputShape, tflite.TfLiteType.float32);
// Run inference
interpreter.run(tensorImage.buffer, outputBuffer.buffer);
// print the result
var output = outputBuffer.getDoubleList();
developer.log('output: $output');
} catch (e) {
developer.log('Failed to load model: $e', name: 'scan_controller.dart');
}
}
}
but it also didn't work
Can someone please guide me on how to properly preprocess the input image to match the input shape [1, 640, 640, 3] and convert it to the required float32 format so that it can be passed to the model?
Upvotes: 2
Views: 90
Reputation: 21
After a while i found the solution.
Use this if you take the frames from the camera, after the execution of this method the data will be filled to the input parameter(float32list)
, image format is YUV420
.
void _processCameraImage(CameraImage image, List<List<List<double>>> input) {
final int width = image.width;
final int height = image.height;
final int uvRowStride = image.planes[1].bytesPerRow;
final int uvPixelStride = image.planes[1].bytesPerPixel!;
final bytes = image.planes[0].bytes;
final uvBytes1 = image.planes[1].bytes;
final uvBytes2 = image.planes[2].bytes;
// Calculate scaling factors
final double scaleX = width / 320;
final double scaleY = height / 320;
for (int y = 0; y < 640; y++) {
for (int x = 0; x < 640; x++) {
final int srcX = (x * scaleX).floor();
final int srcY = (y * scaleY).floor();
final int uvIndex =
uvPixelStride * (srcX >> 1) +
uvRowStride * (srcY >> 1);
final int index = srcY * width + srcX;
final yp = bytes[index];
final up = uvBytes1[uvIndex];
final vp = uvBytes2[uvIndex];
// Optimized YUV to RGB conversion
int r = yp + ((1436 * (vp - 128)) >> 10);
int g = yp - ((46549 * (up - 128)) >> 17) - ((93604 * (vp - 128)) >> 17);
int b = yp + ((1814 * (up - 128)) >> 10);
// Clamp and normalize
input[y][x][0] = (r < 0 ? 0 : (r > 255 ? 255 : r)) / 255.0;
input[y][x][1] = (g < 0 ? 0 : (g > 255 ? 255 : g)) / 255.0;
input[y][x][2] = (b < 0 ? 0 : (b > 255 ? 255 : b)) / 255.0;
}
}}
Upvotes: 0