Way to get CustomPainter to track face in Camera Flutter MLKit

Question

I am trying to track a users face in Flutter and draw a green rectangle over this using CustomPainter. This is the FacePainter which I used. The issue I am having is, it takes one picture at the start and copies the image onto the screen with the detected face and thats all.

My code starts off doing

_initializeControllerFuture = _controller.initialize().then((_) { _controller.startImageStream((CameraImage image) { _processCameraImage(image, widget.camera);

It correctly identifies the face but the there is an issue with the image that is sent to CustomPainter. Since I have a CameraImage, I now need to convert this into an ui.Image for CustomPainter. So I do a conversion from yuv420 to rbga8888, in this order.

  List<Uint8List> getPlanes(CameraImage availableImage) {
List<Uint8List> planes = [];

for (int planeIndex = 0; planeIndex < 3; planeIndex++) {
  Uint8List buffer;
  int width;
  int height;
  if (planeIndex == 0) {
    width = availableImage.width;
    height = availableImage.height;
  } else {
    width = availableImage.width ~/ 2;
    height = availableImage.height ~/ 2;
  }

  buffer = Uint8List(width * height);

  int pixelStride = availableImage.planes[0].bytesPerPixel!;
  int rowStride = availableImage.planes[0].bytesPerRow;
  int index = 0;
  for (int i = 0; i < height; i++) {
    for (int j = 0; j < width; j++) {
      buffer[index++] =
          availableImage.planes[0].bytes[i * rowStride + j * pixelStride];
    }
  }

  planes.add(buffer);
}
return planes;

}

and then once this is done, the plane is passed into

  Uint8List yuv420ToRgba8888(List<Uint8List> planes, int width, int height) {
final yPlane = planes[0];
final uPlane = planes[1];
final vPlane = planes[2];

final Uint8List rgbaBytes = Uint8List(width * height * 4);

for (int y = 0; y < height; y++) {
  for (int x = 0; x < width; x++) {
    final int yIndex = y * width + x;
    final int uvIndex = (y ~/ 2) * (width ~/ 2) + (x ~/ 2);

    final int yValue = yPlane[yIndex] & 0xFF;
    final int uValue = uPlane[uvIndex] & 0xFF;
    final int vValue = vPlane[uvIndex] & 0xFF;

    final int r = (yValue + 1.13983 * (vValue - 128)).round().clamp(0, 255);
    final int g =
        (yValue - 0.39465 * (uValue - 128) - 0.58060 * (vValue - 128))
            .round()
            .clamp(0, 255);
    final int b = (yValue + 2.03211 * (uValue - 128)).round().clamp(0, 255);

    final int rgbaIndex = yIndex * 4;
    rgbaBytes[rgbaIndex] = r.toUnsigned(8);
    rgbaBytes[rgbaIndex + 1] = g.toUnsigned(8);
    rgbaBytes[rgbaIndex + 2] = b.toUnsigned(8);
    rgbaBytes[rgbaIndex + 3] = 255; // Alpha value
  }
}

return rgbaBytes;

}

and the final processing is this

  Future<ui.Image> createImage(
  Uint8List buffer, int width, int height, ui.PixelFormat pixelFormat) {
final Completer<ui.Image> completer = Completer();

ui.decodeImageFromPixels(buffer, width, height, pixelFormat,
    (ui.Image img) {
  completer.complete(img);
});

return completer.future;

}

I also convert the CameraImage into Input Input using the _inputImageFromCameraImage in the official docs Im not sure if its something to do with the way Im rotation the image here

  InputImage? _inputImageFromCameraImage(
  CameraImage image, CameraDescription camera) {
final sensorOrientation = camera.sensorOrientation;

InputImageRotation? rotation;

var rotationCompensation =
    _orientations[_controller.value.deviceOrientation];

if (rotationCompensation == null) return null;
if (camera.lensDirection == CameraLensDirection.front) {
  rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
} else {
  rotationCompensation =
      (sensorOrientation - rotationCompensation + 360) % 360;
}

rotation = InputImageRotationValue.fromRawValue(rotationCompensation) ??
    InputImageRotation.rotation0deg;
final format = InputImageFormatValue.fromRawValue(image.format.raw) ??
    InputImageFormat.nv21;

final inputImageData = InputImageMetadata(
  size: ui.Size(image.width.toDouble(), image.height.toDouble()),
  rotation: rotation,
  format: format,
  bytesPerRow: image.planes[0].bytesPerRow,
);
final plane = image.planes.first;

final inputImage = InputImage.fromBytes(
  bytes: plane.bytes,
  metadata: inputImageData,
);

return inputImage;

}

once the processing of the images is done, its passed into CustomPainter like so

CameraPreview(_controller), if (_currentImage != null) CustomPaint( painter: FacePainter(_currentImage!, _detectedFaces), ),

I believe it might be to do with the way the image is being rotated, but Im completely lost which part is making the image too green and how to get the Painter to draw on the stream instead. Any help is appreciated