tensorflow lite quantized ssd object detection on raspberry pi

Question

Object detection is not working as expected for me on a raspberry pi with tensorflow lite using C++. My code compiles and runs, but the output never seems to get properly populated. Could I be missing any dependencies or accessing the results wrong?

I followed the tutorial on: https://medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193

And have the detect.tflite model from: https://storage.googleapis.com/download.tensorflow.org/models/tflite/pets_ssd_mobilenet_v1_0.75_quant_2018_06_29.zip

I have compiled tensorflow lite and opencv for raspberry pi and modified minimal.cc to read in an image and perform inference as follows:

/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==============================================================================*/
#include <cstdio>
#include "tensorflow/contrib/lite/interpreter.h"
#include "tensorflow/contrib/lite/kernels/register.h"
#include "tensorflow/contrib/lite/model.h"
#include "tensorflow/contrib/lite/optional_debug_tools.h"

#include <iostream>
#include "opencv2/core.hpp"
#include "opencv2/imgcodecs.hpp"

// This is an example that is minimal to read a model
// from disk and perform inference.
//
// Usage: detect <tflite model> <image filepath>

using namespace tflite;

#define TFLITE_MINIMAL_CHECK(x)                              \
  if (!(x)) {                                                \
    fprintf(stderr, "Error at %s:%d\n", __FILE__, __LINE__); \
    exit(1);                                                 \
  }

int main(int argc, char* argv[]) {
  if(argc != 3) {
    fprintf(stderr, "minimal <tflite model filepath> <image filepath>\n");
    return 1;
  } else {
      fprintf(stdout, "Reading model from %s\n", argv[1]);
      fprintf(stdout, "Reading image from %s\n", argv[2]);
  }
  const char* model_filename = argv[1];
  const char* image_filename = argv[2];

  // Load model
  std::unique_ptr<tflite::FlatBufferModel> model =
      tflite::FlatBufferModel::BuildFromFile(model_filename);
  TFLITE_MINIMAL_CHECK(model != nullptr);

  // Build the interpreter
  tflite::ops::builtin::BuiltinOpResolver resolver;
  InterpreterBuilder builder(*model.get(), resolver);
  std::unique_ptr<Interpreter> interpreter;
  builder(&interpreter);
  TFLITE_MINIMAL_CHECK(interpreter != nullptr);

  // Allocate tensor buffers.
  TFLITE_MINIMAL_CHECK(interpreter->AllocateTensors() == kTfLiteOk);
  printf("=== Pre-invoke Interpreter State ===\n");
  tflite::PrintInterpreterState(interpreter.get());

  // Fill input buffers
  // TODO(user): Insert code to fill input tensors
  cv::Mat img = cv::imread(image_filename);
  //std::cout << "before: " << interpreter->typed_input_tensor<uchar>(0) << std::endl;
  //std::cout << "image: " << img.data << std::endl;
  memcpy(interpreter->typed_input_tensor<uchar>(0), img.data, img.total() * img.elemSize());
  //std::cout << "after: " << interpreter->typed_input_tensor<uchar>(0) << std::endl;

  // Run inference
  TFLITE_MINIMAL_CHECK(interpreter->Invoke() == kTfLiteOk);
  printf("\n\n=== Post-invoke Interpreter State ===\n");
  tflite::PrintInterpreterState(interpreter.get());

  // Read output buffers
  // TODO(user): Insert getting data out code.
  cv::Mat results0(10, 4, CV_8U);
  cv::Mat results1(1, 10, CV_8U);
  cv::Mat results2(1, 10, CV_8U);
  cv::Mat results3(1, 1, CV_8U);

  results0.data = interpreter->typed_output_tensor<uchar>(0);
  results1.data = interpreter->typed_output_tensor<uchar>(1);
  results2.data = interpreter->typed_output_tensor<uchar>(2);
  results3.data = interpreter->typed_output_tensor<uchar>(3);

  std::cout << "results 0: " << results0 << std::endl;
  std::cout << "results 1: " << results1 << std::endl;
  std::cout << "results 2: " << results2 << std::endl;
  std::cout << "results 3: " << results3 << std::endl;

  return 0;
}

The results I see are:

results 0: []               
results 1: []               
results 2: []               
results 3: []  

Answer 1

It seems there's something weird when printing cv::Mat with cout: You explicitly defined the size of the metrics, but empty result is printed.

Could you try to print out the values directly without using cv:Mat? You can do something like this for debugging:

const auto* output = interpreter->typed_output_tensor<unsigned char>(0);
for (int i = 0; i < 40; ++i) {
  printf("%d, ", static_cast<int>(output[i]);
}