CODEWITHSUNDEEP
androidc++opencvneural-network
Marcell Molnár
Marcell Molnár

Reputation: 53

Grayscale image classification with neural network on Android with OpenCV

I'm working on and android app that can digitize numbers from a paper. I use native OpenCV code to find the numbers on the image. After that I want to use OpenCV's dnn module to recognise the number. A nice tutorial on creating the neural net can be found here:
https://www.youtube.com/watch?v=kFWKdLOxykE
The mnist_convnet_graph.pbtxt begins with this:

node {
  name: "conv2d_1_input"
  op: "Placeholder"
  attr {
    key: "dtype"
    value {
    type: DT_FLOAT
  }
  }
  attr {
    key: "shape"
    value {
      shape {
        dim {
          size: -1
        }
        dim {
          size: 28
        }
        dim {
          size: 28
        }
        dim {
          size: 1
        }
      }
    }
  }
}

So the input is a 28x28 grayscale image.
In the tutorial java code is used to use the neural net. However, I would like to use it in C++, because of the speed. I successfully load the model with cv::dnn::Net Dnn.readNetFromTensorflow(String model, String config); and pass the object to the NDK side. I create the input for the neural net with the following:

// The part of the image, we are interested in.
Rect roi(static_cast<int>(w), static_cast<int>(h),
             static_cast<int>(w), static_cast<int>(h));
Mat cropped(image_gray, roi);
// Resize image to 28x28.
Mat resized;
cv::resize(cropped, resized, Size(28,28));

After that, the forwarding should work:

const double IN_SCALE_FACTOR = 0.003921; // 1.0/255.0
Mat blob = dnn::blobFromImage(resized, IN_SCALE_FACTOR, Size(28,28));
net.setInput(blob);
Mat detections = net.forward();

where net is the passed cv::dnn::Net object. But the net.forward() command fails and gives:
OpenCV(3.4.5) Error: Assertion failed (inputs.size() == requiredOutputs) in virtual bool cv::dnn::experimental_dnn_34_v11::DataLayer::getMemoryShapes(const std::vector >&, int, std::vector >&, std::vector >&) const, file /build/3_4_pack-android/opencv/modules/dnn/src/dnn.cpp, line 681

I also tried:

but none of these led to the solutions. Anyone has a solutions for this? Any suggestion or idea will be appreciated.

Upvotes: 2

Views: 958

Answers (1)

Marcell Moln&#225;r
Marcell Moln&#225;r

Reputation: 53

Okay, I managed to solve my problem.

Firstly, I realized that the .pb and .pbtxt files are in the wrong directory and getting 2 Failed to upload a file information log.

After putting the files into the right directory, I faced with problem:
error: (-215:Assertion failed) const_layers.insert(std::make_pair(name, li)).second in function 'void cv::dnn::experimental_dnn_34_v11::{anonymous}::addConstNodes(opencv_tensorflow::GraphDef&, std::map<cv::String, int>&, std::set<cv::String>&)'

As Dmitry Kurtaev suggested here, I removed the .pbtxt from the Dnn.readNetFromTensorflow. After that I got error:
OpenCV(3.4.5) Error: Unspecified error (Can't create layer "flatten_1/Shape" of type "Shape") in cv::Ptr<cv::dnn::experimental_dnn_34_v11::Layer> cv::dnn::experimental_dnn_34_v11::LayerData::getLayerInstance(), file /build/3_4_pack-android/opencv/modules/dnn/src/dnn.cpp, line 513

This led me to a link, which I found in Dmitry Kurtaev's comment here. After doing the suggested modifications (removing Const nodes, modifying and removing flatten nodes) on the .pbtxt file, finally I got no errors and got successfully running the neural net.

Note: Adding K.backend.set_learning_phase(0) before creating the model, also can be useful.

Upvotes: 1

Related Questions