CODEWITHSUNDEEP
machine-learningtensorflowraspberry-picomputer-visionkeras
megashigger
megashigger

Reputation: 9053

Speeding up inference of Keras models

I have a Keras model which is doing inference on a Raspberry Pi (with a camera). The Raspberry Pi has a really slow CPU (1.2.GHz) and no CUDA GPU so the model.predict() stage is taking a long time (~20 seconds). I'm looking for ways to reduce that by as much as possible. I've tried:

Is there anything else I can do to increase the speed during inference? Is there a way to simplify a model.h5 and take a drop in accuracy? I've had success with simpler models, but for this project I need to rely on an existing model so I can't train from scratch.

Upvotes: 7

Views: 5467

Answers (2)

dragon7
dragon7

Reputation: 1133

Maybe OpenVINO will help. OpenVINO is an open-source toolkit for network inference, and it optimizes the inference performance by, e.g., graph pruning and fusing some operations. The ARM support is provided by the contrib repository.

Here are the instructions on how to build an ARM plugin to run OpenVINO on Raspberry Pi.

Disclaimer: I work on OpenVINO.

Upvotes: 0

Fábio Perez
Fábio Perez

Reputation: 26098

VGG16 / VGG19 architecture is very slow since it has lots of parameters. Check this answer.

Before any other optimization, try to use a simpler network architecture.

Google's MobileNet seems like a good candidate since it's implemented on Keras and it was designed for more limited devices.

If you can't use a different network, you may compress the network with pruning. This blog post specifically do pruning with Keras.

Upvotes: 2

Related Questions