CODEWITHSUNDEEP
tensorflow-serving
Sam Mefford
Sam Mefford

Reputation: 2475

Is Tensorflow Serving example expected to have 10% error rate?

I followed the docs Building an optimized serving binary then Testing the development environment, and I got

Inference error rate: 10.4%

Is it expected that a fresh install of a release build of TensorFlow Serving would give a 10% error rate on the provided example model?

My environment:

AWS EC2
OS: Amazon Linux AMI release 2018.03
Instance Type: r5.large

Steps to reproduce:

# download tensorflow serving code
git clone https://github.com/tensorflow/serving
cd serving
# build optimized serving binary
docker build --pull -t $USER/tensorflow-serving-devel   -f tensorflow_serving/tools/docker/Dockerfile.devel .
# run & open shell for generated docker image
docker run -it -p 8600:8600 ec2-user/tensorflow-serving-devel:latest
# train the mnist model
python tensorflow_serving/example/mnist_saved_model.py /tmp/mnist_model
# serve the model
tensorflow_model_server --port=8500 --model_name=mnist --model_base_path=/tmp/mnist_model/ &
# test the client
python tensorflow_serving/example/mnist_client.py --num_tests=1000 --server=localhost:8500

Upvotes: 1

Views: 196

Answers (2)

user11530462
user11530462

Reputation:

Is Tensorflow Serving an example expected to have a 10% error rate?

Yes, this particular example is expected to have a 10% error rate as the accuracy of this model on training and testing data is almost the same (around 90%) i.e., this is a very basic neural network as shown here.

If you want a good prediction accuracy, you might have to use resnet_client.py or you can actually add more layers and tune the hyper-parameters to get higher prediction accuracy or lesser inference error rate.

A tutorial on how to use the resent model to serve is given here. This should give you a much lesser inference error rate.

Upvotes: 2

William D. Irons
William D. Irons

Reputation: 2384

The example mnist_saved_model.py as part of tensorflow_serving's example is more focused on speed to create a model and a simple example on how to save a model then accuracy.

In https://www.tensorflow.org/tfx/serving/serving_advanced, it shows that when the above code is trained with 100 iterations it has a 13.1% error rate, and when trained with 2000 iterations it has a 9.5% error rate.

The default if --training_iteration is not specified is 1000, so your 10.4 error rate is inline with these results.

You will find this mnist model provides better accuracy (and takes much longer to train): https://github.com/tensorflow/models/tree/master/official/mnist

This model will work with the slight changes to the mnist_client.py example.

try this:

train the mnist model

git clone https://github.com/tensorflow/models
export PYTHONPATH="$PYTHONPATH:$PWD/models"
pushd models/official/mnist/
python mnist.py --export_dir /tmp/mnist_model

serve the model

tensorflow_model_server --port=8500 --model_name=mnist --model_base_path=/tmp/mnist_model/ &

switch back the original directory

popd

make the follow changes to mnist_client.py to work with the new model

diff --git a/tensorflow_serving/example/mnist_client.py b/tensorflow_serving/example/mnist_client.py
index 592969d..85ef0bf 100644
--- a/tensorflow_serving/example/mnist_client.py
+++ b/tensorflow_serving/example/mnist_client.py
@@ -112,8 +112,8 @@ def _create_rpc_callback(label, result_counter):
       sys.stdout.write('.')
       sys.stdout.flush()
       response = numpy.array(
-          result_future.result().outputs['scores'].float_val)
-      prediction = numpy.argmax(response)
+          result_future.result().outputs['classes'].int64_val)
+      prediction = response[0]
       if label != prediction:
         result_counter.inc_error()
     result_counter.inc_done()
@@ -143,9 +143,9 @@ def do_inference(hostport, work_dir, concurrency, num_tests):
   for _ in range(num_tests):
     request = predict_pb2.PredictRequest()
     request.model_spec.name = 'mnist'
-    request.model_spec.signature_name = 'predict_images'
+    request.model_spec.signature_name = 'classify'
     image, label = test_data_set.next_batch(1)
-    request.inputs['images'].CopyFrom(
+    request.inputs['image'].CopyFrom(
         tf.contrib.util.make_tensor_proto(image[0], shape=[1, image[0].size]))
     result_counter.throttle()
     result_future = stub.Predict.future(request, 5.0)  # 5 seconds

test the client

python tensorflow_serving/example/mnist_client.py --num_tests=1000 --server=localhost:8500

Inference error rate: 0.8%

Upvotes: 2

Related Questions