image classify and tensorflow serving

Question

First of all sorry I am not prcise for this question but I am studying the tensorflow-serving and how to put in production my cnn. sincerely the documentation is quite confuse to me. I hope you can help to understand better the save model architecture. So please reply to me as teacher, i would like to know more about the whole flow.

I am developping a simple cnn to classify an image to 4 output. I need tensorflow-serving to put it in production. The image in input can be watherver size, the CNN should resize it first and predict. Here the code

import numpy as np
import tensorflow as tf
from tensorflow import keras
from keras.preprocessing.image import ImageDataGenerator
from matplotlib import pyplot as plt
from scipy.misc import toimage
from keras.models import Sequential
from keras.layers import *
from keras.optimizers import *
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import tag_constants, signature_constants, signature_def_utils_impl
import cv2



#train_path='Garage/train'
#train_datagen = ImageDataGenerator(rescale=1./255)
#train_batch = train_datagen.flow_from_directory(train_path, target_size=(64,64), class_mode='categorical', batch_size=10, color_mode='grayscale')


#validation_datagen = ImageDataGenerator(rescale=1./255)
#validation_batch = validation_datagen.flow_from_directory(
#        './Garage/validation',
#        target_size=(64, 64),
#        batch_size=3,
#        class_mode='categorical', color_mode='grayscale')

model = Sequential()
model.add(InputLayer(input_shape=[64,64,1]))
model.add(Conv2D(filters=32,kernel_size=5,strides=1,padding='same',activation='relu'))
model.add(MaxPool2D(pool_size=5,padding='same'))

model.add(Conv2D(filters=50,kernel_size=5,strides=1,padding='same',activation='relu'))
model.add(MaxPool2D(pool_size=5,padding='same'))

model.add(Conv2D(filters=80,kernel_size=5,strides=1,padding='same',activation='relu'))
model.add(MaxPool2D(pool_size=5,padding='same'))

model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(512,activation='relu'))
model.add(Dropout(rate=0.5))
model.add(Dense(4,activation='softmax'))
optimizer=Adam(lr=1e-3)

model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])
#model.fit_generator(
#        train_batch,
#        epochs=50,
#        steps_per_epoch=6,
#        validation_data=validation_batch,
#        validation_steps=5)

model.load_weights('model.h5')

#score = model.evaluate_generator(validation_batch,steps=3)
#print('Test loss:', score[0])
#print('Test accuracy:', score[1])

#model.save('model.h5')


from PIL import Image
import requests
from io import BytesIO

response = requests.get('http://192.168.3.21:7451/shot.jpg')
image_pil = Image.open(BytesIO(response.content))
image = np.asarray(image_pil)

img2 = cv2.resize(image,(64,64))
img2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)
img = np.reshape(img2,[1,64,64,1])

classes = model.predict_classes(img)

print(classes)

model_version="1"

sess = tf.Session()

#setting values for the sake of saving the model in the proper format
x = model.input
y = model.output

prediction_signature = tf.saved_model.signature_def_utils.predict_signature_def({"inputs":x}, {"prediction":y})

valid_prediction_signature = tf.saved_model.signature_def_utils.is_valid_signature(prediction_signature)
if(valid_prediction_signature == False):
    raise ValueError("Error: Prediction signature not valid!")

builder = saved_model_builder.SavedModelBuilder('./'+model_version)
legacy_init_op = tf.group(tf.tables_initializer(), name='legacy_init_op')

# Add the meta_graph and the variables to the builder
builder.add_meta_graph_and_variables(
      sess, [tag_constants.SERVING],
      signature_def_map={
           signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:prediction_signature, },
      legacy_init_op=legacy_init_op)

# save the graph
builder.save()

the code will take the picture from a cam http://192.168.3.21:7451/shot.jpg and then it will predict it

When I compile the code it return a lot of errors when it try to save the model. can you please check it and tell me if the save model instructions are right?

I use x = model.input as input from the serving but I would like it take the picture as input from the server. I am quite confuse actually, sorry. The scope is when I request by gRPC to predict the image the model can give me the prediction result Thanks

Answer 1

I tried to comment because I don't have a for sure answer for you but I didn't have enough space. Hopefully this info is helpful and can pass for an "answer".

Anyways, it's tough to say what the problems are for a Tensorflow newb like me without seeing any errors.

One thing I noticed is that the method call for predict_signature_def() seems to not follow the method signature I found at here.

Also, I don't think you want to do your image downloading/processing in the same code you have your model in. TFServe isn't supposed to run pre-post processing; Just host your models.

So what you can do is create something like a RESTful service that you accept the image at, then you run your preprocessing on it and send that processed image in as part of the request to TFServe. It looks something like this:

user+image 
    -> requests classification to RESTful service 
    -> REST API receives image
    -> REST service resizes image
    -> REST service makes classification request to TFS (where your model is)
    -> TFS receives request, including resized/preprocessed image
    -> TFS invokes classification using your model and the arguments you sent
    -> TFS responds to REST service with model's response
    -> REST service responds to user with the classification from your model

This sort of sucks because passing images around the network is inefficient and can be slow but you can optimize when you find pain points.

The main idea is that your models should be saved off into an artifact/binary that can run and doesn't need your code to run. This lets you separate out modeling from data pre and post processing and gives your models a more consistent place to run from; e.g. you don't need to worry about competing versions of dependencies for your model to run.

The downside is it can be a bit of a learning curve to get these pieces to fit nicely after breaking them out from the monolithic architecture it seems like you have.

So, until a real Tensorflow knower comes along with a real answer, I hope this helps a bit.