TensorFlow Serving for images as base64-encoded strings on Cloud ML Engine

Question

How to implement TensorFlow Serving Input function for images as base64-encoded strings and get prediction on Cloud ML Engine

I am planning to deploy the model on Cloud Machine Learning (ML) Engine after training it on-prem., but I have no idea how to implement the serving input function.

Additionally, I have tried to avoid TensorFlow low-level APIs and only focused on TensorFlow high-level APIs (TensorFlow Estimator). Below here in the blocks of code is the example code that I am working on.

import numpy as np
import tensorflow as tf
import datetime
import os

# create model
from tensorflow.python.keras.applications.vgg16 import VGG16
from tensorflow.python.keras import models
from tensorflow.python.keras import layers

conv_base = VGG16(weights='imagenet',
                  include_top=False,
                  input_shape=(150, 150, 3))

model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dense(256, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
conv_base.trainable = False
model.compile(loss='binary_crossentropy',
              optimizer=tf.keras.optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

dt = datetime.datetime.now()
datetime_now = dt.strftime("%y%m%d_%H%M%S")
model_dir = 'models/imageclassifier_'+datetime_now
model_dir = os.path.join(os.getcwd(), model_dir)
if not os.path.exists(model_dir):
    os.makedirs(model_dir)
print ("model_dir: ",model_dir)

est_imageclassifier = tf.keras.estimator.model_to_estimator(keras_model=model, model_dir=model_dir)

# input layer name
input_name = model.input_names[0]
input_name

This section is for image input function.

def imgs_input_fn(filenames, labels=None, perform_shuffle=False, repeat_count=1, batch_size=1):
    def _parse_function(filename, label):
        image_string = tf.read_file(filename)
        image = tf.image.decode_image(image_string, channels=3)
        image.set_shape([None, None, None])
        image = tf.image.resize_images(image, [150, 150])
        image = tf.subtract(image, 116.779) # Zero-center by mean pixel
        image.set_shape([150, 150, 3])
        image = tf.reverse(image, axis=[2]) # 'RGB'->'BGR'
        d = dict(zip([input_name], [image])), label
        return d
    if labels is None:
        labels = [0]*len(filenames)
    labels=np.array(labels)
    # Expand the shape of "labels" if necessary
    if len(labels.shape) == 1:
        labels = np.expand_dims(labels, axis=1)
    filenames = tf.constant(filenames)
    labels = tf.constant(labels)
    labels = tf.cast(labels, tf.float32)
    dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))
    dataset = dataset.map(_parse_function)
    if perform_shuffle:
        # Randomizes input using a window of 256 elements (read into memory)
        dataset = dataset.shuffle(buffer_size=256)
    dataset = dataset.repeat(repeat_count)  # Repeats dataset this # times
    dataset = dataset.batch(batch_size)  # Batch size to use
    iterator = dataset.make_one_shot_iterator()
    batch_features, batch_labels = iterator.get_next()
    return batch_features, batch_labels

I would like to create a serving input function that

Get images as base64-encoded strings in JSON format
Convert them into Tensors and reduce the size to (?, 150, 150, 3) for prediction

As shown below,

def serving_input_receiver_fn():

''' CODE HERE!'''

return tf.estimator.export.ServingInputReceiver(feature_placeholders, feature_placeholders)

To train and evaluate the model,

train_spec = tf.estimator.TrainSpec(input_fn=lambda: imgs_input_fn(train_files,
                                                                   labels=train_labels,
                                                                   perform_shuffle=True,
                                                                   repeat_count=1,
                                                                   batch_size=20), 
                                    max_steps=500)

exporter = tf.estimator.LatestExporter('Servo', serving_input_receiver_fn)

eval_spec = tf.estimator.EvalSpec(input_fn=lambda: imgs_input_fn(val_files,
                                                                 labels=val_labels,
                                                                 perform_shuffle=False,
                                                                 batch_size=1),
                                 exporters=exporter)

tf.estimator.train_and_evaluate(est_imageclassifier, train_spec, eval_spec)

If I understand it correctly, the example of the input file to get the prediction on Cloud ML Engine should be something like

request.json

{"b64": "9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHJC...”}
{"b64": "9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHJC...”}

And

gcloud ml-engine predict --model MODEL_NAME  \
                --version MODEL_VERSION \
                --json-instances request.json

If you are reading until here and have some idea, could you please suggest me how to implement the serving input function for this particular case.

Many thanks in advance,

2nd Post - To update what I have done so far.

According to sdcbr's comment, below here is my serving_input_receiver_fn().

For _img_string_to_tensor() function or (prepare_image function), I guess that I should do image preparation the same way as I trained the model that you can see

imgs_input_fn() => _parse_function().

def serving_input_receiver_fn():
    def _img_string_to_tensor(image_string):
        image = tf.image.decode_image(image_string, channels=3)
        image.set_shape([None, None, None])
        image = tf.image.resize_images(image, [150, 150])
        image = tf.subtract(image, 116.779) # Zero-center by mean pixel
        image.set_shape([150, 150, 3])
        image = tf.reverse(image, axis=[2]) # 'RGB'->'BGR'
        return image

    input_ph = tf.placeholder(tf.string, shape=[None])

    images_tensor = tf.map_fn(_img_string_to_tensor, input_ph, back_prop=False, dtype=tf.float32)

    return tf.estimator.export.ServingInputReceiver({model.input_names[0]: images_tensor}, {'image_bytes': input_ph})

After I trained the model and deployed the saved model on Cloud ML Engine. My input image was prepared into the format shown below.

{"image_bytes": {"b64": "YQ=="}}

But I found the error after getting a prediction via gcloud.

gcloud ml-engine predict --model model_1  \
               --version v1 \
               --json-instances request.json

{ "error": "Prediction failed: Error during model execution: AbortionError(code=StatusCode.INVALID_ARGUMENT, details="assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP] [[{{node map/while/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert}} = Assert[T=[DT_STRING], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](map/while/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, map/while/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]")" }

Did I do something wrong in _img_string_to_tensor function?

and Could you please clarify me more about this tf.placeholder?

input_ph = tf.placeholder(tf.string, shape=[None])

For your above code, you use shape=[1], but I think it should be shape=[None].

sdcbr · Accepted Answer

Something along these lines should work:

def serving_input_receiver_fn():
    def prepare_image(image_str_tensor):
        image = tf.image.decode_image(image_str_tensor,
                                     channels=3)
        image = tf.image.resize_images(image, [150, 150])
        return image

    # Ensure model is batchable
    # https://stackoverflow.com/questions/52303403/
    input_ph = tf.placeholder(tf.string, shape=[None])
    images_tensor = tf.map_fn(
        prepare_image, input_ph, back_prop=False, dtype=tf.float32)
    return tf.estimator.export.ServingInputReceiver(
        {model.input_names[0]: images_tensor},
        {'image_bytes': input_ph})

You can add additional preprocessing in the prepare_image function. Note that images_tensor should get mapped on the name of the layer in your tf.kerasmodel that is supposed to receive the input.

See also this and this related question.

TensorFlow Serving for images as base64-encoded strings on Cloud ML Engine

How to implement TensorFlow Serving Input function for images as base64-encoded strings and get prediction on Cloud ML Engine

2nd Post - To update what I have done so far.

Answers (2)

Answer!

Related Questions