TensorFlow serving failed to serve TensorFlow 2.0 keras.layers.LSTM

Question

Why TensorFlow serving failed to serve this simple LSTM keras layer, while it runs successfully using saved_model_cli run? And How can I fix it?

TensorFlow version 2.0 alpha

$  pip install tensorflow==2.0.0-alpha0 

The SavedModel reproducing:

import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, LSTM

img_width= 10
img_height= 10
def build_model():
    input_img = Input(shape=(img_width*img_height, 1), name='input_data', dtype='float32')
    x = LSTM(12, return_sequences=False,  name='lstm_1')(input_img)
    model = Model(input_img, x)
    return model

def save():
    model = build_model()
    tf.saved_model.save(model, "./test_keras_serving/1/")

if __name__ == '__main__':
    save()

TensorFlow Serving install:

$ docker pull tensorflow/serving

$ docker images -a
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
tensorflow/serving         latest              38bee21b2ca0        2 months ago        229MB

Serving the SavedModel

$ docker run -p 8501:8501 --mount type=bind,source=/the/path/of/dir/test_keras_serving,target=/models/my_model -e MODEL_NAME=my_model -t tensorflow/serving

Prediction python code using TensorFlow Serving.

import json
import requests
import numpy as np

def pred():
    inp_value = np.zeros((2,100,1))
    _url = 'http://localhost:8501/v1/models/my_model:predict'
    headers = {"cache-control": "no-cache", "content-type": "application/json"}
    data = json.dumps({"signature_name": "serving_default","instances": inp_value.tolist()})

    json_response = requests.post(url=_url, data=data, headers=headers)
    print(json_response)

if __name__ == '__main__':
    pred()

In the client, the results are

<Response [400]>

Instead of [200].

And the server shows:

2019-05-12 13:21:49.370594: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at partitioned_function_ops.cc:118 : Invalid argument: Expected input[1] == 'TensorArrayV2Stack/TensorListStack/element_shape:output:0' to be a control input.
        In {{node TensorArrayV2Stack/TensorListStack}}

However, the SavedModel is working fine with saved_model_cli:

$ saved_model_cli run --dir ./test_keras_serving/1 --tag_set serve --signature_def serving_default --input_exprs input_data=np.zeros((2,100,1))

Outputs:

Result for output key lstm_1:
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

How can I get the same results as saved_model_cli using TensorFlow Serving?

Answer 1

Solved this issue with docker pull tensorflow/serving:nightly