CODEWITHSUNDEEP
pythontensorflowtensorflow-serving
Eli
Eli

Reputation: 81

Tensorflow Serving Response

I'm trying to establish communication to a Tensorflow Serving session. The following code works, but it is a bit slow. Is there a way to improve it? I suspect the problem lay in the fourth line - the output is giving as a list of float_val elements, and I need to convert them to a float array and reshape them.

Is there a way to get the server output in the right shape? I have defined the output signature as below (which I think is correct).

prediction_channel, request_form = setup_channel(args.server)    
request_form.inputs['images'].CopyFrom(
                tf.contrib.util.make_tensor_proto(img_transformed, shape=list(img_transformed.shape)))
output = prediction_channel.Predict.future(request_form, 5.0)
output = np.array(output.result().outputs['scores'].float_val).reshape(1, 16, 64, 64)

The first line opens the channel to the server by using the function

def setup_channel(hostport):
   host, port = hostport.split(':')
   channel = implementations.insecure_channel(host, int(port))
    stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
   request = predict_pb2.PredictRequest()
   request.model_spec.name = 'hg'
   request.model_spec.signature_name = 'predict_images'
   return stub, request

The output signature is:

tensor_info_x = tf.saved_model.utils.build_tensor_info(model.input_tensor)
tensor_info_y = tf.saved_model.utils.build_tensor_info(model.predict)

prediction_signature = (
    tf.saved_model.signature_def_utils.build_signature_def(
        inputs={'images': tensor_info_x},
        outputs={'scores': tensor_info_y},
        method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))**

and model predict has a shape of (1, 16, 64, 64).

Upvotes: 1

Views: 1007

Answers (1)

T Humbug
T Humbug

Reputation: 21

I am not sure how you are handling your Predict.future(request_form, 5.0), but the same should apply for the sync response handling; tf offers a utility function make_ndarray:

res = stub.Predict(request, timeout).outputs[tensor_name]
arr = tf.make_ndarray(res)

arr will be an np array of the correct dims.

Where tensor_name is the name defined in your signature e.g. 

tf.saved_model.signature_def_utils.build_signature_def(
    inputs={'images': inp_tensor_info},
    outputs={'scores': out_tensor_info},
    method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME
)

would require 

res = stub.Predict(request, timeout).outputs['scores']
arr = tf.make_ndarray(res)

Upvotes: 2

Related Questions