CODEWITHSUNDEEP
pythontensorflowtensorflow-serving
josephkibe
josephkibe

Reputation: 1343

How do I get the metadata from a served TensorFlow model?

I have a few models running with TensorFlow serving, and I'd like the models to be self-describing, so that our services that call them don't need to directly embed things like the input size. (E.g., we want to be able to swap models out without updating downstream systems.)

When I use the metadata API, I get back a response that has a signature definition, but it's clearly not something that's interpretable. The model works, so I'm pretty sure it's not a problem with the model itself. I've also tried the saved_model_cli tool on the TensorFlow model, and it returns a sane signature definition.

Edit: The metadata API also works just fine if I call the REST API. But I'd really prefer to figure out how to get this working with gRPC.

Edit, 21 Oct 2019: I realized that the data in the response is a Protocol Buffer encoded message. If I understand things correctly this means I need to parse using the Protocol Buffer data. I've tried

from google.protobuf import text_format
text_format.Parse(result.metadata['signature_def'].value,
                  get_model_metadata_pb2.SignatureDefMap)

This, however now returns the error:

ParseError: 2:1 : '>': Expected identifier or number, got >.

Older code:

$ saved_model_cli show --dir mymodel.pb/ --all
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_data'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1, 286339)
        name: input:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output/Softmax:0'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 286338)
        name: output/Softmax:0
  Method name is: tensorflow/serving/predict

And:

from tensorflow_serving.apis import get_model_metadata_pb2
import grpc
import tensorflow as tf
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc
from tensorflow.contrib import util

MODEL_URI = '10.100.55.9:8500'
channel = grpc.insecure_channel(MODEL_URI)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

request = get_model_metadata_pb2.GetModelMetadataRequest()
request.model_spec.name = 'themodel'
request.model_spec.signature_name = 'serving_default'
request.metadata_field.append('signature_def')
result = stub.GetModelMetadata(request, 1.0)

which returns

In [5]: result.metadata['signature_def']                                                                                                                                             
Out[5]: 
type_url: "type.googleapis.com/tensorflow.serving.SignatureDefMap"
value: "\n>\n\025__saved_model_init_op\022%\022#\n\025__saved_model_init_op\022\n\n\004NoOp\032\002\030\001\n\271\001\n\017serving_default\022\245\001\nJ\n\007input_1\022?\n\031serving_default_input_1:0\020\001\032 \022\013\010\377\377\377\377\377\377\377\377\377\001\022\013\010\377\377\377\377\377\377\377\377\377\001\022\004\010\224\261\021\022;\n\005dense\0222\n\031StatefulPartitionedCall:0\020\001\032\023\022\013\010\377\377\377\377\377\377\377\377\377\001\022\004\010\223\261\021\032\032tensorflow/serving/predict"

Upvotes: 1

Views: 1787

Answers (1)

josephkibe
josephkibe

Reputation: 1343

I eventually figured this out myself. There's probably a more elegant way to do this that only requires parsing out the bit of the response that I want, and that doesn't require converting to JSON and then to a Python object to get out the part that I want. But this worked, so I'm going to post it.

By reverse engineering the REST API code, I figured out that there's a nice utility function in the Protocol Buffer library that converts from the Protocol Buffer format into JSON, which can then be parsed by a Python JSON parser of your choice.

from google.protobuf.json_format import MessageToJson
import simplejson as json

parsed = json.loads(MessageToJson(result))

Upvotes: 3

Related Questions