CODEWITHSUNDEEP
tensorflowtensorflow-serving
ExternalCompilerError
ExternalCompilerError

Reputation: 109

How to serve custom meta data for a (tensorflow) model?

While serving TensorFlow models via TensorFlow-Serving, I need to expose custom meta data to the clients (e.g. a model's input data requirements, training information...).

I tried adding the information via tf.add_to_collection( <my_custom_key>, <value> ) before saving the model, and, sure enough, the information showed up in the .pb(txt) file used by the server.

However, currently it looks as if the response to querying metadata (e.g. via GET http://localhost:8501/v1/models/<my_model>/metadata) only returns the contents of the signature_def section (which also cannot be extended, the validator prevents that), and I know of no way to query contents of other sections.

Is there a way to serve/query custom meta data for TF Serving?

Upvotes: 3

Views: 1176

Answers (2)

ExternalCompilerError
ExternalCompilerError

Reputation: 109

While I still did not find a solution for TensorFlow Serving, it may be of interest to other readers that it can be achieved when using NVidia's Triton Inference Server.

While evaluating it as a TFS alternative (mainly for its built-in support for other model formats such as pytorch and ONNX), I found out that in Triton, it is possible to serve custom meta information via the Model Configuration Extension using the 'parameters' property. After adding

parameters: {
  key: "inference_properties"
  value: {
    string_value: "<my-custom-inference-property-info>"
  }
}

to the model's config.pbtxt file, I could retrieve the information on the client side. It's not extremely convenient, as one can only provide a flat map with string values, but still.

Upvotes: 1

user11530462
user11530462

Reputation:

Unfortunately adding logic to allow serving metadata other than signaturedefs is not on the roadmap right now and I'm not sure we have a good understanding of the general use case for which supporting this would make sense.

Regarding how to serve the metadata stored in the saved model, presumably, you'd add a constant to your graph holding the tensor value of interest (the input/output shape), create a new signature using link below and do inference with that signature -- I've never seen this done but I can't imagine why it wouldn't work.

https://www.tensorflow.org/guide/saved_model#manually_build_a_savedmodel

Upvotes: 1

Related Questions