CODEWITHSUNDEEP
tensorflowtensorflow-servingtfx
Andras Gyacsok
Andras Gyacsok

Reputation: 61

TensorFlow Serving Cluster Architecture

Folks, I am writing an application which will produce recommendations based on ML model call. The application will have different models, some of them should be called in sequence. A data scientist should be able, to upload a model in the system. This means that the application should have logic to store models metadata as well as address of a model server. A model server will be instantiated dynamically on a model upload event. I would like to use a cluster of TensorFlow Serving here, however I am stacked with a question of architecture. Is there a way to have something like service registry for TensorFlow servers? What is the best way to build such a cluster of servers with different models?

Upvotes: 2

Views: 180

Answers (1)

RCrowe
RCrowe

Reputation: 363

I need some clarification on what you're trying to do. Is the feature vector for all the models the same? If not then it will be quite a bit harder to do this. Trained models are encapsulated in the SavedModel format. It sounds like you're trying to train an ensemble, but some of the models are frozen? You could certainly write a custom component to make an inference request as part of the input to Trainer, if that's what you need.

UPDATE 1 From your comment below it sounds like what you might be looking for is a service mesh, such as Istio for example. That would help manage the connections between services running inside containers, and the connections between users and services. In this case tf.Serving instances running your models are the services, but the basic request-response pattern is the same. Does that help?

Upvotes: 1

Related Questions