CODEWITHSUNDEEP
tensorflow-serving
georgesykes86
georgesykes86

Reputation: 31

Tensorflow Serving Model Server continuously re-adding the same models when polling s3 for config file

I am using Tensorflow Serving to run models which are stored in an s3 bucket. I am also keeping the model config file in a separate s3 bucket. My use case is that in order to dynamically add models without needing to restart the server I will poll this config file for changes periodically.

In order to do this I have used the following setup:

  1. tensorflow/serving:1.15.0 image deployed into a Kubernetes cluster using helm.
  2. In the helm chart for the deployment the following lines define the run command and args used for polling s3 for the config
command:
 - "/usr/bin/tensorflow_model_server"
args:
 - --model_config_file={path to config file}
 - --model_config_file_poll_wait_seconds=60
  1. The helm chart also sets environment variables for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, S3_ENPOINT and AWS_LOG_LEVEL=3

The model config file has contents:

model_config_list: {
    config: {
        name: "mlp",
        base_path: "s3://bucketname/mlp",
        model_platform: "tensorflow",
        model_version_policy: {
            specific: {
                versions: 20200130
            }
        }
    }
}

Everything seems to be working as expected with Tensorflow loading the correct models and running them properly. The issue I am seeing is that everytime the server polls for the config file in S3 it is readding models even if they are the same. This results in regular logs as below.

2020-02-06 07:07:01.930476: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2020-02-06 07:07:01.930495: I tensorflow_serving/model_servers/server_core.cc:573]  (Re-)adding model: mlp
2020-02-06 08:07:01.965518: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2020-02-06 08:07:01.965548: I tensorflow_serving/model_servers/server_core.cc:573]  (Re-)adding model: mlp
2020-02-06 09:07:01.967228: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2020-02-06 09:07:01.967259: I tensorflow_serving/model_servers/server_core.cc:573]  (Re-)adding model: mlp

My concern is that this will be impacting performance of the model if the polling frequency is too high. I am wondering though whether there are actually any changes going on or if this is just additional logging.

Upvotes: 3

Views: 650

Answers (1)

Eric McLachlan
Eric McLachlan

Reputation: 3530

This all looks normal.

Looking at the code (server_core.cc), it seems that these messages are displayed when reading the model.config file rather than when loading the models... which would have to happen after reading the model.config file. Although I don't understand the code clearly, I think we can conclude that these messages are not displayed when loading the models themselves; but are displayed at an earlier point in the workflow.

It only tries to figure out which models are new after these messages have been displayed. You can see the comparison happening here.

There is one strange thing about your question. From the timestamps of your messages, it seems that your models.config file is being read every 60 minutes instead of every 60 seconds.

Upvotes: 2

Related Questions