CODEWITHSUNDEEP
amazon-web-servicestensorflowamazon-sagemaker
tlanigan
tlanigan

Reputation: 969

How to make parameters available to SageMaker Tensorflow Endpoint

I'm looking to make some hyper parameters available to the serving endpoint in SageMaker. The training instances is given access to input parameters using hyperparameters in:

estimator = TensorFlow(entry_point='autocat.py',
                       role=role,
                       output_path=params['output_path'],
                       code_location=params['code_location'],
                       train_instance_count=1,
                       train_instance_type='ml.c4.xlarge',
                       training_steps=10000,
                       evaluation_steps=None,
                       hyperparameters=params)

However, when the endpoint is deployed, there is no way to pass in parameters that are used to control the data processing in the input_fn(serialized_input, content_type) function.

What would be the best way to pass parameters to the serving instance?? Is the source_dir parameter defined in the sagemaker.tensorflow.TensorFlow class copied to the serving instance? If so, I could use a config.yml or similar.

Upvotes: 2

Views: 2237

Answers (3)

Dan
Dan

Reputation: 794

Ah i have had a similar problem to you where I needed to download something off S3 to use in the input_fn for inference. In my case it was a dictionary.

Three options:

  1. use your config.yml approach, and download and import the s3 file from within your entrypoint file before any function declarations. This would make it available to the input_fn
  2. Keep using the hyperparameter approach, download and import the vectorizer in serving_input_fn and make it available via a global variable so that input_fn has access to it.
  3. Download the file from s3 before training and include it in the source_dir directly.

Option 3 would only work if you didnt need to make changes to the vectorizer seperately after initial training.

Whatever you do, don't download the file directly in input_fn. I made that mistake and the performance is terrible as each invoking of the endpoint would result in the s3 file being downloaded.

Upvotes: 1

Marcio dos Santos
Marcio dos Santos

Reputation: 875

Yes, one option is to add your configuration file to source_dir and load the file in the input_fn.

Another option is to use serving_input_fn(hyperparameters). That function transforms the TensorFlow model in a TensorFlow serving model. For example:

def serving_input_fn(hyperparameters):

    # gets the input shape from the hyperparameters
    shape = hyperparameters.get('input_shape', [1, 7])

    tensor = tf.placeholder(tf.float32, shape=shape)
    # returns the ServingInputReceiver object.

    return build_raw_serving_input_receiver_fn({INPUT_TENSOR_NAME: tensor})()

Upvotes: 0

Guy
Guy

Reputation: 12891

The Hyper-parameters are used in the training phase to allow you to tune (Hyper-Parameters Optimization - HPO) your model. Once you have a trained model, these hyper-parameters are not needed for inference.

When you want to pass features to the serving instances you usually do that in the BODY of each request to the invoke-endpoint API call (for example see here: https://docs.aws.amazon.com/sagemaker/latest/dg/tf-example1-invoke.html) or the call to the predict wrapper in the SageMaker python SDK (https://github.com/aws/sagemaker-python-sdk/tree/master/src/sagemaker/tensorflow). You can see such examples in the sample notebooks (https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/tensorflow_iris_byom/tensorflow_BYOM_iris.ipynb)

Upvotes: 0

Related Questions