Vertex AI - Endpoint Call with JSON - Invalid JSON payload received

Question

I successfully trained and deployed a Tensorflow Recommender model on Vertex AI.

Everything is online and to predict the output. In the notebook I do:

loaded = tf.saved_model.load(path)
scores, titles = loaded(["doctor"])

That returns:

Recommendations: [b'Nelly & Monsieur Arnaud (1995)'
 b'Three Lives and Only One Death (1996)' b'Critical Care (1997)']

That is, the payload (input for the neural network) must be ["doctor"]

Then I generate the JSON for payload (the error is here):

!echo {"\""instances"\"" : [{"\""input_1"\"" : {["\""doctor"\""]}}]} > instances0.json

And submit to the endpoint:

!curl -X POST  \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-west1-aiplatform.googleapis.com/v1/projects/my_project/locations/us-west1/endpoints/123456789:predict \
-d @instances0.json > results.json

... as seen here: https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/vertex_endpoints/tf_hub_obj_detection/deploy_tfhub_object_detection_on_vertex_endpoints.ipynb#scrollTo=35348dd21acd

However, when I use this payload, I get error 400:

code: 400
message: "Invalid JSON payload received. Expected an object key or }. s" : [{"input_1" : {["doctor"]}}]} ^"
status: "INVALID_ARGUMENT"

This below don't work either:

!echo {"inputs": {"input_1": ["doctor"]}} > instances0.json

Even with validated JSON Lint, it does not return the proper prediction.

In another Stackoverflow question is suggested to remove the " \ " in the payload, but this didn't work either.

Running:

!saved_model_cli show --dir /home/jupyter/model --all

I get:

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is: 

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_1'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: serving_default_input_1:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output_1'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 10)
        name: StatefulPartitionedCall_1:0
    outputs['output_2'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 10)
        name: StatefulPartitionedCall_1:1
  Method name is: tensorflow/serving/predict


Concrete Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
        Argument #2
          DType: NoneType
          Value: None
        Argument #3
          DType: bool
          Value: True
    Option #2
      Callable with:
        Argument #1
          queries: TensorSpec(shape=(None,), dtype=tf.string, name='queries')
        Argument #2
          DType: NoneType
          Value: None
        Argument #3
          DType: bool
          Value: True
    Option #3
      Callable with:
        Argument #1
          input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
        Argument #2
          DType: NoneType
          Value: None
        Argument #3
          DType: bool
          Value: False
    Option #4
      Callable with:
        Argument #1
          queries: TensorSpec(shape=(None,), dtype=tf.string, name='queries')
        Argument #2
          DType: NoneType
          Value: None
        Argument #3
          DType: bool
          Value: False

  Function Name: '_default_save_signature'
    Option #1
      Callable with:
        Argument #1
          input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')

  Function Name: 'call_and_return_all_conditional_losses'
    Option #1
      Callable with:
        Argument #1
          input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
        Argument #2
          DType: NoneType
          Value: None
        Argument #3
          DType: bool
          Value: False
    Option #2
      Callable with:
        Argument #1
          queries: TensorSpec(shape=(None,), dtype=tf.string, name='queries')
        Argument #2
          DType: NoneType
          Value: None
        Argument #3
          DType: bool
          Value: True
    Option #3
      Callable with:
        Argument #1
          queries: TensorSpec(shape=(None,), dtype=tf.string, name='queries')
        Argument #2
          DType: NoneType
          Value: None
        Argument #3
          DType: bool
          Value: False
    Option #4
      Callable with:
        Argument #1
          input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
        Argument #2
          DType: NoneType
          Value: None
        Argument #3
          DType: bool
          Value: True

The point is: I'm passing an array and I'm not sure if it must be in b64 format.

This Python code works, but returns a different result than expected:

import tensorflow as tf
import base64
from google.protobuf import json_format
from google.protobuf.struct_pb2 import Value
import numpy as np
from google.cloud import aiplatform
import os
vertex_model = tf.saved_model.load("gs://bucket/model")

serving_input = list(
    vertex_model.signatures["serving_default"].structured_input_signature[1].keys()
)[0]

print("Serving input :", serving_input)

aip_endpoint_name = (
    f"projects/my-project/locations/us-west1/endpoints/12345567"
)
endpoint = aiplatform.Endpoint(aip_endpoint_name)

def encode_input(input):
    return base64.b64encode(np.array(input)).decode("utf-8")

instances_list = [{serving_input: {"b64": encode_input(np.array(["doctor"]))}}]
instances = [json_format.ParseDict(s, Value()) for s in instances_list]

results = endpoint.predict(instances=instances)
print(results.predictions[0]["output_2"])


['8 1/2 (1963)', 'Sword in the Stone, The (1963)', 'Much Ado About Nothing (1993)', 'Jumanji (1995)', 'As Good As It Gets (1997)', 'Age of Innocence, The (1993)', 'Double vie de Véronique, La (Double Life of Veronique, The) (1991)', 'Piano, The (1993)', 'Eat Drink Man Woman (1994)', 'Bullets Over Broadway (1994)']

Any ideas on how to fix / encode the payload ?

Answer 1

I solved the issue. The problem was base64 encoding. This is working perfectly:

def encode_64(input):
    message = input
    message_bytes = message.encode('ascii')
    base64_bytes = base64.b64encode(message_bytes)
    base64_message = base64_bytes.decode('ascii')
    return base64_message


instances_list = [{serving_input: {"b64": encode_64("doctor")}}]
instances = [json_format.ParseDict(s, Value()) for s in instances_list]

results = endpoint.predict(instances=instances)
print(results.predictions[0]["output_2"][:3])

['Nelly & Monsieur Arnaud (1995)', 'Three Lives and Only One Death (1996)', 'Critical Care (1997)']