gcloud problems with local prediction

Question

I am using gcloud local prediction to test my exported model. The model is a TensorFlow object detection model which has been trained on custom dataset. I am using the following gcloud command:

gcloud ml-engine local predict --model-dir=/path/to/saved_model/ --json-instances=input.json --signature-name="serving_default" --verbosity debug 

When I am not using verbose the command does not output anything. With verbose set as debug I get the following traceback:

DEBUG: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 984, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 784, in Run
    resources = command_instance.Run(args)
  File "/google-cloud-sdk/lib/surface/ai_platform/local/predict.py", line 83, in Run
    signature_name=args.signature_name)
  File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/local_utils.py", line 103, in RunPredict
    proc.stdin.write((json.dumps(instance) + '\n').encode('utf-8'))
IOError: [Errno 32] Broken pipe 

Details on my exported model :

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['inputs'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: encoded_image_string_tensor:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300, 4)
        name: detection_boxes:0
    outputs['detection_classes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300)
        name: detection_classes:0
    outputs['detection_features'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1, -1, -1, -1)
        name: detection_features:0
    outputs['detection_multiclass_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300, 2)
        name: detection_multiclass_scores:0
    outputs['detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300)
        name: detection_scores:0
    outputs['num_detections'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: num_detections:0
    outputs['raw_detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300, 4)
        name: raw_detection_boxes:0
    outputs['raw_detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300, 2)
        name: raw_detection_scores:0
  Method name is: tensorflow/serving/predict

I used the following code to generate my input.json for prediction :

with open('input.json', 'wb') as f:
    img = Image.open("image.jpg")
    img = img.resize((width, height), Image.ANTIALIAS)
    output_str = io.BytesIO()
    img.save(output_str, "JPEG")
    image_byte_array = output_str.getvalue()
    image_base64 = base64.b64encode(image_byte_array)
    json_entry = {"b64": image_base64.decode()}
    #instances.append(json_entry
    request = json.dumps({'inputs': json_entry})
    f.write(request.encode('utf-8'))
f.close()

{"inputs": {"b64": "/9j/4AAQSkZJRgABAQAAAQABAAD/......}}

I am testing the prediction with one image.

Answer 1

Unlike @Roman Kovtuh, I was able to run using python3. However, his technique for creating an exception handler allowed me to determine that tensorflow was not installed in the environment that was visible to the process. Once done, the process worked.

My changes to googlecloudsdk/command_lib/ml_engine/local_utils.py:

106,109c106
<     try:
<       proc.stdin.write((json.dumps(instance) + '\n').encode('utf-8'))
<     except Exception as e:
<       print(f'Error displaying errors with instance {str(instance)[:100]}.  Exception {e}')
---
>     proc.stdin.write((json.dumps(instance) + '\n').encode('utf-8'))

I upvoted @Roman Kovtuh because this really helped.

Answer 2

I've met the same problem and found out that ml_engine/local_utils.py uses python to run ml_engine/local_predict.pyc that is built for python2.7. My python is python3, so when ml_engine/local_utils.py tries to run ml_engine/local_predict.pyc using python (actually python3), it fails with error:

RuntimeError: Bad magic number in .pyc file

Solution 1:

You can just make python2 as default one in system.

Solution 2:

I changed ml_engine/local_utils.py with such patch:

83c83
<   python_executables = files.SearchForExecutableOnPath("python")
---
>   python_executables = files.SearchForExecutableOnPath("python2")
114a115
>   log.debug(args)
124,126c125,130
<   for instance in instances:
<     proc.stdin.write((json.dumps(instance) + "\n").encode("utf-8"))
<   proc.stdin.flush()
---
>   try:
>     for instance in instances:
>       proc.stdin.write((json.dumps(instance) + "\n").encode("utf-8"))
>     proc.stdin.flush()
>   except:
>     pass

try-catch needed to make script able to read and print an error ocurred while running ml_engine/local_predict.pyc.

Answer 3

When running the command, the local SDK file /usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/local_utils.py, failure seems to be happening when reading file contents:

  for instance in instances:
    proc.stdin.write((json.dumps(instance) + '\n').encode('utf-8'))
  proc.stdin.flush()

In your case, I expect to see JSON correctly formatted, otherwise we normally get:

ERROR: (gcloud.ai-platform.local.predict) Input instances are not in JSON format. See "gcloud ml-engine predict --help" for details.

This is a snippet of the code I normally use to generate my b64 encoded image with resize.

import base64
from PIL import Image

INPUT_FILE = 'image.jpg'
OUTPUT_FILE = 'image_b64.json'


def convert_to_base64_resize(image_file):
  """Open image, resize, base64 encode it and create a JSON request"""
  img = Image.open(image_file).resize((240, 240))
  img.save(image_file)  
  with open(image_file, 'rb') as f:
    jpeg_bytes = base64.b64encode(f.read()).decode('utf-8')   
    predict_request = '{"image_bytes": {"b64": "%s"}}' % jpeg_bytes 
    # Write JSON to file
    with open(OUTPUT_FILE, 'w') as f:
      f.write(predict_request)
    return predict_request

convert_to_base64_resize(INPUT_FILE)

Will be great to see a copy of your JSON file or image and compare contents.

For normal troubleshooting, I also use tensorflow serving, specially in order to validate that my model works locally. (TensorFlow serving supports pointing to the GCS location) Remember that local prediction with json instances expect this format:

{"image_bytes": {"b64": body }}

I assume your model looks like this after the changes suggested above:

...
signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['image_bytes'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: input_tensor:0
...

Answer 4

According to this page, the binary input must be suffixed by _bytes

In your TensorFlow model code, you must name the aliases for your binary input and output tensors so that they end with '_bytes'.

Try to suffix your input with _bytes, or rebuild your model with a compliant input_serving function.