CODEWITHSUNDEEP
pythontensorflow
Leyens
Leyens

Reputation: 81

Using a saved model for inference multiple times without reloading the model each time

My goal is to load a saved model once and use it for inference multiple times on different images to save time between each prediction. In my case, after loading the model, the first prediction is fine. However, if I try to use the model a second time, the result is empty. Is there a way to use the loaded model for inference multiple times or am I doing something terribly wrong?

I am using a trained YoloV4-tiny model that has been converted from a .wheights file to a .pb file using this repository. The tensorflow version I am using is tf-nightly 2.5.0.

Code sample:

saved_model_loaded = tf.saved_model.load(model_weights, tags=[tag_constants.SERVING])
infer = saved_model_loaded.signatures['serving_default']
batch_data = tf.constant(image_data)

prediction_bbox = infer(batch_data)
print('First prediction: ')
print(prediction_bbox)
prediction_bbox = infer(batch_data)
print('Second prediction: ')
print(prediction_bbox)

Output:

First prediction:
{'tf.concat_12': <tf.Tensor: shape=(1, 14, 56), dtype=float32, numpy=
array([[[3.04199923e-02, 4.36968982e-01, 1.12345181e-01, 4.82889146e-01,
     7.35657290e-04, 7.31759297e-04, 8.13401653e-04, 7.43044075e-06,
     1.19315519e-03, 1.88542936e-06, 2.82156630e-03, 1.12371512e-04,
     [...]
     2.79035070e-04, 6.39518723e-04, 1.51291722e-04, 7.65962759e-04]]],
  dtype=float32)>}
Second prediction: 
{'tf.concat_12': <tf.Tensor: shape=(1, 0, 56), dtype=float32, numpy=array([], shape=(1, 0, 56),dtype=float32)>}

Upvotes: 2

Views: 1225

Answers (1)

Leyens
Leyens

Reputation: 81

I managed to solve this problem by loading the model with keras:

saved_model_loaded = tf.keras.models.load_model(model_weights)
batch_data = tf.constant(image_data)
model.predict(batch_data)

With keras, the model can be used for prediction multiple times after loading it only once.

Upvotes: 2

Related Questions