CODEWITHSUNDEEP
pythontensorflowmachine-learningkerasdeep-learning
Simone Starace
Simone Starace

Reputation: 166

Tensorflow 2 getting "WARNING:tensorflow:x out of the last x calls to <function> triggered tf.function retracing."

I'm working on a project where I have trained a series of binary classifiers with Keras, with Tensorflow as the backend engine. The input data I have is a series of images, where each binary classifier must make the prediction on the images, later I save the predictions on a CSV file.

The problem I have is when I get the predictions from the first series of binary classifiers there isn't any warning, but when the 5th or 6th binary classifier calls the method predict on the input data I get the following warning:

WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function..predict_function at 0x2b280ff5c158> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.

To answer each point in the parenthesis, here are my answers:

  1. The predict method is called inside a for loop.
  2. I don't pass tensors but a list of NumPy arrays of gray scale images, all of them with the same size in width and height. The only thing that can change is the batch size because the list can have only 1 image or more than one.
  3. As I wrote in point 2, I pass a list of NumPy arrays.

I have debugged my program and found that this warning always happens when the method predict is called. To summarize the code I have written is the following:

import cv2 as cv
import tensorflow as tf
from tensorflow.keras.models import load_model
# Load the models
binary_classifiers = [load_model(path) for path in path2models]
# Get the images
images = [#Load the images with OpenCV]
# Apply the resizing and reshapes on the images.
my_list = list()
for image in images:
    image_reworked = # Apply the resizing and reshaping on images
    my_list.append(image_reworked)

# Get the prediction from each model
# This is where I get the warning
predictions = [model.predict(x=my_list,verbose=0) for model in binary_classifiers]

What I have tried

I have defined a function as tf.function and putted the code of the predictions inside the tf.function like this

@tf.function
def testing(models, faces):
    return [model.predict(x=faces,verbose=0) for model in models]

But I ended up getting the following error:

RuntimeError: Detected a call to Model.predict inside a tf.function. Model.predict is a high-level endpoint that manages its own tf.function. Please move the call to Model.predict outside of all enclosing tf.functions. Note that you can call a Model directly on Tensors inside a tf.function like: model(x).

So calling the method predict is basically already a tf.function. So it's useless to define a tf.function when the warning I get it's from that method.

I have also checked those other two questions:

  1. Tensorflow 2: Getting "WARNING:tensorflow:9 out of the last 9 calls to triggered tf.function retracing. Tracing is expensive"
  2. Loading multiple saved tensorflow/keras models for prediction

But neither of the two questions answers my question about how to avoid this warning. Plus I have also checked the links in the warning message but I couldn't solve my problem.

What I want

I simply want to avoid this warning. While I'm still getting the predictions from the models I noticed that the python program takes way too much time on doing predictions for a list of images.

What I'm using

Solution

After some tries to suppress the warning from the predict method, I have checked the documentation of Tensorflow and in one of the first tutorials on how to use Tensorflow it is explained that, by default, Tensorflow is executed in eager mode, which is useful for testing and debugging the network models. Since I have already tested my models many times, it was only required to disable the eager mode by writing this single python line of code:

tf.compat.v1.disable_eager_execution()

Now the warning doesn't show up anymore.

Upvotes: 7

Views: 6693

Answers (1)

user11530462
user11530462

Reputation:

For the benefit of community providing solution here

After some tries to suppress the warning from the predict method, I have checked the documentation of Tensorflow and in one of the first tutorials on how to use Tensorflow it is explained that, by default, Tensorflow is executed in eager mode, which is useful for testing and debugging the network models. Since I have already tested my models many times, it was only required to disable the eager mode by writing this single python line of code:

tf.compat.v1.disable_eager_execution()

Now the warning doesn't show up anymore. (paraphrased from Simone)

tf.compat.v1.disable_eager_execution() can only be called before any Graphs, Ops, or Tensors have been created. It can be used at the beginning of the program for migration projects from TensorFlow 1.x to 2.x.

For more details you can refer Eager execution

Upvotes: 4

Related Questions