Trouble with the predict function of my model while in a separate thread

Question

I have a saved model that I can generate a prediction when I run it on the main thread. When I try to use the same function in a thread I get the following error.

Note a summary of my code will be pasted after the error.

Exception in thread Thread-6:
Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "D:\PiChess\Core\ComModule.py", line 49, in threader
    b = Globals.moduleManager.GetPrediction(predictionSet)
  File "D:\PiChess\Core\ManagementModel.py", line 56, in GetPrediction
    return self.Model.predict(data, steps=10)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1878, in predict
    self, x, batch_size=batch_size, verbose=verbose, steps=steps)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py", line 295, in predict_loop
    batch_outs = f(ins)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\keras\backend.py", line 2983, in __call__
    self._make_callable(feed_arrays, feed_symbols, symbol_vals, session)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\keras\backend.py", line 2928, in _make_callable
    callable_fn = session._make_callable_from_options(callable_opts)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\client\session.py", line 1471, in _make_callable_from_options
    return BaseSession._Callable(self, callable_options)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\client\session.py", line 1425, in __init__
    session._session, options_ptr, status)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Requested tensor connection from unknown node: "dense_input:0".

Exception ignored in: <bound method BaseSession._Callable.__del__ of <tensorflow.python.client.session.BaseSession._Callable object at 0x000001E7749C5438>>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\client\session.py", line 1455, in __del__
    self._session._session, self._handle, status)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No such callable handle: 2093487919952

The model is loaded into memory when I start running the code. To do this I created a globals object that loads the model the following way.

self.ModelName = ModelName
self.Model = load_model("Models/" + self.ModelName + '.mdl')
self.Model._make_predict_function()

To use the predict function it is run in a thread

predict = self.Model.predict(data, steps=10)

I know the data supplied is correct for I get a result if I just cut out the threading aspect out of my code and make my code into a single thread.

I cut out most of code since its split up into multiple files and as I tried to paste here stack overflow told me I had to much code.

If needed please let me know which sections of code would be most beneficial to post in as an edit.

Answer 1

not sure u guys have found the solution or not, i just encountered the similar problem, and solved it, just share here.

In short, the fix is very simple: after u called the K.clear_session() u must reload the model again, even in same process!

My previous problematic code:

model = create_input_selection_model(config.input_shape)

for i in range(batches):
  # Do a big batch prediction (100K or more depends on ur physical memory size).
  sub_preds = model.predict_generator()
  # For solving the memory problem, call 'K.clear_session()' to release GPU memory.
  K.clear_session()

Above code will run into the problem OP posted.

Fix is quite simple: reload the model in each for loop iteration:

for i in range(batches):

  model = create_input_selection_model(config.input_shape) # <- reload model here.

  # Do a big batch prediction (100K or more depends on ur physical memory size).
  sub_preds = model.predict_generator()
  # For solving the memory problem, call 'K.clear_session()' to release GPU memory.
  K.clear_session()

Answer 2

You may need to store both the graph (and sometimes session) that were used to load the model, and pass them to your separate thread.

If you are using Keras:

import keras.backend as K

# Load model
self.sess = tf.Session()
K.set_session(sess)
self.ModelName = ModelName
self.Model = load_model("Models/" + self.ModelName + '.mdl')
self.Model._make_predict_function()
self.graph = tf.get_default_graph()

# Predict
K.set_session(sess)
with self.sess.as_default():
    with self.graph.as_default():
        predict = self.Model.predict(data, steps=10)

If you are using TensorFlow:

# Load model
self.sess = tf.Session()
self.sess.run(tf.global_variables_initializer())
self.Model.load(self.sess, "Models/" + self.ModelName + '.mdl')
self.graph = tf.get_default_graph()

# Predict
with self.graph.as_default():
    predict = self.Model.predict(self.sess, ...)

A somewhat related answer: The problem you are running into is because the tensorflow model is not loaded and used in the same thread.