CODEWITHSUNDEEP
pythontensorflowkerasthread-safetyreinforcement-learning
LeckieNi
LeckieNi

Reputation: 466

Running Keras model for prediction in multiple threads

similar to this question I was running an asynchronous reinforcement learning algorithm and need to run model prediction in multiple threads to get training data more quickly. My code is based on DDPG-keras on GitHub, whose Neural Network was build on top of Keras & Tensorflow. Pieces of my code are shown below:

While running these codes, I encountered a Tensorflow Exception:

Using TensorFlow backend.
create_actor_network
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/Users/niyan/code/routerRL/A3C.py", line 26, in run
    action = self.actor.model.predict(state[np.newaxis, :])[0]
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/keras/engine/training.py", line 1269, in predict
    self._make_predict_function()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/keras/engine/training.py", line 798, in _make_predict_function
    **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1961, in function
    return Function(inputs, outputs, updates=updates)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1919, in __init__
    with tf.control_dependencies(self.outputs):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3583, in control_dependencies
    return get_default_graph().control_dependencies(control_inputs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3314, in control_dependencies
    c = self.as_graph_element(c)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2405, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2484, in _as_graph_element_locked
    raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("concat:0", shape=(?, 4), dtype=float32) is not an element of this graph.

So how can I use a trained Keras model (using Tensorflow as backend) to concurrently predict in multiple threads?

Update on April 2nd: I tried coping model over weight, but didn't work:

for roundNo in xrange(self.param['max_round']):
    for agent in self.AgentPool:
        agent.syncModel(self.getEnv(), self.actor, self.critic, eps)
        agent.start()
    for agent in self.AgentPool:
        agent.join()

def syncModel(self, env_, actor_, critic_, eps_):
    """synchronize A-C models before collecting data"""
    # TODO copy env, actor, critic
    self.env = env_     # shallow copy
    self.actor.model.set_weights(actor_.model.get_weights())        # deep copy, by weights
    self.critic.model.set_weights(critic_.model.get_weights())      # deep copy, by weights
    self.eps = eps_     # shallow copy
    self.data = {}

EDIT: see this jaara/AI-blog on Github, seems

model._make_predict_function()  # have to initialize before threading

works.

The author explained a little on this issue. For further discussion, see this issue on Keras

Upvotes: 23

Views: 8838

Answers (4)

aargun
aargun

Reputation: 136

Because of Global Interpreter Lock of Python, you should consider using multiprocessing instead of threading. Ray is a great API to build distributed applications with Python and they already have a reinforcement learning framework called RLlib. I would highly recommend taking a look at Ray, especially for reinforcement learning applications.

Upvotes: 0

aSaffary
aSaffary

Reputation: 863

multi threading in python doesn't necessarily make a better use of your resources since python uses global interpreter lock and only one native thread can run at a time.

in python, usually you should use multi processing to utilize your resources, but since we're talking about keras models, I'm not sure even that is the right thing to do. loading several models in several processes has its own overhead, and you could simply increase the batch size as others have already pointed out.

OR if you have a heavy pre-processing stage you could preprocess your data in one process and predict them in another (although I doubt that would be necessary either).

Upvotes: 3

k.avinash
k.avinash

Reputation: 111

Keras is not thread safe, for predicting large batch you can use batch_size to set Max limits. If you are deploying to production than the ideal is to convert the model weights tensorflow protobuf and than use tensorflow serving.

You can follow this blog http://machinelearningmechanic.com/keras/2019/06/26/keras-serving-keras-model-quickly-with-tensorflow-serving-and-docker-md.html

Upvotes: 0

DachuanZhao
DachuanZhao

Reputation: 1339

It's a bad idea to predict data in multiple threads . You can use greater batch_size in model.predict when you predict data offline and use tensorflow serving when you predict data online.

Upvotes: 0

Related Questions