CODEWITHSUNDEEP
tensorflowparallel-processinggpumultiple-instances
Abhijith bhat n
Abhijith bhat n

Reputation: 71

how can i train two tensorflow scripts on single gpu parallelly?

I am getting error when i run 2 tensorflow scripts on single GPU. I have tried growth and GPU memory allocations steps, still the first script executes without problem while 2nd script failes with ResourceExhaustedError , Graph session creation error.

Kindly help.

Upvotes: 2

Views: 1381

Answers (2)

hegerber
hegerber

Reputation: 201

You should do the following:

# don't allow cases where a single script takes up all VRAM
# this way we can try to run several scripts at the same time
config = tf.ConfigProto()
config.gpu_options.allow_growth = True

with tf.Session(config=config) as sess:
    ...

If any of you know how to train two graphs in parallel in a single script, please let me know.

Upvotes: 0

bnorm
bnorm

Reputation: 399

Run each program separately first for a few iterations and check nvidia-smi dmon to see how much memory that program actually requires. Then set config.gpu_options.per_process_gpu_memory_fraction = ... in your session configuration based on the memory information you learned from nvidia-smi dmon. If the memory required for both is greater than what you have available then you will run into this resources exhausted error.

Upvotes: 1

Related Questions