CODEWITHSUNDEEP
pythontensorflow
Maybe
Maybe

Reputation: 2279

tensorflow use all GPU memory

I run my network on COCO2014(input images have shape 256, 256, 3) using tf.data.Dataset, and have tensorflow session configured as follows

sess_config = tf.ConfigProto(intra_op_parallelism_threads=1,
                             inter_op_parallelism_threads=1,
                             allow_soft_placement=True)
sess_config.gpu_options.allow_growth=True
sess = tf.Session(config=sess_config)

I spot this will always consume all my GPU memory(11G). I even tries to configure the session as follows

sess_config = tf.ConfigProto(intra_op_parallelism_threads=1,
                             inter_op_parallelism_threads=1,
                             allow_soft_placement=True)
sess_config.gpu_options.per_process_gpu_memory_fraction = 0.4
sess = tf.Session(config=sess_config)

But still all GPU memory is used. Why would this happens? How can I fix it?

Upvotes: 1

Views: 394

Answers (2)

Maybe
Maybe

Reputation: 2279

The reason why Tensorflow use all GPU memory is that I use another temporary plain tf.Session(). Although this temporary session is closed immediately after it is used, Tensorflow does not free the GPU memory it allocates. Solution could be 1) do not use two sessions. 2) configure the temporary session as the one described in the question.

Upvotes: 1

townie
townie

Reputation: 322

1) smaller memory footprint: sess_config.gpu_options.per_process_gpu_memory_fraction = 0.1

2) reducing the size of the image processed

(256, 256, 3) -> (128, 128, 3)

Upvotes: 0

Related Questions