CODEWITHSUNDEEP
tensorflowobject-detection-api
Steve Goley
Steve Goley

Reputation: 193

Defining GPU options within Tensorflow Object Detection API

I am able to train on a local machine that has 4x 1080Ti's and as others have noted TF grabs all the available memory on my machine. After poking around for a bit most searches lead me to solutions for the base TF and not the Object Detection API, for example:

How to prevent tensorflow from allocating the totality of a GPU memory?

How do I access these sorts of options within the Object Detection API? How can I have similar TF style control over training within the OD API? Is there a proper way to within the OD API / slim API?

I tried adding a GPUOptions message to that training.proto but that didn't seem to make a difference.

Upvotes: 4

Views: 3489

Answers (2)

Eypros
Eypros

Reputation: 5723

I would like also to add the second choice for instructing tensorflow to use a portion of the available memory. According to this guide there are 2 options:

  • gpu_options.per_process_gpu_memory_fraction = 0.xxx

  • gpu_options.allow_growth

With the first one (as it has previously mentioned in the answers) a percentage of the whole available GPU memory is defined to be used while with the second you instruct tensorflow to use only as much memory as it's necessary to.

According to the documentation the first one should be somehow better if we know beforehand the necessary memory requirement since it allows for

... more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.

On the other hand if we don't know the GPU memory requirements beforehand I would prefer the second one even with a slight performance sacrifice.

As for the actual use in tensorflow's recent API changes I have inserted them like this:

session_config = tf.ConfigProto()
session_config.gpu_options.allow_growth = True
# or use
# session_config.gpu_options.per_process_gpu_memory_fraction = 0.xxx

tf.logging.set_verbosity(tf.logging.INFO)
config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir, session_config=session_config)

in main() in model_main.py.

Tested on tensorflow 1.12.0.

Upvotes: 5

Raf
Raf

Reputation: 165

I ran into a similar problem and added this attribute to the session configuration in trainer.py which reduced the amount of video memory used:

session_config.gpu_options.per_process_gpu_memory_fraction = 0.6

Effect confirmed using 

watch nvidia-smi

Upvotes: 2

Related Questions