CODEWITHSUNDEEP
c++tensorflowgpu
Hajdu Gábor
Hajdu Gábor

Reputation: 329

Tensorflow with c++ per_process_gpu_memory_fraction not working

I have my build of tensorflow on windows where i use two sessions. I would like to set the amount of memory the session uses from my gpu, but it doesn't seem to work. My code looks somewhat like this:

void createSession(Session** sess, bool allow_growth, double memory_fraction)
{
    tensorflow::SessionOptions session_options;
    session_options.config.mutable_gpu_options()->set_allow_growth(allow_growth);
    session_options.config.mutable_gpu_options()->set_per_process_gpu_memory_fraction(memory_fraction);

    Status status = NewSession(session_options, sess);
    ...
    GraphDef graph_def;
    ...
    status = (*sess)->Create(graph_def);
    ...
}

This function is called twice at initialization, once for each session i need. No matter how i change the values of allow_growth and memory_fraction, the two session always uses 80 percent of my gpu memory.

My current tensorflow version is 1.13, but the same problem was existent with 1.12 version.

Why is this not working correctly?

Upvotes: 4

Views: 384

Answers (1)

Mate Zabo
Mate Zabo

Reputation: 1646

Unfortunately in Tensorflow the GPU memory allocator is created once per GPU device per process. So you can set this value only once before you initialize your first session. After this point it will use the same amount for every session you create. You cannot change it with the per_process_gpu_memory_fraction() method.

I assume that before your session initialization there is an other session initialization maybe with different options or with the default SessionOptions and that's why your modification has no effect on it.

Upvotes: 4

Related Questions