CODEWITHSUNDEEP
pythontensorflowmemory-leakskerasgpu
Punisher
Punisher

Reputation: 684

Keras Memory Leak

I am using Keras TensorFlow 1.8 and having a memory leak in my gpu (1080 ti). After training the network, my memory is used even after closing python completely. In nvidia-smi it no longer shows the python but the memory usage is still there.

I cannot restart the computer because other users are running processes (I am sure they are not using the gpu).

[edit: I uploaded the wrong screenshot]

enter image description here

Upvotes: 2

Views: 1648

Answers (2)

benvigano
benvigano

Reputation: 55

After speding way too much trying to keras.clear_session() and gc.collect() my way through this, I gave up and created a reliable workaround. It's a decorator that allows to run a function in a separate script.

It's called scriptifier, it installs via pip install scriptifier

It automatically generates the script and takes care of passing the arguments and returns as long as they are pickleable or keras models or lists of keras models... (for documentation see: github)

It should look like this:

from scriptifier import scriptifier

def func_1(in):
    ...
    model.fit()
    ...
    return out

scriptified_func_1 = scriptifier.run_as_script(func_1)
out = scriptified_func_1(in)

Upvotes: 0

marilena.oita
marilena.oita

Reputation: 994

Always 

K.clear_session()

where K is defined as 

from keras import backend as K

at the end of your processing.

It prevents Tensorflow memory leakage.

You could also try 

import gc
gc.collect()

or ,

from the beginning of your tf session, prevent tensorflow using the whole gpu power:

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)

Upvotes: 3

Related Questions