CODEWITHSUNDEEP
pythontensorflowkerasdeep-learning
MasayoMusic
MasayoMusic

Reputation: 614

Usage of k.clear_session() and tf.reset_default_graph() to clear graph between subsequent models

I can"t seem to clear the graph properly when loading multiple models subsequently.

k.clear_session()   
tf.reset_default_graph()

is just closing the program in Python after the first model is loaded. If remove the above lines, I am able to load subsequent models, but then I run into the memory leak.

>>> import keras
Using TensorFlow backend.
>>> keras.__version__
'2.2.4'
>>> import tensorflow as tf
>>> tf.__version__
'1.8.0'
>>> 





def evaluate_models(models_path_dir):
    models_paths = [os.path.join(models_path_dir, model) for model in os.listdir(models_path_dir) if model.endswith(".hdf5")]
    models_pairs = get_model_key(models_paths, global_model_keys)
    print(len(model_pairs)) #15
    for model_pair in models_pairs:
        model_path,model_key = model_pair
        img_height, img_width = 480, 480
        evaluate_validation_data(model_path, model_key)



def evaluate_validation_data(model_path,model_key):
    preprocess =  model_key
    valid_datagen = ImageDataGenerator(preprocessing_function = preprocess)
    valid_generator = valid_datagen.flow_from_directory(
    validation_data_dir,
    target_size = (img_height, img_width),
    batch_size = 30, 
    class_mode = 'categorical',
    shuffle = False)

    model = load_model(model_path)
    print("model path",model_path)
    print("image size", (img_height, img_width))
    print( model.evaluate_generator(valid_generator))
    k.clear_session()
    tf.reset_default_graph()

Is my usage of k.clear_session() and tf.reset_default() incorrect?

Thank you.

UPDATE:

I tried changing my code as follows and I am still running into the same problem:

def evaluate_validation_data(model_path,model_key):
        preprocess =  model_key
        valid_datagen = ImageDataGenerator(preprocessing_function = preprocess)
        valid_generator = valid_datagen.flow_from_directory(
        validation_data_dir,
        target_size = (img_height, img_width),
        batch_size = 10, 
        class_mode = 'categorical',
        shuffle = False)

        model = load_model(model_path)
        print("model path",model_path)
        print("image size", (img_height, img_width))
        print( model.evaluate_generator(valid_generator))
        k.clear_session()
        #tf.reset_default_graph()




>>> import keras
Using TensorFlow backend.
>>> keras.__version__
'2.2.4'
>>> import tensorflow as tf
>>> tf.__version__
'1.8.0'
>>>

Here is the what happens when the program executes:

39
Found 374 images belonging to 5 classes.
loaded model
model path E:\USER\TEST\model.hdf5
image size (480, 480)
[0.5056040882665843, 0.8609625604700915]
Found 374 images belonging to 5 classes.

And then closes

Upvotes: 3

Views: 7121

Answers (2)

MasayoMusic
MasayoMusic

Reputation: 614

It seems there is a bug with Keras above 2.2 and tf 1.8?

https://github.com/keras-team/keras/issues/10399

and I need to downgrade Keras to 2.1?

EDIT:

Just tested. Downgrading it 2.1 take cares of the bug.

Upvotes: 1

Juliano S Assine
Juliano S Assine

Reputation: 101

Loose the tf.reset_default_graph() and you should be good. As for the memory leaks, be sure you are running Keras 2.2.4 (and preferably tensorflow>=1.10 has better keras integration), I had a similar issue with Keras 2.2.2 crashing when loading multiple models in sequence and it disapeared after I updated to Keras 2.2.4.

Upvotes: 0

Related Questions