CODEWITHSUNDEEP
pythontensorflowkeras
Sumerechny
Sumerechny

Reputation: 168

Fitting multiple Keras models consecutively leads to GPU OOM

I want to fit and save three models consecutively on the same data (the NIH-Chest set - https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community). Even though I can do this separately I am wondering if I can do this from a list (see code below). At this point however this leads to a GPU memory saturation and consequently an OOM error. This is caused by the dataset not being flushed from the GPU memory and trying to reload it as well for the second and third fit.

All the solutions relating clearing the GPU memory either involve closing Cuda or restarting Keras or the kernel...which in causes fit to be erased (no surprise there).

I am wondering, is it possible to fit multiple models consecutively from a list or should I just go back to basics and fit and save every model separate?

Small code snippet (for a quick test run > small# epochs and validation);

epochs = 2
i = 0
for model in model_list:
    i += 1
    model.fit_generator(train_data, epochs = epochs, validation_data = validation_data, validation_steps = 100)
    model.save('model_{}'.format(i))

Upvotes: 1

Views: 1076

Answers (3)

DirtyNor
DirtyNor

Reputation: 1

I resolved this issue recently, in my case the other answers were missing a garbage collect call.

import gc
import tensorflow as tf

... 
for iModelParams in modelsParams:
  iModel = BuildModel(iModelParams)
  iModel.fit(...)
  del(model)
  gc.collect()
  tf.keras.backend.clear_session()

Probably would be cleaner to just use with: but I haven't tried it out.

Upvotes: 0

Utku Can
Utku Can

Reputation: 995

Just add this at the end of loop,

K.clear_session()

Note: to import it

from keras import backend as K

Upvotes: 2

Ashwin Geet D'Sa
Ashwin Geet D'Sa

Reputation: 7369

You can use del(model) at the end of the loop. This would delete the memory of the model which you would not intend to use later.

Upvotes: 3

Related Questions