CODEWITHSUNDEEP
tensorflowneural-networkout-of-memorygpukeras
mdornfe1
mdornfe1

Reputation: 2160

Why is Keras throwing a ResourceExhaustedError?

Keras is throwing a ResourceExhaustedError when training a convolutional autoencoder. I'm running the Tensorflow backend. The computer has both an Nvidia Tesla with 11 Gbs of memory and an Nvidia Quadro with 6 Gbs of memory. It seems like Tensorflow is using both GPUs? But I'm not too clear on that. Here is a minimal example of the code I'm using. In my example data is a numpy array of dimension=(100,1080,1920,1).

from keras.layers import Convolution2D, MaxPooling2D, UpSampling2D, Activation
from keras.models import Sequential
model = Sequential()
model.add(Convolution2D(16, 3, 3, border_mode='same', input_shape=(1080, 1920, 1)))
model.add(Activation('relu'))
model.add(MaxPooling2D((2, 2), border_mode='same'))
model.add(Convolution2D(16, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(UpSampling2D((2, 2)))
model.add(Convolution2D(1, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.compile(optimizer='adadelta', loss='binary_crossentropy')
model.fit(data, data)

It seems like the GPUs are running out of memory. The autoencoder has 2625 variables. So that doesn't seem like enough to fill up the video ram. The array data has a size of 1600 MBs. So that also shouldn't fill up the video ram. I'm guessing the problem has something to do with nb_epoch and batch_size, but I'm not entirely clear on what those parameters do. Is there a way alter those parameters to fix my problem?

Upvotes: 3

Views: 5981

Answers (2)

hyun woo Cho
hyun woo Cho

Reputation: 2250

change Convolution2D - > Conv2D
refer to : https://stackoverflow.com/a/46032412/7148586

Upvotes: 1

Marcin Możejko
Marcin Możejko

Reputation: 40506

Let's try to approximate how much of memory do you need for this network (despite the memory for parameters):

  1. Input: 1600MB
  2. First layer : ~ 1600MB * 16 / 4 = 6400MB (you have 16 times more channels and pooling layer - decreases the size of its input by 4).
  3. Second layer : 6400MB (the input and the output of a layer is the same).
  4. Third Layer : 25600MB (your are increasing the amount of data 4 times).
  5. Output layer : 1600MB (the same as input).

You may see that third layer alone need more than 11GB of memory. Moreover - all forward values are stored - for a backpropagation algorithm - so in fact you need to sum up all of this values in order to obtain the final estimation of a memory need.

Upvotes: 7

Related Questions