Memory error using Theano on AWS GPU

Question

I am building a simple a FeedForward Network for a classification task. I am using keras on top of theano on AWS g2.2xlarge instance. My network config is as follows:

batch_size = 32
nb_epoch = 50
dimof_input = 8100
dimof_middle = 16384
dimof_end = 16384
dropout = 0.2
dimof_output = 3

print('Build model...')

model = Sequential()
model.add(Dense(dimof_middle, input_shape=(dimof_input,)))
model.add(Activation('relu'))
model.add(Dropout(0.2))

model.add(Dense(dimof_end))
model.add(Activation('relu'))
model.add(Dropout(0.2))

#model.add(Dense(dimof_output))
#model.add(Activation('sigmoid'))
#model.add(Dense(dimof_output, activation='softmax'))

model.add(Dense(dimof_output))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam')

When I compile the model, I get MemoryError

MemoryError: ('Error allocating 1073741824 bytes of device memory (CNMEM_STATUS_OUT_OF_MEMORY).', "you might consider using 'theano.shared(..., borrow=True)'")

From AWS this is a 4GB video memory, 16 GB ram machine. If I half the value of 'dimof_middle' to 8196. everything runs fine.

1073741824 bytes ~ 1 GB which is well within GPU memory

What wrong am I doing ? Any help would be much appreciated!

Answer 1

The error message is not about the total amount of allocated memory. It is about the amount that it was trying to allocate when failed. Theano allocates and deallocates memory during computation.

You can disable CNMeM and use nvidia-smi to find out real allocation.

http://deeplearning.net/software/theano/library/config.html#envvar-THEANORC