CODEWITHSUNDEEP
pythontensorflowkerasconv-neural-network
Olca Orakcı
Olca Orakcı

Reputation: 382

CNN training system memory issue

i am trying to train a CNN on CPU with a input of (102480,40,40,2) dimensions and output of (40,40). Before i trained a model using (102480,40,40) input and it worked perfectly. Now when i run the script systems memory usage increases until it needs more and my pc freezes. I only have 8 GB ram.

i tried adding this line to my code which did not help at all.

sess = tf.Session(config=tf.ConfigProto(inter_op_parallelism_threads=1, intra_op_parallelism_threads=1))

This is my model:

model = Sequential()
model.add(Conv2D(64,(3,3), input_shape = train_data.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Conv2D(64,(3,3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))


model.add(Dense(1600))
model.add(Reshape((40,40)))
model.add(Activation('sigmoid'))

model.compile(loss='MSE',
             optimizer='SGD',
             metrics=['MAE'])

history = model.fit(train_data, train_target,
          batch_size=256,
          epochs=100,
          validation_split=0.1,
          callbacks = [cp_callback])

Can i somehow decrease the system memory usage without losing accuracy?

Upvotes: 0

Views: 914

Answers (2)

Olca Orakcı
Olca Orakcı

Reputation: 382

Instead of decreasing the batch size (which can impact the accuracy) I simply changed the data types from float64 to float32. This cut the memory usage by half.

Upvotes: 1

Nosvan
Nosvan

Reputation: 151

The calculation of the model memory depends on the batch size. In fact, you can roughly estimate the memory usage by multiplying the batch size by the input shape. It's like you have 256 times the input size in your case. You can try to lower the batch size, but that may impact your accuracy. A way to minimize it's as stated in this answer: https://datascience.stackexchange.com/a/12533

In the case that you do need bigger batch sizes but it will not fit on your GPU, you can feed a small batch, save the gradient estimates and feed one or more batches, and then do a weight update. This way you get a more stable gradient because you increased your virtual batch size.

Finally, you can calculate the approximate memory usage using the solution of this answer https://stackoverflow.com/a/46216013/6598433 by ZFTurbo that I put below:

def get_model_memory_usage(batch_size, model):
    import numpy as np
    from keras import backend as K

    shapes_mem_count = 0
    for l in model.layers:
        single_layer_mem = 1
        for s in l.output_shape:
            if s is None:
                continue
            single_layer_mem *= s
        shapes_mem_count += single_layer_mem

    trainable_count = np.sum([K.count_params(p) for p in set(model.trainable_weights)])
    non_trainable_count = np.sum([K.count_params(p) for p in set(model.non_trainable_weights)])

    number_size = 4.0
    if K.floatx() == 'float16':
         number_size = 2.0
    if K.floatx() == 'float64':
         number_size = 8.0

    total_memory = number_size*(batch_size*shapes_mem_count + trainable_count + non_trainable_count)
    gbytes = np.round(total_memory / (1024.0 ** 3), 3)
    return gbytes

Upvotes: 0

Related Questions