CODEWITHSUNDEEP
pythonmemorymachine-learningtensorflowdeep-learning
Maksim Khaitovich
Maksim Khaitovich

Reputation: 4792

TensorFlow - nn.max_pooling increases memory usage enormously

I try to create a simple convolution neural network in TensorFlow. Everything seems fine when I run my code below. I run it in Spyder IDE and monitor memory usage - it grows to 64-65% on my laptop and not goes any further.

batch_size = 16
patch_size = 5
depth = 16
num_hidden = 64

graph = tf.Graph()

with graph.as_default():

  # Input data.
  tf_train_dataset = tf.placeholder(
    tf.float32, shape=(batch_size, image_size, image_size, num_channels))
  tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
  tf_valid_dataset = tf.constant(valid_dataset)
  tf_test_dataset = tf.constant(test_dataset)

  # Variables.
  layer1_weights = tf.Variable(tf.truncated_normal(
      [patch_size, patch_size, num_channels, depth], stddev=0.1))
  layer1_biases = tf.Variable(tf.zeros([depth]))
  layer2_weights = tf.Variable(tf.truncated_normal(
      [patch_size, patch_size, depth, depth], stddev=0.1))
  layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))
  layer3_weights = tf.Variable(tf.truncated_normal(
      [image_size // 4 * image_size // 4 * depth, num_hidden], stddev=0.1))
  layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
  layer4_weights = tf.Variable(tf.truncated_normal(
      [num_hidden, num_labels], stddev=0.1))
  layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))

  # Model.
  #Now instead of using strides = 2 for convolutions we will use maxpooling with
  #same convolution sizes
  def model(data):
    conv = tf.nn.conv2d(data, layer1_weights, [1, 2, 2, 1], padding='SAME')
    hidden = tf.nn.relu(conv + layer1_biases)
    conv = tf.nn.conv2d(hidden, layer2_weights, [1, 2, 2, 1], padding='SAME')
    hidden = tf.nn.relu(conv + layer2_biases)
    shape = hidden.get_shape().as_list()
    reshape = tf.reshape(hidden, [shape[0], shape[1] * shape[2] * shape[3]])
    hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)
    return tf.matmul(hidden, layer4_weights) + layer4_biases

  # Training computation.
  logits = model(tf_train_dataset)
  loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))

  # Optimizer.
  optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

  # Predictions for the training, validation, and test data.
  train_prediction = tf.nn.softmax(logits)
  valid_prediction = tf.nn.softmax(model(tf_valid_dataset))
  test_prediction = tf.nn.softmax(model(tf_test_dataset))

num_steps = 1001

with tf.Session(graph=graph) as session:
  tf.initialize_all_variables().run()
  print('Initialized')
  for step in range(num_steps):
    offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
    batch_data = train_dataset[offset:(offset + batch_size), :, :, :]
    batch_labels = train_labels[offset:(offset + batch_size), :]
    feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
    _, l, predictions = session.run(
      [optimizer, loss, train_prediction], feed_dict=feed_dict)
    if (step % 50 == 0):
      print('Minibatch loss at step %d: %f' % (step, l))
      print('Minibatch accuracy: %.1f%%' % accuracy(predictions, batch_labels))
      print('Validation accuracy: %.1f%%' % accuracy(
        valid_prediction.eval(), valid_labels))
  print('Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels))

Ok, after that I try to introduce a maxpooling with kernel of 2, and reduce the size of data with maxpooling instead of conv layer. It looks like below:

conv = tf.nn.conv2d(data, layer1_weights, [1, 1, 1, 1], padding='SAME')
    maxpool = tf.nn.max_pool(conv, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME')
    hidden = tf.nn.relu(maxpool + layer1_biases)
    conv = tf.nn.conv2d(hidden, layer2_weights, [1, 2, 2, 1], padding='SAME')

Everything else remain same. But when I do this (notice, I introduced only one maxpooling layer) the memory usage grows to 100% and my iPython kernel just dies. Any ideas of such strange behavior, why the memory usage gets so big? Am I doing something wrong? Any suggestions on how to decrease memory usage?

Upvotes: 1

Views: 441

Answers (1)

shekkizh
shekkizh

Reputation: 165

Assume you are using one 3x3 filter on single channel 6x6 input.

When you are doing strided convolution of strides 2 you produce a result that is 3x3.

So effectively you use input 36 units + filter 9 units + output 9 units of memory.

Now when you are trying to apply max pool after un-strided convolution, your convolution layer produces 6x6 output on which you apply maxpool to get 3x3 output. Notice that we have an intermediate result of size 6x6 that is in memory before applying max pool.

So over here use input 36 units + filter 9 units + intermediate result 36 units + output 9 units of memory

This would explain the memory usage. This is not a strange behavior. As to why it depleted your resources completely is dependent on your image size, batch size and number of filters you are using.

Upvotes: 2

Related Questions