CODEWITHSUNDEEP
pythontensorflowkerasdropout
Luciano Dourado
Luciano Dourado

Reputation: 502

What is the proper way to implement a dropout layer in a CNN?

Is this proceeding correct?

My intention was to add a dropout layer after concatenation, but to do so i needed to adjust the concat layer's output to the appropriate shape (samples, timesteps, channels), hence expanding the dimension from (None, 4096) to (None, 1, 4096) and consequently undo the operation after the output.

enter image description here

Upvotes: 4

Views: 1767

Answers (2)

AloneTogether
AloneTogether

Reputation: 26708

If you plan to use the SpatialDropout1D layer, it has to receive a 3D tensor (batch_size, time_steps, features), so adding an additional dimension to your tensor before feeding it to the dropout layer is one option that is perfectly legitimate. Note, though, that in your case you could use both SpatialDropout1D or Dropout:

import tensorflow as tf

samples = 2
timesteps = 1
features = 5
x = tf.random.normal((samples, timesteps, features))
s = tf.keras.layers.SpatialDropout1D(0.5)
d = tf.keras.layers.Dropout(0.5)

print(s(x, training=True))
print(d(x, training=True))

tf.Tensor(
[[[-0.5976591  1.481788   0.         0.         0.       ]]

 [[ 0.        -4.6607018 -0.         0.7036132  0.       ]]], shape=(2, 1, 5), dtype=float32)
tf.Tensor(
[[[-0.5976591  1.481788   0.5662646  2.8400114  0.9111476]]

 [[ 0.        -0.        -0.         0.7036132  0.       ]]], shape=(2, 1, 5), dtype=float32)

I think that SpatialDropout1D layers are most suitable after CNN layers.

Upvotes: 1

Sascha Kirch
Sascha Kirch

Reputation: 514

Instead of explicitly adding a new dimension, in tensorflow 2.7.0 you could just use keepdims=True as an argument of the GlobalAveragePooling2D layer.

Example:

def TestModel():
  # specify the input shape
  in_1 = tf.keras.layers.Input(shape = (256,256,3))
  in_2 = tf.keras.layers.Input(shape = (256,256,3))

  x1 = tf.keras.layers.Conv2D(64, (3,3))(in_1)
  x1 = tf.keras.layers.LeakyReLU()(x1)
  x1 = tf.keras.layers.GlobalAveragePooling2D(keepdims = True)(x1)

  x2 = tf.keras.layers.Conv2D(64, (3,3))(in_2)
  x2 = tf.keras.layers.LeakyReLU()(x2)
  x2 = tf.keras.layers.GlobalAveragePooling2D(keepdims = True)(x2)
 
  x = tf.keras.layers.concatenate([x1,x2])
  x = tf.keras.layers.SpatialDropout2D(0.2)(x)
  x = tf.keras.layers.Dense(1000)(x)

  # create the model
  model = tf.keras.Model(inputs=(in_1,in_2), outputs=x)

  return model

#Testcode
model = TestModel()
model.summary()
tf.keras.utils.plot_model(model, show_shapes=True, expand_nested=False, show_dtype=True, to_file="model.png")

enter image description here

If you require to squeeze it in the end, you could still do it.

Upvotes: 3

Related Questions