CODEWITHSUNDEEP
pythontensorflow
H. Senkaya
H. Senkaya

Reputation: 773

How to use Dropout in RNN correctly

I want to use the dropout function of tensorflow to check if I can improve my results (TPR, FPR) of my recurrent neural network. However I implemented it by following a guide. So I am not sure if I did any mistakes. But if I train my model with with e.g. 10 epochs I get nearly the same results after validation. Thats why I am not sure if I use the dropout function correctly. Is this the right implementation in the following code or did I made something wrong? If I did everything right why do I get nearly the same result then?

hm_epochs = 10
n_classes = 2
batch_size = 128
chunk_size = 341
n_chunks = 5
rnn_size = 32

dropout_prop = 0.5 # Dropout, probability to drop a unit

batch_size_validation = 65536
x = tf.placeholder('float', [None, n_chunks, chunk_size])
y = tf.placeholder('float')

def recurrent_neural_network(x):
     layer = {'weights':tf.Variable(tf.random_normal([rnn_size, n_classes])),
              'biases':tf.Variable(tf.random_normal([n_classes]))}

     x = tf.transpose(x, [1,0,2])
     x = tf.reshape(x, [-1, chunk_size])
     x = tf.split(x, n_chunks, 0)

     lstm_cell = rnn.BasicLSTMCell(rnn_size) 
     outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)

     output = tf.matmul(outputs[-1], layer['weights']) + layer['biases']

     #DROPOUT Implementation -> is this code really working?
     #The result is nearly the same after 20 epochs...
     output_layer = tf.layers.dropout(output, rate=dropout_prop)

     return output

def train_neural_network(x):
     prediction = recurrent_neural_network(x)
     cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=prediction,labels=y))
     optimizer = tf.train.AdamOptimizer().minimize(cost)

     with tf.Session() as sess:
         sess.run(tf.global_variables_initializer())

         for epoch in range(1,hm_epochs+1):
             epoch_loss = 0

             for i in range(0, training_data.shape[0], batch_size):
                 epoch_x = np.array(training_data[i:i+batch_size, :, :], dtype='float')
                 epoch_y = np.array(training_labels[i:i+batch_size, :], dtype='float')

                 if len(epoch_x) != batch_size:
                     epoch_x = epoch_x.reshape((len(epoch_x), n_chunks, chunk_size))
                 else:
                     epoch_x = epoch_x.reshape((batch_size, n_chunks, chunk_size))

                 _, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y}) 
                 epoch_loss += c

 train_neural_network(x)
 print("rnn - finished!")

Upvotes: 1

Views: 6509

Answers (1)

Kevin He
Kevin He

Reputation: 1250

In its most basic form, dropout should happen inside the cell and applied to the weights. You only applied it afterwards. This article explained it pretty well with some good visualization and few variations.

To use it in your code, you can either

  1. Implement your own RNN cell where the keep probability is a parameter that initializes the cell or is a parameter that got passed in when it's called every time.

  2. Use an rnn dropout wrapper here.

Upvotes: 2

Related Questions