CODEWITHSUNDEEP
pythontensorflowlstm
nickzxd
nickzxd

Reputation: 61

Tensorflow: DropoutWrapper leads to different output?

I build a LSTM like:

lstm_cell = tf.nn.rnn_cell.LSTMCell(n_hidden, forget_bias=1.0, state_is_tuple=True, activation=tf.nn.tanh)
lstm_cell = tf.nn.rnn_cell.DropoutWrapper(lstm_cell, output_keep_prob=0.5)
lstm_cell = tf.nn.rnn_cell.MultiRNNCell([lstm_cell] * 3, state_is_tuple=True)

Then i train the model, and save variables. The next time i load saved variables and skip training, it gives me a different prediction.

If i change the output_keep_prob to 1, this model can always show me the same prediction, but if the output_keep_prob is less than 1, like 0.5, this model shows me different prediction every time.

So i guess if the DropoutWrapper leads to different output? If so, how can i solve this problem?

Thanks

Upvotes: 5

Views: 2982

Answers (2)

Dwight Crow
Dwight Crow

Reputation: 368

Dropout will randomly activate a subset of your net, and is used during training for regularization. Because you've hardcoded dropout as 0.5, it means every time you run the net half your nodes will be randomly silenced, thus producing a different and random result.

You can sanity check this is what's happening by setting a seed, so that the same nodes will be 'randomly' silenced by dropout each time. However, what you probably want to do is make dropout a placeholder so that you can set it to 1.0 (ie keep all the nodes) during test time, which will produce the same output for each input deterministically.

Upvotes: 1

kempy
kempy

Reputation: 616

Try using the seed keyword argument to DropoutWrapper(...):

lstm_cell = tf.nn.rnn_cell.DropoutWrapper(lstm_cell, output_keep_prob=0.5, seed=42)

See the docs here for DropoutWrapper.__init__

Upvotes: 1

Related Questions