CODEWITHSUNDEEP
tensorflowmachine-learningkeras
ojp
ojp

Reputation: 1033

Activate recurrent dropout when evaluating a model using tf.Keras.Sequential API and LSTM

I have pre-trained an LSTM encoder/decoder without using dropout so that I can use GPU and speed up training. I have now re-built the model and added dropout and recurrent dropout and would like to activate this during inference to estimate the confidence interval for my predictions. I am not sure where or how to add the training=True argument in my model using the sequential API. I thought I could do it when evaluating the model but this doesn't seem to be the case:

encoder_decoder_dropout.evaluate(val, steps=VALIDATION_STEPS, training=True)

evaluate() got an unexpected keyword argument 'training'

My model is below. Any suggestions on how to activate the dropout during inference would be very much appreciated.

encoder = encoder_decoder_dropout_trained.layers[0]

decoder_dropout = tf.keras.Sequential([
  tf.keras.layers.RepeatVector(look_back, input_shape=[60]),
  tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(100, dropout=0.2,
                                                     recurrent_dropout=0.2,
                                                     return_sequences=False)), 
  tf.keras.layers.Dense(look_forward*num_features,
                        kernel_initializer=tf.initializers.glorot_normal()),
  tf.keras.layers.Reshape([look_forward, num_features])
], name = 'decoder')

encoder_decoder_dropout = tf.keras.Sequential([encoder, decoder_dropout])
encoder_decoder_dropout.set_weights(encoder_decoder_dropout_trained.get_weights())

Upvotes: 1

Views: 658

Answers (1)

Marco Cerliani
Marco Cerliani

Reputation: 22031

To activate dropout for inference time u simply have to specify training=True in the layer of interest (in lstm in your case)

with training=False

inp = Input(shape=(10, 1))
x = LSTM(1, recurrent_dropout=0.3)(inp, training=False)
m = Model(inp,x)
# m.compile(...)
# m.fit(...)

X = np.random.uniform(0,1, (1,10,1))

output = []
for i in range(0,100):
    output.append(m.predict(X)) # always the same

with training=True

inp = Input(shape=(10, 1))
x = LSTM(1, recurrent_dropout=0.3)(inp, training=True)
m = Model(inp,x)
# m.compile(...)
# m.fit(...)

X = np.random.uniform(0,1, (1,10,1))

output = []
for i in range(0,100):
    output.append(m.predict(X)) # always different

you need to use the keras functional format for this

Upvotes: 1

Related Questions