How to disable dropout while prediction in keras?

Question

I am using dropout in neural network model in keras. Little bit code is like

model.add(Dropout(0.5))
model.add(Dense(classes))

For testing, I am using preds = model_1.predict_proba(image).

But while testing Dropout is also participating to predict the score which should not be happen. I search a lot to disable the dropout but didn't get any hint yet.

Do anyone have solution to disable the Dropout while testing in keras??

Answer 1

1. This happens if you use a dropout layer.

2. If you have trained the model without validation data , model does not save in the checkpoint location.

3)if 1) and 2) is correct and if you save model using this approach "model.save('model_name.h5')" , the randomness of the dropout is preserved during prediction and hence different result every time. :-)

Answer 2

The Dropout layer has a call argument named 'training', when you use model.fit, Keras sets automatically this argument to true, and when you call model and use model(input), Keras sets this argument to false.

You can use this argument in custom layers and models to control Dropout manually. See Keras's official documentation for more information.

Answer 3

To activate dropout for inference time u simply have to specify training=True in the layer of interest (Dropout in our case):

with training=False

inp = Input(shape=(10,))
x = Dropout(0.3)(inp, training=False)
x = Dense(1)(x)
m = Model(inp,x)
# m.compile(...)
# m.fit(...)

X = np.random.uniform(0,1, (1,10))

output = []
for i in range(0,100):
    output.append(m.predict(X)) # always the same

with training=True

inp = Input(shape=(10,))
x = Dropout(0.3)(inp, training=True)
x = Dense(1)(x)
m = Model(inp,x)
# m.compile(...)
# m.fit(...)

X = np.random.uniform(0,1, (1,10))

output = []
for i in range(0,100):
    output.append(m.predict(X)) # always different

by default, training is set to False

HERE a full example of the usage of droput at inference time

Answer 4

As newer Tensorflow versions are usually eager, you can try things like:

train_prediction = model_or_layer(input_data_numpy, training=True)
test_prediction = model_or_layer(input_data_numpy, training=False)

This will give you predictions considering the behavior of the desired phase. For custom training loops (where instead of model.fit you make the eager predictions and apply the gradients yourself), it's important to use this:

#training
for batchX, batchY in data_generator:
    with with tf.GradientTape() as tape:
       train_outputs = model(batchX, training=True)
       loss = some_loss(batchY, train_outputs)
    gradients = tape.gradient(loss, model.trainable_weights)
    optimizer.apply_gradients(zip(gradients, model.trainable_weights))

#validating
for batchX, batchY in val_generator:
    val_outputs = model(batchX, training=False)
    #calculate metrics

---

I never tested the following, but in non-eager mode, you can also probably build a new model using the existing layers, but passing training=False to the call (functional API model):

#if using an existing layer
output_symbolic_tensor = someExistingLayer(input_symbolic_tensor, training=True) 

#if you're creating a new layer now
output_symbolic_tensoe = SomeLayer(*layer_params)(input_symbolic_tensor, training=True)

Answer 5

As previously stated, dropout in Keras happens only at train time (with proportionate weight adjustment during training such that learned weights are appropriate for prediction when dropout is disabled).

This is not ideal for cases in which we wish to use a dropout NNET as a probabilistic predictor (such that it produces a distribution when asked to predict the same inputs repeatedly). In other words, Keras' Dropout layer is designed to give you regularization at train time, but the "mean function" of the learned distribution when predicting.

If you want to retain dropout for prediction, you can easily implement a permanent dropout ("PermaDropout") layer (this was based on suggestions made by F. Chollet on the GitHub discussion area for Keras):

from keras.layers.core import Lambda
from keras import backend as K

def PermaDropout(rate):
    return Lambda(lambda x: K.dropout(x, level=rate))

By replacing any dropout layer in a Keras model with "PermaDropout", you'll get the probabilistic behavior in prediction as well.

# define the LSTM model
n_vocab = text_to_train.n_vocab

model = Sequential()
model.add(LSTM(n_vocab*4, 
          input_shape=input_shape, 
          return_sequences=True))
# Replace Dropout with PermaDropout
# model.add(Dropout(0.3)
model.add(PermaDropout(0.3))
model.add(LSTM(n_vocab*2))
# Replace Dropout with PermaDropout
# model.add(Dropout(0.3)
model.add(PermaDropout(0.3))
#model.add(Dense(n_vocab*2))
model.add(Dense(n_vocab, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Answer 6

Keras does this by default. In Keras dropout is disabled in test mode. You can look at the code here and see that they use the dropped input in training and the actual input while testing.

As far as I know you have to build your own training function from the layers and specify the training flag to predict with dropout (e.g. its not possible to specify a training flag for the predict functions). This is a problem in case you want to do GANs, which use the intermediate output for training and also train the network as a whole, due to a divergence between generated training images and generated test images.

Answer 7

Dropout removes certain neurons form play, and to compensate for that we usually take one of two ways.

1. scaling the activation at test time
2. inverting the dropout during the training phase

And keras uses the second form of correction as you can see here