Keras model prediction gives opposite results

Question

I trained a model called model_2 in Keras and made predictions using model.predict but I notice as I rerun the code the results are completely different. For example, first time column 0 has all probability values close to 1, but next time it has probability values all close to 0. Has it to do with the memory or the stateful parameter which I have seen mentioned in other posts?

X = df.iloc[:,1:10161]
X = X.to_numpy()                      
X = X.reshape([X.shape[0], X.shape[1],1]) 
X_train_1 = X[:,0:10080,:]
X_train_2 = X[:,10080:10160,:].reshape(17,80)

inputs_1 = keras.Input(shape=(10080, 1))
layer1 = Conv1D(64, 14)(inputs_1)
layer2 = layers.MaxPool1D(5)(layer1)
layer3 = Conv1D(64, 14)(layer2)       
layer4 = layers.GlobalMaxPooling1D()(layer3)
layer5 = layers.Dropout(0.2)(layer4)

inputs_2 = keras.Input(shape=(80,))
layer6 = layers.concatenate([layer5, inputs_2])
layer7 = Dense(128, activation='relu')(layer6)
layer8 = layers.Dropout(0.5)(layer7)
layer9 = Dense(2, activation='softmax')(layer8)

model_2 = keras.models.Model(inputs = [inputs_1, inputs_2], outputs = [layer9])

adam = keras.optimizers.Adam(lr = 0.0001)
model_2.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['acc'])

prediction = pd.DataFrame(model_2.predict([X_train_1,X_train_2]),index = df.iloc[:,0])  
pred = np.argmax(model_2.predict([X_train_1,X_train_2]), axis=1) 
display(prediction, pred)

Examples of the contradictory results:

trial 1:

    0               1
id      
11  1.131853e-07    1.000000
22  1.003963e-06    0.999999
33  1.226156e-07    1.000000
44  9.985497e-08    1.000000
55  1.234705e-07    1.000000
66  1.189311e-07    1.000000
77  6.631822e-08    1.000000
88  9.586067e-08    1.000000
99  9.494666e-08    1.000000

trial 2:

    0           1
id      
11  0.183640    0.816360
22  0.487814    0.512187
33  0.151600    0.848400
44  0.135977    0.864023
55  0.120982    0.879018
66  0.171371    0.828629
77  0.199774    0.800226
88  0.133711    0.866289
99  0.125785    0.874215

trial 3:

    0           1
id      
11  0.900128    0.099872
22  0.573520    0.426480
33  0.948409    0.051591
44  0.955184    0.044816
55  0.959075    0.040925
66  0.945758    0.054242
77  0.956582    0.043418
88  0.954180    0.045820
99  0.964601    0.035399

trial 4:

    0   1
id      
11  1.0 4.697790e-08
22  1.0 2.018885e-07
33  1.0 2.911827e-08
44  1.0 2.904826e-08
55  1.0 1.368165e-08
66  1.0 2.742492e-08
77  1.0 1.461449e-08
88  1.0 2.302636e-08
99  1.0 2.099636e-08

---

Model was been trained with:

n_folds = 10
skf = StratifiedKFold(n_splits=n_folds, shuffle=True)
skf = skf.split(X_train_1, Y_cat)

cv_score = []

for i, (train, test) in enumerate(skf):

    model_2 = my_model()

    history = model_2.fit([X_train_1[train], X_train_2[train]], Y[train], validation_data=([X_train_1[test], X_train_2[test]], Y[test]), epochs=120, batch_size=10) 
    
    result = model_2.evaluate([X_train_1[test], X_train_2[test]], Y[test])
    keras.backend.clear_session()

Answer 1

U need to fix the random generator seeds for both tensorflow and sklearn. Try the following:

tf.random.set_seed(42)

and

skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)

This will allow you to replicate your results.

Answer 2

After you train a model, you have store the weights of that model in a file. If you don't do this, you don't keep the trained models if you run your program (or any other) again.

You can store the model weights during training by using the ModelCheckpoint callback. https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint

You can load the weights before inference (prediction) using model.load_weights(...) https://www.tensorflow.org/tutorials/keras/save_and_load

To clarify why your results are random. You keep initializing a new model with random weights and then directly try to predict the answer with the untrained model.

[EDIT] The reason your untrained model keeps giving the opposite results, may be due to the inputs combined with the weight initialization. The weight initialization is random, but there is some "smarts" behind the initialization to prevent dead connections. Here you can find more details on all weight initializers: https://www.tensorflow.org/api_docs/python/tf/keras/initializers

Good Luck!

Answer 3

It's completely normal, when you create a new model, its weights are initialized as random, so the prediction will change each time you run this code.