CODEWITHSUNDEEP
pythontensorflowkerasreproducible-research
Dmitry Toda
Dmitry Toda

Reputation: 43

How to get reproducible weights initializaiton in Keras?

  1. I set both numpy and tensorflow random seeds as suggested
  2. Generate some data - this part is reproducible, gives same results always
  3. Create a simple network and make a prediction (without training, just with random weights) - prediction is different every time
import numpy as np
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras import Sequential, optimizers
import tensorflow as tf

np.random.seed(32)
tf.set_random_seed(33)

random_data = np.random.rand(10, 2048)
print(random_data[:,0])

def make_classifier():
    model = Sequential()
    model.add(Dense(1024, activation='relu', input_dim=2048))
    model.add(Dropout(0.5))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer=optimizers.RMSprop(lr=2e-4),
              loss='binary_crossentropy')
    return model
model = make_classifier()
# model.summary()
model.predict(random_data)

When I re-run the whole cell, the print statement always outputs [0.85888927 0.23846818 0.17757634 0.07244977 0.71119893 0.09223853 0.86074647 0.31838194 0.7568638 0.38197083]. However the prediction is different every time:

array([[0.5825965 ],
       [0.8677979 ],
       [0.70151913],
       [0.64572096],
       [0.78101623],
       [0.76483005],
       [0.7946336 ],
       [0.6281612 ],
       [0.8208673 ],
       [0.8273002 ]], dtype=float32)

array([[0.51012236],
       [0.6562015 ],
       [0.5593666 ],
       [0.686155  ],
       [0.6488372 ],
       [0.5966359 ],
       [0.6236731 ],
       [0.58099884],
       [0.68447435],
       [0.58886844]], dtype=float32)

And so on.

  1. How do I get reproducible prediction for a just-initialized network?
  2. When does the weights initialization happen exactly? when I compile the model or?..

Upvotes: 4

Views: 3722

Answers (4)

Jason De Melo
Jason De Melo

Reputation: 31

I was struggling with that too. The version of Tensorflow and Keras is 2.14.0.

What worked for me was:

seed = 0
np.random.seed(seed)
tf.random.set_seed(seed)
random.seed(seed)

I hope this helps!

Upvotes: 1

Litchy
Litchy

Reputation: 673

I think a better practice is

tf.keras.utils.set_random_seed(
    seed
)

ref: https://www.tensorflow.org/api_docs/python/tf/keras/utils/set_random_seed

Upvotes: 1

Lucas Azevedo
Lucas Azevedo

Reputation: 2370

I've been struggling a lot with this and turns out there are quite a few points that have to be set in order to achieve complete consistency for every case:

First of all, make sure the data (and the order of the data) that you feed to your model is consistent. Then, for the model weights initialization:

1)numpy random seed

import numpy as np
np.seed(1)

2)tensor flow random seed

import tensorflow as tf
tf.set_random_seed(2)

3)python random seed

import random
random.seed(3)

Additionally to that, you have to set two (if you have multiprocessing capabilities) arguments to model.fit. These ones are not often mentioned on the answers I've seen around:

model.fit(..., shuffle=False, use_multiprocessing=False)

Only then I have achieved complete consistency in training runs.

Hope that helps people!

Upvotes: 4

Nicolas Gervais
Nicolas Gervais

Reputation: 36684

tf.keras.initializers objects have a seed argument for reproducible initialization.

import tensorflow as tf
import numpy as np

initializer = tf.keras.initializers.GlorotUniform(seed=42)

for _ in range(10):
    print(np.round(initializer((4,)), 3))

[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]

In a Keras layer, you can use it like this:

tf.keras.layers.Dense(1024, 
                      activation='relu', 
                      input_dim=2048,
                      kernel_initializer=tf.keras.initializers.GlorotUniform(seed=42))

Upvotes: 3

Related Questions