CODEWITHSUNDEEP
pythontensorflowkeras
rindis
rindis

Reputation: 1087

Issue fitting multiple keras Sequential models in the same python script

Please see gist here for the full code. In this question I've pasted what I regard the important part of the code at the bottom.

First I generate a dataset as shown in [2], which consists of 2 features (x and y) and 3 labels (grey, blue and orange). Then I make 4 keras sequential models with identical layer structure, optimizer, loss function, etc. Lastly, I call fit on each model and plot the resulting metrics shown in [3]. As you can see the models all perform differently and I'm wondering why? I've locked the random seed value so each time I run this script I get the exact same result.

In this example the models have identical structures, so I would expect the metric plots to be identical. Eventually I would like to vary the number of layers, layer size, loss function, etc. between the models to see its effects, but that does not seem feasible in this setup. Am I approaching this incorrectly?

An interesting thing to note is that by setting the batch_size to 32 this effect is not as prominent, but it is still present and reproducible (See [4]).

Data

Metrics

Metrics with batch_size=32

# ---- MAKE MODELS ---- #
NUMBER_OF_MODELS = 4
models = []
for i in range(NUMBER_OF_MODELS):
    model = keras.models.Sequential(name=f'{i}')

    model.add(keras.layers.Dense(8, activation='relu', input_shape=df_train['features'].values.shape[-1:]))
    model.add(keras.layers.Dense(3, activation='softmax'))

    model.compile(optimizer=keras.optimizers.Adam(),
                  loss=keras.losses.CategoricalCrossentropy(),
                  metrics=[keras.metrics.CategoricalAccuracy()])

    model.summary()
    models.append(model)
# --------------------- #

# ---- TRAIN MODELS ---- #
histories = []
for model in models:
    with tf.device('/cpu:0'):
        history = model.fit(x=df_train['features'].values, y=df_train['labels'].values,
                            validation_data=(df_val['features'].values, df_val['labels'].values),
                            batch_size=512, epochs=100, verbose=0)
        histories.append(history)
# ---------------------- #

Upvotes: 1

Views: 313

Answers (1)

Marco Cerliani
Marco Cerliani

Reputation: 22031

you simply need to set the seed every time you define and fit the model

following your code I collapse all in these lines:

NUMBER_OF_MODELS = 4
models = []
histories = []

for i in range(NUMBER_OF_MODELS):

    set_seed_TF2(33)

    model = keras.models.Sequential(name=f'{i}')

    model.add(keras.layers.Dense(8, activation='relu', input_shape=df_train['features'].values.shape[-1:]))
    model.add(keras.layers.Dense(3, activation='softmax'))

    model.compile(optimizer=keras.optimizers.Adam(),
                  loss=keras.losses.CategoricalCrossentropy(),
                  metrics=[keras.metrics.CategoricalAccuracy()])
    
    with tf.device('/cpu:0'):
        
        history = model.fit(x=df_train['features'].values, y=df_train['labels'].values,
                            validation_data=(df_val['features'].values, df_val['labels'].values),
                            batch_size=512, epochs=100, verbose=0)
        histories.append(history)
        
    models.append(model)

the magic function is set_seed_TF2

def set_seed_TF2(seed):
    
    tf.random.set_seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    random.seed(seed)

that must be call every time you initialize the model and fit

with this in mind, you can produce every time the same metrics/predictions:

enter image description here

here the running notebook: https://colab.research.google.com/drive/1nHEDI6d3LsRPQUXGiTOYfNOfw954Pu-H?usp=sharing

this works for CPU only

Upvotes: 1

Related Questions