CODEWITHSUNDEEP
python-3.xkerasjupyter-notebook
avsr
avsr

Reputation: 143

Is it ok to run same model multiple times?

My Question is I ran a keras model for 100 epochs(gave epochs=100) and stopped for some time for cooling purpose of CPU and GPU. Than I ran 100 epochs again and loss is decreasing from where it has stopped in the previous 100 epochs . Is it works in all conditions. Like if there are on 1000 epochs I want to train my model, can I stop after every 100 epochs and wait until my cpu and GPU cools and run the next 100 epochs. Can I do this?

Upvotes: 2

Views: 524

Answers (1)

Simon Delecourt
Simon Delecourt

Reputation: 1599

It will not work in all conditions. For examples if you shuffle the data and perform a validation split like this :

fit(x,y,epochs=1, verbose=1, validation_split=0.2, shuffle=True)

You will use the entire dataset for training which is not what you expect.

Furthermore, by doing multiple fit you will erase history information (accuracy, loss, etc at each epoch), given by :

model.history

So some callback functions that use this history will not work properly, like EarlyStopping (source code here).

Otherwise, it works as it does not mess around with the keras optimizer as you can see in the source code of keras optimizers (Adadelta optimizer).

However, I do not recommend to do this. Because it could cause bugs in future development. A cleaner way to do that would be to create a custom callback function with a delay like this :

import time

class DelayCallback(keras.callbacks.Callback):

    def __init__(self,delay_value=10, epoch_to_complete=10):
        self.delay_value = delay_value # in second
        self.epoch_to_complete = epoch_to_complete 


    def on_epoch_begin(self, epoch, logs={}):
        if (epoch+1) % self.epoch_to_complete == 0:
            print("cooling down")
            time.sleep(self.delay_value) 
        return


model.fit(x_train, y_train,
          batch_size=32,
          epochs=20,
          verbose=1, callbacks=[DelayCallback()])

Upvotes: 3

Related Questions