CODEWITHSUNDEEP
pythontensorflowkeras
Toomas-Siim Teresk
Toomas-Siim Teresk

Reputation: 35

Keras LSTM Continue training after save

i'm working on a LSTM model and i'd like to save it and continue later with extra data as it accumulates. My problem is that after save the model and load it again next time i run the script, the prediction is completely wrong, it just mimics the data i entered into it.

Here's the model initialization:

# create and fit the LSTM network
if retrain == 1:
    print "Creating a newly retrained network."
    model = Sequential()
    model.add(LSTM(inputDimension, input_shape=(1, inputDimension)))
    model.add(Dense(inputDimension, activation='relu'))
    model.compile(loss='mean_squared_error', optimizer='adam')
    model.fit(trainX, trainY, epochs=epochs, batch_size=batch_size, verbose=2)
    model.save("model.{}.h5".format(interval))
else:
    print "Using an existing network."
    model = load_model("model.{}.h5".format(interval))
    model.compile(loss='mean_squared_error', optimizer='adam')
    model.fit(trainX, trainY, epochs=epochs, batch_size=batch_size, verbose=2)
    model.save("model.{}.h5".format(interval))
    del model
    model = load_model("model.{}.h5".format(interval))
    model.compile(loss='mean_squared_error', optimizer='adam')

The first dataset, when retrain is set to 1, is around 10 000 entries with around 3k epoch and 5% batch size. The second dataset is a single entry data. as in one row, with again 3k epochs and batch_size=1

Solved

I was reloading the scaler incorrectly:

scaler = joblib.load('scaler.{}.data'.format(interval))
dataset = scaler.fit_transform(dataset)

Correct:

scaler = joblib.load('scaler.{}.data'.format(interval))
dataset = scaler.transform(dataset)

fit_transform recalculates the multipliers for the scaled values, that means there will be an offset from the original data.

Upvotes: 3

Views: 1615

Answers (1)

dennis-w
dennis-w

Reputation: 2156

From the functional keras model api for model.fit():

initial_epoch: Integer. Epoch at which to start training (useful for resuming a previous training run).

Setting this parameter might solve your problem.

I think the source of the problem is the adaptive learning rate from adam. During training the learning rate anturally declines for more finetuning of the model. When you retrain your model with only one sample the weight updates are much too big (because of the resetted learning rate) which can totally destroy your previous weights.

If initial_epoch is not good, than try to start your second training with a lower learning rate.

Upvotes: 1

Related Questions