CODEWITHSUNDEEP
pythontensorflowkerasconv-neural-network
MRM
MRM

Reputation: 1179

How to save Keras model progress into a file?

I have a neural network model coded in Keras.

When I run it on my laptop, I have the following output that shows the progress of the model:

Train on 4 samples, validate on 1 samples Epoch 1/1 4/4 [==============================] - 22s 5s/step - loss: 0.2477 - val_loss: 0.2672

However, when I submit this code to a cluster for running, I do not know how many epochs are left, so I would like to save the above output into a file while the model is running.

How can I do that?

Upvotes: 6

Views: 2157

Answers (2)

ebeneditos
ebeneditos

Reputation: 2612

You can do it with a for loop and model.save:

import os

PATH_TO_MODELS = 'path to models directory'

TOTAL_EPOCHS = 8 # number of epochs you want to save

for epoch in range(TOTAL_EPOCHS):
    model.fit(..., epochs = 1)
    save_name = 'model_%sepochs.h5' % str(epoch)
    model.save(os.path.join(PATH_TO_MODELS, save_name))

This will save the output model from each epoch into a distinct file, so that you can keep track of them.

Upvotes: 1

Chris Farr
Chris Farr

Reputation: 3789

At least one way is to add the remote monitor call back and actually view it in real time. I haven't played with this yet but I know it exists and have wanted to.

keras.callbacks.RemoteMonitor(root='http://localhost:9000', path='/publish/epoch/end/', field='data', headers=None, send_as_json=False)

You can find the documentation here.

Another option is to use TensorBoard. Keras also has a callback for that in the link I already provided. Here is the information from TensorFlow.

Upvotes: 1

Related Questions