CODEWITHSUNDEEP
pythonmachine-learningkerasneural-network
glaey
glaey

Reputation: 35

Cross-validation: cannot clear the model with clear_session() to train a new model

I am trying to evaluate a training for a Named Entity Recognition task with cross-validation. The issue is the keras model is not cleared properly to perform the other iteration of the cross-validation.

I have tried several solutions that were suggested in stack, github or forums to clear the model but none work. I'll focus on the the two that seem to be close to the canonical solution.

1st one: I create the model (architecture+compile) outside the loop of the cross-validation but the call of keras.backend.clear_session() inside the loop. I got the error: " tensorflow.python.framework.errors_impl.InvalidArgumentError: Tensor words_input:0, specified in either feed_devices or fetch_devices was not found in the Graph "

2nd one: I create the model inside the loop and call keras.backend.clear_session() inside too. I got the error: " Duplicate node name in graph: 'training/Nadam/Pow' "

I am sorry that the code may not be clear enough but I cannot show it for privacy issues

1st

kf = KFold(folds, shuffle=True)
model = create_model(...)
for i, (train_idx, test_idx) in enumerate(kf.split(...)):
   ...
   train_batch, train_batch_len = createBatches(train_set)
   test_batch, test_batch_len = createBatches(test_set)
   model, _, _= training(model, ..., epochs=epochs)
   clear_session()

2nd

kf = KFold(folds, shuffle=True)
for i, (train_idx, test_idx) in enumerate(kf.split(...)):
   model = create_model(...)
   ...
   train_batch, train_batch_len = createBatches(train_set)
   test_batch, test_batch_len = createBatches(test_set)
   model, _, _= training(model, ..., epochs=epochs)
   clear_session()

inside create_model(), we define the architecture of the layers and the call compile(...)

I do want to perform a cross-validation with a clear model at each iteration of the kfold. Another thing to mention was I tried to just set the initial weights saved at the beginning at each iteration but as I work on hundreds epochs it seems to have storage issue-since what I understood that keras creates nodes at each epochs. In doing this if I use a small amount of epochs it works fine for the whole kfold but when i performed on hundred epochs the training performance remains at 0 F1-score after the second iteration.

Upvotes: 2

Views: 1157

Answers (1)

Ioannis Nasios
Ioannis Nasios

Reputation: 8527

You have to also delete the model before clearing season

from keras import backend as K
del model
K.clear_session()
gc.collect()

Upvotes: 1

Related Questions