CODEWITHSUNDEEP
pythontensorflowmachine-learningkerasneural-network
Panri93
Panri93

Reputation: 213

Finding the optimum learning rate & epochs in a Neural Network

I have created a single layer neural network with two outputs (one for each class, 0 or 1) trained with the sigmoid method and SGD optimizer. I have also trained the NN without any hidden layer. Furthermore I have validated the performance of the model using StratifiedKFold with 4 splits. The model trained is designed with a lr=0.1 and epochs=150, however, I don´t know if these values are optimising the model. For this reason, I would like to run 20 combinations of the learning rate parameter and epochs to see the most accurate result and for which combination of these parameters I´m obtaining it. Below the restrictions:

Please, see below the code:

from sklearn.model_selection import StratifiedKFold
from keras.models import Sequential
from keras.layers.core import Dense
from keras import layers
from keras.optimizers import SGD
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score

#Function to create the NN model
def create_model():    
    #Neural Network model
    ann = Sequential()
    #Number of columns of training dataset
    n_cols = x_train.shape[1]
    #Output
    ann.add(Dense(units=1,activation='sigmoid',input_shape=(n_cols,)))
    #SGD Optimizer
    sgd = SGD(lr=0.1)
    #Compile with SGD optimizer and binary_crossentropy
    ann.compile(optimizer=sgd,
                loss='binary_crossentropy',
                metrics=['accuracy'])
    return ann

#Creating the model
model=KerasClassifier(build_fn=create_model,epochs=150,batch_size=10,verbose=0)
#Evaluating the model using StratifiedKFold
kfold = StratifiedKFold(n_splits=4, shuffle=True, random_state=2)
results=cross_val_score(model,x_train,y_train,cv=kfold)
#Accuracies
print(results)

To create the 20 combinations formed by the learning rate and epochs, firstly, I have created random values of lr and epochs:

   #Epochs
   epo = np.random.randint(10,150)
   #Learning Rate
   learn = np.random.randint(0.01,1)

My problem is that I don´t know how to fit this into the code of the NN in order to find which is the combination that gives the best accuracy of the model.

Upvotes: 0

Views: 1655

Answers (2)

Geeocode
Geeocode

Reputation: 5797

First in the create_model() function you defined the optimizer you passed the learning rate as parameter to:

#SGD Optimizer
sgd = SGD(lr=0.1)

This is the starting learning rate of the optimization process, from this point the optimizer handles the optimal learning rate. Nevertheless you can pass several starting learning rate inside a loop calling repeatedly the create_model() function and passsing a learning rate parameter to that.

Furthermore as parsa mentioned choosing the right epoch number is based on the validation result, which shows where your model became overfitted. There is the point is, where the epoch number reached its optimum.

Upvotes: 0

parsa
parsa

Reputation: 370

there is no need to optimize the number of epochs you could easily use early stop which will stop when there is no improvement in your loss or accuracy so just set your epochs a big number (for example 300 ) and add:

keras.callbacks.callbacks.EarlyStopping(monitor='val_loss', min_delta=0.1)

you also can call the best weights (just before the model started to overfit) by:

restore_best_weights=True

Upvotes: 1

Related Questions