Retrain your CNN model successifely with two different datasets

Question

I had implemented a CNN with 3 Convolutional layers with Maxpooling and dropout after each layer I had noticed that when I trained the model for the first time it gave me 88% as testing accuracy but after retraining it for the second time successively, with the same training dataset it gave me 92% as testing accuracy.

I could not understand this behavior, is it possible that the model had overfitting in the second training process?

Thank you in advance for any help!

Answer 1

Well I am no expert when it comes to machine learning but I do know the math behind it. What you are doing when you train a neural network you basicly find the local minima to the loss function. What this means is that the end result will heavily depend on the initial guess of all of the internal varaibles.

Usually the variables are randomized as a initial estimation and you could therefore reach quite different results from running the training process multiple times.

That being said, from when I studied the subject I was told that you usually reach similar regardless of the initial guess of the parameters. However it is hard to say if 0.88 and 0.92 would be considered similar or not.

Hope this gives a somewhat possible answer to your question.

As mentioned in another answer, you could remove the randomization, both in the parameter initialization of the parameters and the randomization of the data used for each epoch of training by introducing a seed. This would insure that when you run it twice, everything will get "randomized" in the exact same order. In tensorflow this is done using for example tf.random.set_seed(1), the number 1 can be changed to any number to get a new seed.

Answer 2

It is quite possible if you have not provided the seed number set.seed( ) in the R language or tf.random.set_seed(any_no.) in python