CODEWITHSUNDEEP
pythonneural-networkdeep-learning
user9761720
user9761720

Reputation:

Validation loss increasing

I am trying to solve dogs vs cats problem in kaggle with my numpy neural network.

Validation loss

Validation loss graph

Training loss

training loss graph

My training loss decreases well but the validation loss increases, so my model is definitely overfitting. I used two hidden layers with size 125, 50. I used learning rate of 0.075 ran the model with 600 iterations.

I also tried using regularization with lambda = 0.01 or 0.03, but still didn't help.

Any solutions to this problem?

Upvotes: 6

Views: 22292

Answers (1)

Bastian
Bastian

Reputation: 1593

Add Dropout to each layer with a dropout-probability of .5. See how it influences the validation error and if your training error won’t go deeper than a specific point, either decrease the probability to .3 or sequentially remove it from the first layers. It’s a bit of try-and-error here.

I guess you mean L2 regularization (weight-decay) with lamda. Nowadays, nets usually use dropout and a very small L2 like 0.0005. The L2 causes the weights to be close to zero and prevents them from exploding. Dropout is a better regulizer because it randomly drops nodes during training which causes some sort of specialization.

Good luck.

Upvotes: 10

Related Questions