CODEWITHSUNDEEP
deep-learningcross-validationobjective-function
Skonitsa
Skonitsa

Reputation: 106

Deep learning, Loss does not decrease

I have tried to finetune a pretrained model using a training set that has 20 classes. The important thing to mention is that even though I have 20 classes, one class consist the 1/3 of the training images. Is that a reason that my loss does not decrease and testing accuracy is almost 30%?

Thank you for any advise

Upvotes: 2

Views: 6183

Answers (2)

Rose Yu
Rose Yu

Reputation: 41

I had similar problem. I resolved it by increasing the variance of the initial values for the neural network weights. This serves as pre-conditioning for the neural network, to prevent the weights from dying out during back-prop.

I came across neural network lectures from Prof. Jenny Orr's course and found it very informative. (Just realized that Jenny co-authored many papers with Yann LeCun and Leon bottou in the early years on neural network training).

Hope it helps!

Upvotes: 4

Indie AI
Indie AI

Reputation: 601

Yes it is very possible that your net is overfitting to the unbalanced labels. One solution is you can perform data augmentation on the other labels to balance them out. For example, if you have image data: you can do random crops, take horizontal/vertical flips, a variety of techniques.

Edit:

One way to check if you are overfitting to the unbalanced labels is to compute a histogram of your nets predicted labels. If it's highly skewed towards the unbalanced class, you should try the above data augmentation method and retrain your net and see if that helps.

Upvotes: 1

Related Questions