CODEWITHSUNDEEP
python-3.xmachine-learningdeep-learningstatistics
ram raj
ram raj

Reputation: 21

How to avoid over fitting?

I have a situation where:

  1. My training accuracy is 93%
  2. CV accuracy is 55%
  3. Test accuracy is 57%

I think this is a classical case of overfitting.

As per my knowledge, I can use regularization. I have read cross validation will also helps in solving my over fitting problem.

Some inquiries I have regarding this:

  1. Whether cross validation is used only for hyperparameter tuning, or will it have a role in solving over fitting problem?
  2. If cross validation solves overfitting problems, how?
  3. Whether cross validation is used only as a check to see whether the model is over fitting or not?

Upvotes: 1

Views: 901

Answers (1)

Andrew
Andrew

Reputation: 501

I think you are confused on what exactly cross validation is. I will link to OpenML's explanation for 10-fold cross validation so you get a better idea.

Over-fitting occurs normally when there is not enough data for your model to train on, resulting in it learning patterns/similarities between the data set that is not helpful, such as putting too much focus on outlying data that would be ignored if given a larger data set.

Now to your questions:

1-2. Cross-validation is just one solution that is helpful for preventing/solving over-fitting. Through partitioning the data set into k-sub groups, or folds, you then can train your model on k-1 folds. The last fold will be used as your unseen validation data to test your model upon. This will sometimes help prevent over-fitting. A factor in this working though depends on how long/how many epochs you are training your data for. Since you said you have a relatively small data set, you want to make sure you aren't 'over-learning' on this data. Implementing cross-validation will not do you much good if you are training for hundreds/thousands of epochs on a really small data set.

  1. Cross-validation doesn't tell you if your data is being over-fitted. It may give you hints that it is if your results are vastly different after several times running the program, but it is not going to be clear cut.

The biggest problem, and you said it yourself in the comments, is you don't have a lot of data. The best, although not always the easiest way, is to increase your data size so your model won't learn unimportant tendencies and put too much focus on the outliers.

I will link to a website that is incredibly helpful in explaining the problems of over-fitting and gives a variety of ways to attempt to overcome this problem.

Let me know if I was of help!

Upvotes: 1

Related Questions