CODEWITHSUNDEEP
machine-learningneural-networkclassificationsvmlogistic-regression
user5875020
user5875020

Reputation:

Machine learning model suggestion for large imbalance data

I have data set for classification problem. I have in total 50 classes. 

 Class1: 10,000 examples 
 Class2: 10 examples
 Class3: 5 examples 
 Class4: 35 examples
 .
 .
 . 
and so on.

I tried to train my classifier using SVM ( both linear and Gaussian kernel). My accurate is very bad on test data 65 and 72% respectively. Now I am thinking to go for a neural network. Do you have any suggestion for any machine learning model and algorithm for large imbalanced data? It would be extremely helpful to me

Upvotes: 4

Views: 1222

Answers (4)

Aashutosh sinha
Aashutosh sinha

Reputation: 21

There are multiple ways to handle with imbalanced datasets, you can try

  1. Up sampling
  2. Down Sampling
  3. Class Weights

I would suggest either Up sampling or providing class weights to balance it

https://towardsdatascience.com/5-techniques-to-work-with-imbalanced-data-in-machine-learning-80836d45d30c

You should think about your performance metric, don't use Accuracy score as your performance metric , You can use Log loss or any other suitable metric

https://machinelearningmastery.com/failure-of-accuracy-for-imbalanced-class-distributions/

Upvotes: 1

Lerner Zhang
Lerner Zhang

Reputation: 7130

Data-level methods:

Undersampling runs the risk of losing important data from removing data. Oversampling runs the risk of overfitting on training data, especially if the added copies of the minority class are replicas of existing data. Many sophisticated sampling techniques have been developed to mitigate these risks.

One such technique is two-phase learning. You first train your model on the resampled data. This resampled data can be achieved by randomly undersampling large classes until each class has only N instances. You then fine-tune your model on the original data.

Another technique is dynamic sampling: oversample the low-performing classes and undersample the high-performing classes during the training process. Introduced by Pouyanfar et al., the method aims to show the model less of what it has already learned and more of what it has not.

Algorithm-level methods

  1. Cost-sensitive learning
  2. Class-balanced loss
  3. Focal loss

References:

  1. esigning Machine Learning Systems
  2. Survey on deep learning with class imbalance

Upvotes: 0

Eyal Shulman
Eyal Shulman

Reputation: 749

You should provide more information about the data set features and the class distribution, this would help others to advice you. In any case, I don't think a neural network fits here as this data set is too small for it.

Assuming 50% or more of the samples are of class 1 then I would first start by looking for a classifier that differentiates between class 1 and non-class 1 samples (binary classification). This classifier should outperform a naive classifier (benchmark) which randomly chooses a classification with a prior corresponding to the training set class distribution. For example, assuming there are 1,000 samples, out of which 700 are of class 1, then the benchmark classifier would classify a new sample as class 1 in a probability of 700/1,000=0.7 (like an unfair coin toss).

Once you found a classifier with good accuracy, the next phase can be classifying the non-class 1 classified samples as one of the other 49 classes, assuming these classes are more balanced then I would start with RF, NB and KNN.

Upvotes: 3

Marcin Możejko
Marcin Możejko

Reputation: 40516

From my experience the most successful ways to deal with unbalanced classes are :

  1. Changing distribiution of inputs: 20000 samples (the approximate number of examples which you have) is not a big number so you could change your dataset distribiution simply by using every sample from less frequent classes multiple times. Depending on a number of classes you could set the number of examples from them to e.g. 6000 or 8000 each in your training set. In this case remember to not change distribiution on test and validation set.

  2. Increase the time of training: in case of neural networks, when changing distribiution of your input is impossible I strongly advise you trying to learn network for quite a long time (e.g. 1000 epochs). In this case you have to remember about regularisation. I usually use dropout and l2 weight regulariser with their parameters learnt by random search algorithm.

  3. Reduce the batch size: In neural networks case reducing a batch size might lead to improving performance on less frequent classes.

  4. Change your loss function: using MAPE insted of Crossentropy may also improve accuracy on less frequent classes.

Feel invited to test different combinations of approaches shown by e.g. random search algorithm.

Upvotes: 0

Related Questions