CODEWITHSUNDEEP
c#machine-learningdecision-treeadaboostboosting
H_Lev1
H_Lev1

Reputation: 253

AdaBoost repeatedly chooses same weak learners

I have implemented a version of the AdaBoost boosting algorithm, where I use decision stumps as weak learners. However often I find that after training the AdaBoost algorithm, a series of weak learners is created, such that this series is recurring in the whole set. For example, after training, the set of weak learners looks like A,B,C,D,E,D,E,D,E,D,E,F,E,D,E,D,E etc.

I believe I am updating the weights of the data properly after each assignment of a new weak learner. Here I classify each data point and then set the weight of this data point.

// After we have chosen the weak learner which reduces the weighted sum error by the most, we need to update the weights of each data point.
double sumWeights = 0.0f; // This is our normalisation value so we can normalise the weights after we have finished updating them
foreach (DataPoint dataP in trainData) {
      int y = dataP.getY(); // Where Y is the desired output
      Object[] x = dataP.getX();
      // Classify the data input using the weak learner. Then check to see if this classification is correct/incorrect and adjust the weights accordingly.
      int classified = newLearner.classify(x);
      dataP.updateWeight(y, finalLearners[algorithmIt].getAlpha(), classified);
      sumWeights += dataP.getWeight();

}

Here is my classify method in the WeakLearner class

// Method in the WeakLearner class
public int classify(Object[] xs) {
            if (xs[splitFeature].Equals(splitValue))
                return 1;
            else return -1;
}

Then I have a method which updates the weight of a DataPoint

public void updateWeight(int y, double alpha, int classified) {
            weight = (weight * (Math.Pow(e, (-y * alpha * classified))));
}

And I'm not sure why this is happening, are there any common factors why the same weak learners would generally be chosen?

Upvotes: 3

Views: 943

Answers (1)

Faraz Gerrard Jamal
Faraz Gerrard Jamal

Reputation: 248

You could increase the value of alpha and check. Maybe, not enough weight is being given to the misclassified samples, hence ,they are showing up again and again.

Upvotes: 0

Related Questions