CODEWITHSUNDEEP
pythonscikit-learnadaboost
배상일
배상일

Reputation: 107

Why's there a difference in prediction result between AdaBoost with n_estimators=1 that uses SVC as a base estimator, and just SVC

I am currently using daily financial data to fit my SVM and AdaBoost. To check my result, I tried AdaBoost with n_estimators=1 so that it would return same result as I just run a single SVM.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.svm import SVC


svm2 = SVC(C=box_const, kernel='rbf', degree=3, gamma='scale', coef0=0.0,
          shrinking=True, tol=0.001, cache_size=1000, class_weight='balanced',
          verbose=False, max_iter=-1, decision_function_shape='ovr', probability=True)
model2 = AdaBoostClassifier(base_estimator=svm2,
                           n_estimators=1,
                           algorithm='SAMME.R')


model2.fit(X_train, y_train)
svm2.fit(X_train, y_train)

However, on the contrary, I found that even though I set n_estimators=1, they produced different prediction result. Have I done something wrong? Or is there a specific reason to this result?

>>> model2.predict(X_test)
array([1., 1., 1., 1., 1.])
>>> model2.base_estimator
SVC(C=1, cache_size=1000, class_weight='balanced', probability=True)

>>> svm2.predict(X_test)
array([0., 1., 1., 0., 0.])
>>> svm2
SVC(C=1, cache_size=1000, class_weight='balanced', probability=True)

[Edit] I've found out that there is a significant difference regarding the ways I add sample_weight to the scikit learn's SVC.

When I define my model as such

svm2 = SVC(C=box_const, kernel='rbf', degree=3, gamma='scale', coef0=0.0,
          shrinking=True, tol=0.001, cache_size=1000, class_weight='balanced',
          verbose=False, max_iter=-1, decision_function_shape='ovr', probability=True)

These two yields the same prediction results

svm2.fit(X, y, sample_weight=[1] * len(X))
svm2.fit(X, y)

while

svm2.fit(X, y, sample_weight=[1 / len(X)] * len(X))

yields different results. I believe that since AdaBoost initializes the sample weights with 1 / len(X), this kind of problem occurs. Have I done something wrong in inserting sample weights to SVM?

Upvotes: 2

Views: 339

Answers (2)

Ben Reiniger
Ben Reiniger

Reputation: 12602

This was answered over at stats.SE a few years ago:
Sample weights scaling in sklearn.svm.SVC
See also the documentation:

sample_weight : array-like of shape (n_samples,), default=None

Per-sample weights. Rescale C per sample. Higher weights force the classifier to put more emphasis on these points.

To summarize, sample weights are incorporated into the SVM algorithm in sklearn by modifying the regularization parameter C. You can compensate for AdaBoost by increasing the parameter C in your underlying SVC.

Upvotes: 1

Shaikh Abuzar
Shaikh Abuzar

Reputation: 101

You haven't done anything wrong. The classifier sets a new random state every time you run it. To fix that just set the random_state parameter to any value you like.

Eg:

AdaBoostClassifier(base_estimator=svm2,
                     n_estimators=1,
                     algorithm='SAMME.R',
                     random_state=21 
                   )

For more details check the official documentation

Upvotes: 0

Related Questions