CODEWITHSUNDEEP
pythonscikit-learnlogistic-regressionimbalanced-data
Crunchy Shark
Crunchy Shark

Reputation: 13

Why the equivalent class_weights for Logistic Regression in sklearn generates different outcomes?

When working on the imbalanced dataset, I found an interesting question about LogisticRegression in scikit-learn.

For the parameter class_weight, if I send {1:0.5, 0:0.5}, I will get a different outcome with {1:1, 0:1} even though they are actually the same weights mathematically.

Here is what I got,

import numpy as np
from sklearn.linear_model import LogisticRegression
np.random.seed(1)

def sigmoid(x):
    return 1/(np.exp(-x)+1)

x1 = np.random.normal(0, 4, 100000)
x2 = np.random.normal(0, 1, 100000)
X = np.array([x1, x2]).T

proba = sigmoid(0.1 + 2*x1 + 3*x2)
y = np.random.binomial(1, proba)

lr1 = LogisticRegression(C=1, class_weight = {1:0.5, 0:0.5}).fit(X, y)
print(lr1.score(X,y)) # 0.93656

lr2 = LogisticRegression(C=1, class_weight = {0:1, 1:1}).fit(X, y)
print(lr2.score(X,y)) # 0.93653

Upvotes: 1

Views: 188

Answers (2)

StupidWolf
StupidWolf

Reputation: 46898

For LogisticRegression, the default sets penalty='l2' . See help page . You will only get the same outcome for equivalent weights if penalty='none' :

lr1 = LogisticRegression(C=1, class_weight = {1:0.5, 0:0.5} , penalty='none').fit(X, y)
print(lr1.score(X,y)) 

lr2 = LogisticRegression(C=1, class_weight = {0:1, 1:1},penalty='none').fit(X, y)
print(lr2.score(X,y)) 

0.93652
0.93652

As touched upon in this post and also the above mentioned LogisticRegression help page, a larger regularization (or smaller C) will make the coefficients (and results) more similar:

lr1 = LogisticRegression(C=100, class_weight = {1:0.5, 0:0.5} , penalty='l2').fit(X, y)
print(lr1.coef_)

lr2 = LogisticRegression(C=100, class_weight = {0:1, 1:1},penalty='l2').fit(X, y)
print(lr2.coef_)

[[2.00034043 2.98401278]]
[[2.00035828 2.98404571]]

Compared to :

lr1 = LogisticRegression(C=0.1, class_weight = {1:0.5, 0:0.5} , penalty='l2').fit(X, y)
print(lr1.coef_)

lr2 = LogisticRegression(C=0.1, class_weight = {0:1, 1:1},penalty='l2').fit(X, y)
print(lr2.coef_)

[[1.96628865 2.9210898 ]]
[[1.98293929 2.95188187]]

Upvotes: 0

lejlot
lejlot

Reputation: 66795

The way class_weight is implemented is that it affects sample_weight, and these, on the other hand multiply the loss. Unfortunately they do not affect the regulariser, so its relative strength changes

lr2 = LogisticRegression(C=0.5, class_weight = {0:1, 1:1}).fit(X, y)

will give you desired

print(lr2.score(X,y)) # 0.93656

and analogously

lr2 = LogisticRegression(C=0.25, class_weight = {0:2, 1:2}).fit(X, y)
print(lr2.score(X,y)) # 0.93656

So in general 1/C (regularisation strength) should be equal to your total sum of weights reweighting, since vaguely it is imlpemented as

LOSS := 1/C ||w||^2 + SUM_i sample_weight_i loss(pred(x_i), y_i)

Upvotes: 1

Related Questions