CODEWITHSUNDEEP
pythonscikit-learn
Joel2342
Joel2342

Reputation: 43

Improve speed of scikit-learn multinomial logistic regression

i am trying to train a logistic regression model in scikit learn and it is taking very long to train, around 2 hours. The size of the dataset is 21613 x 19. I am new to scikit learn, as such i dont know whether my code is wrong or that it just takes very long to train. Any suggestion on how to improve the training speed would be very much appreciated!

code used to train is below

# get the LogisticRegression estimator
from sklearn.linear_model import LogisticRegression

# training the model
# apply algorithm  to data using fit()
clf = LogisticRegression(solver='newton-cg',multi_class='multinomial')
clf.fit(X_train,y_train)

Upvotes: 0

Views: 3008

Answers (3)

Michael James Kali Galarnyk
Michael James Kali Galarnyk

Reputation: 3653

You might want to change your solver. The documentation says that scikit-learn has 5 different solvers: 'sag', 'saga', 'newton-cg', 'lbfgs', and 'liblinear' (not suitable for multinomial).

import time
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# Set training and validation sets
X, y = make_classification(n_samples=1000000, n_features=19, n_classes = 8, n_informative=8)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=1000)

# Solvers
solvers = ['newton-cg', 'sag', 'saga', 'lbfgs']

for sol in solvers: 
    start = time.time()
    logreg = LogisticRegression(solver=sol, multi_class='multinomial')
    logreg.fit(X_train, y_train)
    end = time.time()
    print(sol + " Fit Time: ",end-start)

Output (from 16GB RAM 8 Core Macbook): enter image description here

Choosing the right solver for a problem can save a lot of time (code adapted from here). To determine which solver is right for your problem, you can check out the table from the documentation to learn more (notice that 'newton-cg' is not faster for large datasets).

enter image description here

Upvotes: 1

Nicolas Gervais
Nicolas Gervais

Reputation: 36604

If you have a specific reason for using this solver, one thing you can do is parallelize the computations by setting the n_jobs=-1 argument.

If you're open to using other solvers, you can use faster solvers with a one-versus-rest strategy. For instance:

clf = LogisticRegression(solver='liblinear', multi_class='ovr')

It's all in the documentation, which can help you guide your choice of solver:

solver : {‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’}, default=’lbfgs’

Algorithm to use in the optimization problem.

  • For small datasets, ‘liblinear’ is a good choice, whereas ‘sag’ and ‘saga’ are faster for large ones.

  • For multiclass problems, only ‘newton-cg’, ‘sag’, ‘saga’ and ‘lbfgs’ handle multinomial loss; ‘liblinear’ is limited to one-versus-rest schemes.

  • ‘newton-cg’, ‘lbfgs’, ‘sag’ and ‘saga’ handle L2 or no penalty

  • ‘liblinear’ and ‘saga’ also handle L1 penalty

  • ‘saga’ also supports ‘elasticnet’ penalty

  • ‘liblinear’ does not support setting penalty='none'

Upvotes: 2

ajay sagar
ajay sagar

Reputation: 199

It's probably that slow because of the solver you have chosen. The newton-cg is a newton method. It's slow for large datasets because it computes the second derivatives. Use a different solver like sag or saga, they are fast for big datasets.

Upvotes: 1

Related Questions