CODEWITHSUNDEEP
pythonscikit-learnnlp
Padam Sethia
Padam Sethia

Reputation: 115

Value error when running sklearn classifier model

I'm quite new to sklearn , and I'm trying to build a simple text classifier using scikit , but running into ValueError . It shows the error at fit() , but other tutorials are using it as it is and it runs fine.

Here's my code :

from sklearn.datasets import fetch_20newsgroups
from sklearn.cross_validation import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB

news = fetch_20newsgroups(subset='all')
print len(news.data)



def train(classifier , X , y):
        X_train , y_train , X_test , y_test = train_test_split(X,y,test_size =            0.20, random_state = 33)
        classifier.fit(X_train ,y_train)
        print "Accuracy %s" % classifier.score(X_test , y_test)
        return classifier

model1 = Pipeline([('vectorizer' , TfidfVectorizer()),('classifier' , MultinomialNB()),])

train(model1 , news.data , news.target)

When running it , I'm getting a value error

Traceback (most recent call last):
  File "/home/padam/Documents/git/ticketClassifier/news.py", line 30, in <module>
    train(model1 , news.data , news.target)
  File "/home/padam/Documents/git/ticketClassifier/news.py", line 24, in train
    classifier.fit(X_train ,y_train)
  File "/usr/lib/python2.7/dist-packages/sklearn/pipeline.py", line 165, in fit
    self.steps[-1][-1].fit(Xt, y, **fit_params)
  File "/usr/lib/python2.7/dist-packages/sklearn/naive_bayes.py", line 527, in fit
    X, y = check_X_y(X, y, 'csr')
  File "/usr/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 520, in check_X_y
    check_consistent_length(X, y)
  File "/usr/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 176, in check_consistent_length
    "%s" % str(uniques))
ValueError: Found arrays with inconsistent numbers of samples: [ 3770 15076]

What does it mean by inconsistent number of samples . Other stackoverflow solutions suggest rearranging the matrix for numpy matrix . But I did not use numpy. Thanks!

Upvotes: 0

Views: 2592

Answers (1)

Vivek Kumar
Vivek Kumar

Reputation: 36599

The error is in how you are using the train_test_split.

You are using it as 

X_train , y_train , X_test , y_test = train_test_split(X, y,
                                                 test_size = 0.20, 
                                                 random_state = 33)

But the output order is different actually as given in documentation. It is:

X_train , X_test , y_train ,  y_test = train_test_split(X, y,
                                                 test_size = 0.20, 
                                                 random_state = 33)

Also, one recommendation is that if you are using scikit version >= 0.18, then change the package from cross_validation to model_selection, because its deprecated and will be removed in new versions.

So instead of:-

from sklearn.cross_validation import train_test_split

Use the following:

from sklearn.model_selection import train_test_split

Upvotes: 2

Related Questions