CODEWITHSUNDEEP
pythonmachine-learningclassificationsvmtext-classification
prashantgpt91
prashantgpt91

Reputation: 1795

Classification using SVM

In an attempt to classify text I want to use SVM. I want to classify test data into one of the labels(health/adult) The training & test data are text files

I am using python's scikit library. While I was saving the text to txt files I encoded it in utf-8 that's why i am decoding them in the snippet. Here's my attempted code

String = String.decode('utf-8')
String2 = String2.decode('utf-8')
bigram_vectorizer = CountVectorizer(ngram_range=(1, 2),
                                     token_pattern=r'\b\w+\b', min_df=1)

X_2 = bigram_vectorizer.fit_transform(String2).toarray()
X_1 = bigram_vectorizer.fit_transform(String).toarray()
X_train = np.array([X_1,X_2])
print type(X_train)
y = np.array([1, 2])
clf = SVC()
clf.fit(X_train, y)

#prepare test data
print(clf.predict(X))

This is the error I am getting

  File "/Users/guru/python_projects/implement_LDA/lda/apply.py", line 107, in <module>
    clf.fit(X_train, y)
  File "/Users/guru/python_projects/implement_LDA/lda/lib/python2.7/site-packages/sklearn/svm/base.py", line 150, in fit
    X = check_array(X, accept_sparse='csr', dtype=np.float64, order='C')
  File "/Users/guru/python_projects/implement_LDA/lda/lib/python2.7/site-packages/sklearn/utils/validation.py", line 373, in check_array
    array = np.array(array, dtype=dtype, order=order, copy=copy)
ValueError: setting an array element with a sequence.

When I searched for the error, I found some results but they even didn't help. I think I am logically wrong here in applying SVM model. Can someone give me a hint on this?

Ref: [1][2]

Upvotes: 0

Views: 283

Answers (1)

frist
frist

Reputation: 1958

You have to combine your samples, vectorize them and then fit the classifier. Like this:

String = String.decode('utf-8')
String2 = String2.decode('utf-8')
bigram_vectorizer = CountVectorizer(ngram_range=(1, 2),
                                     token_pattern=r'\b\w+\b', min_df=1)

X_train = bigram_vectorizer.fit_transform(np.array([String, String2]))
print type(X_train)
y = np.array([1, 2])
clf = SVC()
clf.fit(X_train, y)

#prepare test data
print(clf.predict(bigram_vectorizer.transform(np.array([X1, X2, ...]))))

But 2 sample it's a very few amount of data so likely your prediction will not be accurate.

EDITED:

Also you can combine transformation and classification in one step using Pipeline.

from sklearn.pipeline import Pipeline

print type(X_train) # Should be a list of texts length 100 in your case
y_train = ... # Should be also a list of length 100
clf = Pipeline([
    ('transformer', CountVectorizer(...)),
    ('estimator', SVC()),
])
clf.fit(X_train, y_train)

X_test = np.array(["sometext"]) # array of test texts length = 1
print(clf.predict(X_test))

Upvotes: 2

Related Questions