Machine Learning/NLP text classification: training a model from corpus of text files - scikit learn

Question

I am very new to machine learning and I was wondering if somebody could take me through this code and why it is not working. It is my own variation of the scikit-learn tutorial found at: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html which is basically what I am trying to do. I need to train a model with a labelled training set so that when I use my test set, it can predict the label of the test set. Also it would be really useful if somebody could show me how to save and load the model. Thank you very much. This is what I have so far:

import codecs
import os

import numpy as np
import pandas as pd

from Text_Pre_Processing import Pre_Processing

filenames = os.listdir(
    "...scikit-machine-learning/training_set")
files = []
array_data = []
array_label = []
for file in filenames:
    with codecs.open("...scikit-machine-learning/training_set/" + file, "r",
                     encoding='utf-8', errors='ignore') as file_data:
        open_file = file_data.read()
        open_file = Pre_Processing.lower_case(open_file)
        open_file = Pre_Processing.remove_punctuation(open_file)
        open_file = Pre_Processing.clean_text(open_file)
        files.append(open_file)
# ----------------------------------------------------
# PUTTING LABELS INTO LIST
for file in files:
    if 'inheritance' in file:
        array_data.append(file)
        array_label.append('Inheritance (object-oriented programming)')
    elif 'pagerank' in file:
        array_data.append(file)
        array_label.append('PageRank')
    elif 'vector space model' in file:
        array_data.append(file)
        array_label.append('Vector Space Model')
    elif 'bayes' in file:
        array_data.append(file)
        array_label.append('Bayes' + "'" + ' Theorem')
    else:
        array_data.append(file)
        array_label.append('Dynamic programming')
#----------------------------------------------------------

csv_array = []
for i in range(0, len(array_data)):
    csv_array.append([array_data[i], array_label[i]])

# format of array [[string, label], [string, label], [string, label]]
import csv

with open('data.csv', 'w') as target:
    writer = csv.writer(target)
    writer.writerows(zip(test_array))

data = pd.read_csv('data.csv')
numpy_array = data.as_matrix()

X = numpy_array[:, 0]
Y = numpy_array[:, 1]

from sklearn.model_selection import train_test_split

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.4, random_state=42)

from sklearn.feature_extraction.text import CountVectorizer

from sklearn.feature_extraction.text import TfidfTransformer

from sklearn.naive_bayes import MultinomialNB

from sklearn.pipeline import Pipeline

text_clf = Pipeline(['vect', CountVectorizer(stop_words='english'), 'tfidf', TfidfTransformer(),
                     'clf', MultinomialNB()])

text_clf = text_clf.fit(X_train, Y_train)

predicted = text_clf.predict(X_test)
np.mean(predicted == Y_test)

I saw online people using csv files to input the data so I tried that too, i may not need it so i apologise if that is incorrect.

error being shown:

C:.../scikit-machine-learning/train.py:63: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  numpy_array = data.as_matrix()
Traceback (most recent call last):
  File "C:/...scikit-machine-learning/train.py", line 66, in <module>
    Y = numpy_array[:,1]
IndexError: index 1 is out of bounds for axis 1 with size 1

Thank you very much for your help, please let me know if you need further explanation.

sample of two entries in csv:

"['dynamic programming is an algorithmic technique used to solve certain optimization problems where the object is to find the best solution from a number of possibilities it uses a so called bottomup approach meaning that the problem is solved as a set of subproblems which in turn are made up of subsubproblemssubproblems are then selected and used to solve the overall problem these subproblems are only solved once and the solutions are saved so that they will not need to be recalculated again whilst calculated individually they may also overlap when any subproblem is met again it can be found and reused to solve another problem since it searches all possibilities it is also very accurate this method is far more efficient than recalculating and therefore considerably reduces computation it is widely used in computer science and can be applied for example to compress data in high density bar codes dynamic programming is most effective and therefore most often used on objects that are ordered from left to right and whose order cannot be rearranged this means it works well on character chains for example ', 'Dynamic programming']"

"['inheritance is one of the basic concepts of object oriented programming its objective is to add more detail to preexisting classes whilst still allowing the methods and variables of these classes to be reused the easiest way to look at inheritance is as an is a kind of relationship for example a guitar is a kind of string instrument electric acoustic and steel stringed are all types of guitar the further down an inheritance tree you get the more specific the classes become an example here would be books books generally fall into two categories fiction and nonfiction each of these can then be subdivided into more groups fiction for example can be split into fantasy horror romance and many more nonfiction splits the same way into other topics such as history geography cooking etc history of course can be subdivided into time periods like the romans the elizabethans the world wars and so on', 'Inheritance (object-oriented programming)']"

Answer 1

You need to remove the characters [' and '] from the csv since read_csv is treating them as a string (one column) rather than a two column dataframe. There is also a typo error on the line text_clf = Pipeline so I fixed it too. Goodluck!

data = pd.read_csv('data.csv', header=None)
numpy_array = data.as_matrix()

strarr = numpy_array[:, 0]
X=[strarr[i].split(",")[0].replace("[",'').replace("'",'') for i in range(len(strarr))]
Y=[strarr[i].split(",")[1].replace("]",'').replace("'",'') for i in range(len(strarr))]

from sklearn.model_selection import train_test_split

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.4, random_state=42)

from sklearn.feature_extraction.text import CountVectorizer

from sklearn.feature_extraction.text import TfidfTransformer

from sklearn.naive_bayes import MultinomialNB

from sklearn.pipeline import Pipeline

text_clf = Pipeline([('vect', CountVectorizer(stop_words='english')), ('tfidf', TfidfTransformer()), ('clf', MultinomialNB())])

text_clf = text_clf.fit(X_train, Y_train)

predicted = text_clf.predict(X_test)
np.mean(predicted == Y_test)