CODEWITHSUNDEEP
pythonnumpyscikit-learndecision-tree
swati saoji
swati saoji

Reputation: 2087

How to Prepare Data for DecisionTreeClassifier Scikit

I have the following Data in csv , the top row indicate column headers and data is indexed, all the data is dicretized. I need to make a decision tree classifier Model . Could some one guide me with this. 

    ,age,workclass,fnlwgt,education,education-num,marital-status,occupation,relationship,race,sex,capital-gain,capital-loss,hours-per-week,native-country,class
0,"(16.927, 41.333]", State-gov,"(10806.885, 504990]", Bachelors,"(12, 16]", Never-married, Adm-clerical, Not-in-family, White, Male,"(0, 5000]",,"(30, 50]", United-States, <=50K
1,"(41.333, 65.667]", Self-emp-not-inc,"(10806.885, 504990]", Bachelors,"(12, 16]", Married-civ-spouse, Exec-managerial, Husband, White, Male,,,"(0, 30]", United-States, <=50K
2,"(16.927, 41.333]", Private,"(10806.885, 504990]", HS-grad,"(8, 12]", Divorced, Handlers-cleaners, Not-in-family, White, Male,,,"(30, 50]", United-States, <=50K
3,"(41.333, 65.667]", Private,"(10806.885, 504990]", 11th,"(-1, 8]", Married-civ-spouse, Handlers-cleaners, Husband, Black, Male,,,"(30, 50]", United-States, <=50K
4,"(16.927, 41.333]", Private,"(10806.885, 504990]", Bachelors,"(12, 16]", Married-civ-spouse, Prof-specialty, Wife, Black, Female,,,"(30, 50]", Cuba, <=50K

My aprroach so far:

df, filen = decision_tree.readCSVFile("../Data/discretized.csv")
print df[:3]
newdf = decision_tree.catToInt(df)
print newdf[:3]
model = DecisionTreeClassifier(random_state=0)
print cross_val_score(model, newdf, newdf[:,14], cv=10)

catToInt function:

def catToInt(df):
    mapper={}
    categorical_list = list(df.columns.values)
    newdf = pd.DataFrame(columns=categorical_list)
    #Converting Categorical Data
    for x in categorical_list:
        mapper[x]=preprocessing.LabelEncoder()
    for x in categorical_list:
        someinput = df.__getattr__(x)
        newcol = mapper[x].fit_transform(someinput)
        newdf[x]= newcol
    return newdf

The error :

        print cross_val_score(model, newdf, newdf[:,14], cv=10)
  File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 1787, in __getitem__
    return self._getitem_column(key)
  File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 1794, in _getitem_column
    return self._get_item_cache(key)
  File "C:\Python27\lib\site-packages\pandas\core\generic.py", line 1077, in _get_item_cache
    res = cache.get(item)
TypeError: unhashable type

So I am able to transform categorical data to int. but I thing i am missing something in next step .

Upvotes: 0

Views: 1555

Answers (1)

swati saoji
swati saoji

Reputation: 2087

This is what I got as the solution by following the comments above and more searching. I got the intended result but i understand there would be more refined way to do this. 

from sklearn.tree import DecisionTreeClassifier
from sklearn.cross_validation import cross_val_score
import pandas as pd
from sklearn import preprocessing
def main():
    df, _ = readCSVFile("../Data/discretized.csv")
    newdf, classl = catToInt(df)
    model = DecisionTreeClassifier()
    print cross_val_score(model, newdf, classl, cv=10)


def readCSVFile(filepath):
    df = pd.read_csv(filepath, index_col=0)
    (_, _, sufix) = filepath.rpartition('\\')
    (prefix, _, _) =sufix.rpartition('.')
    print "csv read and converted to dataframe !!"
    # df['class'] = df['class'].apply(replaceLabel)
    return df, prefix

def catToInt(df):
    # replace the Nan with "NA" which acts as a unique category
    df.fillna("NA", inplace=True)
    mapper={}

    # make list of all column headers 
    categorical_list = list(df.columns.values)

    #exclude the class column
    categorical_list.remove('class')
    newdf = pd.DataFrame(columns=categorical_list)

    #Converting Categorical Data to integer labels
    for x in categorical_list:
        mapper[x]=preprocessing.LabelEncoder()
    for x in categorical_list:
        newdf[x]= mapper[x].fit_transform(df.__getattr__(x))

    # make a class series encoded : 
    le = preprocessing.LabelEncoder()
    myclass = le.fit_transform(df.__getattr__('class'))

   #newdf is the dataframe with all columns except classcoumn and myclass is the class column 
    return newdf, myclass

main()

some links other than comments above that helped me :

  1. http://fastml.com/converting-categorical-data-into-numbers-with-pandas-and-scikit-learn/
  2. http://biggyani.blogspot.com/2014/08/using-onehot-with-categorical.html

Output:

csv read and converted to dataframe !!
[ 0.83418628  0.83930399  0.83172979  0.82804504  0.83930399  0.84254709
  0.82985258  0.83022732  0.82428835  0.83678067]

It might help some novice user of sklearn like me. Suggestions/edits and better answers are welcome.

Upvotes: 1

Related Questions