Unable to solve an error while running Gridsearch

Question

I am new to the realm of machine learning, and I started competing in Kaggle competitions to get some practical experience. I am competing in the knowledge competition CIFAR 10- Object Recognition in Images, where you have to classify thousands of images in 10 classes,all the data I use can be found there. I tried to implement Gridsearch to optimize the parameters of my machine learning algorithm, but whenever I try to fit my classifier with my training data I get an error. I have found the function that raised the error, and it has something to do with my labels not having the right type, but I have no clue on how to solve it. The Labels I use are strings, and I preprocess them so that I can feed them to the algorithm. Am I doing something wrong there? Or perhaps something goes wrong when I split the dataset for the grid search? Frankly, I lack the experience and knowledge to solve this problem, and I could definitely use your help.

The code involved:

import glob
import os
from sklearn.svm import SVC
from sklearn import preprocessing
import pandas as pd
from sklearn import cross_validation 
from sklearn import metrics
from sklearn.grid_search import GridSearchCV

def label_preprocessing(Labels):
    Labels = np.array(Labels)[:,1]
    le = preprocessing.LabelEncoder()
    le.fit_transform(Labels)
    return Labels
def model_selection(train,Labels):
    parameters = {"C":[0.1,1,10,100],"gamma":[0.0001,0.001,0.01,0.1]}
    X_train, X_test, y_train, y_test = cross_validation.train_test_split(train, Labels, test_size = 0.2, random_state = 0)
    svm = SVC()
    clf  = GridSearchCV(svm,parameters)
    clf  = clf.fit(X_train,y_train)
    print ("20 fold cv score: ",np.mean(cross_validation.cross_val_score(clf,X_test,y_test,cv = 10,scoring = "roc_auc")))
    return clf

if  __name__ == "__main__":
    train_images = np.array(file_open(image_dir1,"*.png"))[:100]
    test_images = np.array(file_open(image_dir2,"*.png"))[:100]
    Labels = label_preprocessing(pd.read_csv(image_dir3)[:100])
    train_set = [matrix_image(image) for image in train_images]
    test_set = [matrix_image(image) for image in test_images]
    train_set = np.array(train_set)
    test_set = np.array(test_set)

    print("selecting best model and evaluating it")
    svm = model_selection(train_set,Labels)
    print("predicting stuff")
    result = svm.predict(test_set)

    np.savetxt("submission.csv", result, fmt = "%s", delimiter = ",")

full traceback:

Traceback (most recent call last):
  File "C:\Users\Abdc\workspace\final_submission\src\SVM.py", line 49, in <module>
    svm = model_selection(train_set,Labels)
  File "C:\Users\Abdc\workspace\final_submission\src\SVM.py", line 35, in model_selection
    clf  = clf.fit(X_train,y_train)
  File "C:\Python27\lib\site-packages\sklearn\grid_search.py", line 707, in fit
    return self._fit(X, y, ParameterGrid(self.param_grid))
  File "C:\Python27\lib\site-packages\sklearn\grid_search.py", line 493, in _fit
    for parameters in parameter_iterable
  File "C:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", line 517, in __call__
    self.dispatch(function, args, kwargs)
  File "C:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", line 312, in dispatch
    job = ImmediateApply(func, args, kwargs)
  File "C:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", line 136, in __init__
    self.results = func(*args, **kwargs)
  File "C:\Python27\lib\site-packages\sklearn\grid_search.py", line 311, in fit_grid_point
    this_score = clf.score(X_test, y_test)
  File "C:\Python27\lib\site-packages\sklearn\base.py", line 294, in score
    return accuracy_score(y, self.predict(X))
  File "C:\Python27\lib\site-packages\sklearn\metrics\metrics.py", line 1064, in accuracy_score
    y_type, y_true, y_pred = _check_clf_targets(y_true, y_pred)
  File "C:\Python27\lib\site-packages\sklearn\metrics\metrics.py", line 123, in _check_clf_targets
    raise ValueError("{0} is not supported".format(y_type))
ValueError: unknown is not supported

This is the function that raised the error. It can be found in the sklearn.metrics module:

def _check_clf_targets(y_true, y_pred):
    """Check that y_true and y_pred belong to the same classification task

    This converts multiclass or binary types to a common shape, and raises a
    ValueError for a mix of multilabel and multiclass targets, a mix of
    multilabel formats, for the presence of continuous-valued or multioutput
    targets, or for targets of different lengths.

    Column vectors are squeezed to 1d.

    Parameters
    ----------
    y_true : array-like,

    y_pred : array-like

    Returns
    -------
    type_true : one of {'multilabel-indicator', 'multilabel-sequences', \
                        'multiclass', 'binary'}
        The type of the true target data, as output by
        ``utils.multiclass.type_of_target``

    y_true : array or indicator matrix or sequence of sequences

    y_pred : array or indicator matrix or sequence of sequences
    """
    y_true, y_pred = check_arrays(y_true, y_pred, allow_lists=True)
    type_true = type_of_target(y_true)
    type_pred = type_of_target(y_pred)

    y_type = set([type_true, type_pred])
    if y_type == set(["binary", "multiclass"]):
        y_type = set(["multiclass"])

    if len(y_type) > 1:
        raise ValueError("Can't handle mix of {0} and {1}"
                         "".format(type_true, type_pred))

    # We can't have more than one value on y_type => The set is no more needed
    y_type = y_type.pop()

    # No metrics support "multiclass-multioutput" format
    if (y_type not in ["binary", "multiclass", "multilabel-indicator",
                       "multilabel-sequences"]):
        raise ValueError("{0} is not supported".format(y_type))

    if y_type in ["binary", "multiclass"]:
        y_true = column_or_1d(y_true)
        y_pred = column_or_1d(y_pred)

    return y_type, y_true, y_pred

Extra information about the Labels:

Content of labels and dtype:

In [21]:
Labels = np.array(Labels)[:,1]
Labels


Out[21]:
array(['frog', 'truck', 'truck', ..., 'truck', 'automobile', 'automobile'], dtype=object)

Content of labels after preprocessing

In [25]:

Labels = np.array(Labels)[:,1]
Labels
le = preprocessing.LabelEncoder()
Labels = le.fit_transform(Labels)
Labels



Out[25]:
array([6, 9, 9, ..., 9, 1, 1])

Shape of labels after preprocessing:

In [18]:
    Labels = np.array(Labels)[:,1]
    Labels.shape
    le = preprocessing.LabelEncoder()
    Labels = le.fit_transform(Labels)
    Labels.shape

Out[18]:
(50000L,)

The original content can be found here: https://www.kaggle.com/c/cifar-10/data. Which contains an ID for a datapoint and its class Label. So it is a nx2 matrix.

Answer 1

This might be caused by issue #2374. As a workaround you can try to use Labels = Labels.astype(str).

Also, I would suggest you to follow the PEP8 code conventions to share python code with the community. In particular variable names are usually lowercase.