CODEWITHSUNDEEP
pythoncsv
123hello
123hello

Reputation: 51

Column separation python

I am working on my bachelor thesis and am working with python to analyze my data. Unfortunately I am not a programming expert nor do I know anyone who is working python.

I have a code that seperates columns in CSV files with a comma. I want the code to seperate the columns with a |.

I have tried to replace the comma in line 58 with a | but that does not work, surprise surprise. Because I am such a noob in the programming field, a google search did not make any sense to me at all. Any help would be largely appreciated!

from sklearn.feature_extraction.text import CountVectorizer
from sklearn import linear_model
import csv
import cPickle
from sklearn.metrics import accuracy_score

def main():
    train_file = "train.csv"
    test_file  = "test.csv"
    # Read documents
    train_docs, Y = read_docs(train_file)

    # Define which features to extract (character bigrams in this case)
    extract = CountVectorizer(lowercase=False, ngram_range=(2,2), 
                              analyzer="char")

    extract.fit(train_docs) # create vocabulary from training data

    # Extract features from train data
    X = extract.transform(train_docs)

    # Initialize model
    model = linear_model.LogisticRegression()

    # Train model
    model.fit(X, Y)

    # Write model to file so it can be reused
    cPickle.dump((extract,model),open("model.pickle","w")) 

    # Print coefficients to see which features are important
    for i,f in enumerate(extract.get_feature_names()):
        print f, model.coef_[0][i]

    # Testing
    # Read test data
    test_docs, Y_test = read_docs(test_file)

    # Extract features from test data
    X_test = extract.transform(test_docs)

    # Apply model to test data
    Y_predict = model.predict(X_test)

    # Evaluation
    print accuracy_score(Y_test, Y_predict)

def read_docs(filename):
    '''
    Return X,Y where X is the list of documents and Y the list of their
    labels.
    '''
    X = []
    Y = []
    with open(filename) as f:
        r = csv.reader(f)
        for row in r:
            text,label = row
            X.append(text)
            Y.append(int(label))
    return X,Y


main()

At this moment I got as far as this:

 csv.register_dialect('pipes', delimiter='|')

    with open(filename) as f:
        r = csv.reader(f, dialect ='pipes')
        for row in r:
            text,label = row
            X.append(text)
            Y.append(int(label))
    return X,Y

But i keep getting an error now:

Traceback (most recent call last):
  File "D:/python/logreggwen.py", line 67, in <module>
    main()
  File "D:/python/logreggwen.py", line 11, in main
    train_docs, Y = read_docs(train_file)
  File "D:/python/logreggwen.py", line 61, in read_docs
    text,label = row
ValueError: need more than 1 value to unpack

Upvotes: 1

Views: 136

Answers (1)

Dr. Jan-Philip Gehrcke
Dr. Jan-Philip Gehrcke

Reputation: 35756

You need to tell the CSV reader what delimiter your data file uses:

csv.reader(f, delimiter='|')

But actually, you need to read the corresponding documentation:

https://docs.python.org/2/library/csv.html#examples

Upvotes: 1

Related Questions