CODEWITHSUNDEEP
python-2.7csvparsingoptimization
NGB
NGB

Reputation: 430

Parse CSV efficiently in python

I am writing a CSV parser which has following structure

class decode:
def __init__(self):
    self.fd = open('test.csv')


def decodeoperation(self):
    for row in self.fd:
        getcmd = self.decodecmd(row)
        if cmd == 'A'
            self.decodeAopt()

        elif cmd == 'B':
            self.decodeBopt()

def decodeAopt(self):
    for row in self.fd:
        #decodefurther dependencies based on cmd A till 
        #a condition occurs on any further row

    return  
def decodeBopt(self):
    for row in self.fd:
        #decodefurther dependencies based on cmd B till 
        #a condition occurs on any further row
    return

The current code is working fine for me but I am not feeling good to iterate through the CSV file in all the methods. Could it be done in a better way?

Upvotes: 0

Views: 136

Answers (1)

PaulMcG
PaulMcG

Reputation: 63709

There is nothing inherently wrong with using a common iterator across multiple methods, as long as you can determine in advance which method to dispatch to at any given point in the sequence (which you are doing by decoding the cmd from the row and getting 'A', 'B', etc.). The design has issues if you have to read several items before you could determine which method to call, and might have to back up if you picked the wrong method and needed to try another. In parsing, this is called backtracking. Since you are passing around a file object, backing up is difficult. Note that your separate decoder methods will have to know when to stop before reading the next row that contains a command, so they will need some sort of terminating sentinel row that they can recognize.

Some general comments on your Python and class design:

You have a nice simple if-elif-elif dispatch table that can translate to a Python dict like this:

# put this code in place of your "if cmd == ... elif elif elif..." code
dispatch = {
    # note - no ()'s, we just want to reference the methods, not call them
    'A': self.decodeAopt,  
    'B': self.decodeBopt,
    'C': self.decodeCopt,
    # look how easy it is to add more decoders
}

# lookup which decoder to use for the current cmd
decoder = dispatch[cmd]

# run it
decoder()

# or do it all in one line
dispatch[cmd]()

Instead of having your __init__ method open a file, let it accept an iterator object. This will make it much easier to write tests for your object, since you'll be able to pass simple Python lists containing CSV rows.

class decode:
    def __init__(self, sequence):
        self.fd = sequence

You might want to rename this var from 'fd' to something like 'seq', since it doesn't have to be a file, but could be any iterable that gives you decodable rows.

If you are doing your own CSV parsing, look at using the builtin csv module. It will do quite a bit of work for you, like parsing quoted strings that could contain commas, and can give you easy-to-work-with dicts for each row, given headers read from the input file, or specified by you. If you have modified __init__ as I suggested, you can use it like:

import csv

# assuming test.csv has a header row
reader = csv.DictReader(open('test.csv'))

# or specify headers if not - I encourage you to give these columns better names
reader.fieldnames = ['cmd', 'val1', 'val2', 'val3']

decoder = decode(reader)
decoder.decodeoperation()

Then you can write in decodeoperation:

cmd = row['cmd']

Note that this would impart a slightly different design to your class, that it would expect to be given a sequence of dicts, rather than a sequence of strings.

Upvotes: 2

Related Questions