CODEWITHSUNDEEP
pythonpandas
Louis Thibault
Louis Thibault

Reputation: 21410

How can I emulate converters when constructiing a DataFrame?

I'm trying to create a small class to handle reading data from an ASCII file. Below is the code I've written.

class EyelinkParser(object):
    eyesample = namedtuple('Eyesample', ('time', 'x', 'y', 'pupil'))
    etevent = namedtuple('EyeTrackerEvent', ('time', 'msg'))
    _pos_cnvrt = lambda v: float(v.strip()) if '.' not in v else str('NaN')
    converters = {'time': lambda t: int(t.strip()),
                  'x': _pos_cnvrt,
                  'y': _pos_cnvrt,
                  'pupil': _pos_cnvrt,
                  'msg': lambda s: s.strip()
                 } 

    def __init__(self, fileobj):
        self.fileobj = fileobj
        self.started = False

        self.sample = []
        self.event = []

        self.parse()

    def parse(self):
        for line in self.fileobj:
            line = line.split('\t')
            if line[0] in ['START', 'END']:
                self.started = line[0] == 'START'

            if self.started:
                self.process_line(line)

        self.sample = pd.DataFrame(self.sample, columns=['time', 'x', 'y', 'pupil'], converters=self.converters)
        self.event = pd.DataFrame(self.event, columns=['time', 'msg'], converters=self.converters)

    def process_line(self, line):
        if len(line) == 2 and line[0] == 'MSG':
            msg_data = line[1].split()
            if len(msg_data) == 2:
                self.event.append(self.etevent(*msg_data))
        elif len(line) == 4:
            # TODO:  replace '.' with NaNs
            self.sample.append(self.eyesample(*line))

Apparently the DataFrame class doesn't support converters. Is there an easy way to accomplish what I'm trying to do?

In summary, how can I specify the type casting of values in each column of a DataFrame?

Upvotes: 0

Views: 208

Answers (1)

Rob Story
Rob Story

Reputation: 121

I don't know how to do this explicitly as part of calling the DataFrame. When I've run into this problem, I casted after the fact with either of the following:

Passing a type to each column: 

 self.sample['x'].astype(int)

But since you're passing functions, you will probably need to use the following: 

self.sample['x'].map(_pos_cnvrt) 
self.sample['msg'].map(lambda s:s.strip())

Also, pandas has some baked in vectorized string methods to help: 

self.sample['msg'].str.strip()

Upvotes: 1

Related Questions