CODEWITHSUNDEEP
pythoncsvindexingpandas
jenryb
jenryb

Reputation: 2117

IndexError obstructing code from working with larger csv file

I have data that sorts a csv by using groupby and then plots the information. I used a small sample of information to create the code. It ran smoothly and so then I tried running it with the huge file of data.

I am pretty new at Python and this problem has been quite frustrating so even suggestions on how to troubleshoot this problem would be helpful.

My code is stopping in this section:

import pandas as pd

df =pd.DataFrame.from_csv('MYDATA.csv')
mode = lambda ts: ts.value_counts(sort=True).index[0]

I tried selecting only parts of the huge data file and it ran, but for the entire thing I am getting this error:

IndexError: index 0 is out of bounds for axis 0 with size 0

But I've looked at the two data set side-by-side and the columns are the same! I noticed that the big file has some utf8 issues with accents and I am working on combing those out, but this IndexError is perplexing me.

Here is the traceback

runfile('C:/Users/jbyrusb/Documents/Python Scripts/Tests/tests/TopSixCustomersExecute.py', wdir='C:/Users/jbyrusb/Documents/Python Scripts/Tests/tests')
Traceback (most recent call last):

  File "<ipython-input-45-53a2a006076e>", line 1, in <module>
    runfile('C:/Users/jbyrusb/Documents/Python Scripts/Tests/tests/TopSixCustomersExecute.py', wdir='C:/Users/jbyrusb/Documents/Python Scripts/Tests/tests')

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 682, in runfile
    execfile(filename, namespace)

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "C:/Users/jbyrusb/Documents/Python Scripts/Tests/tests/TopSixCustomersExecute.py", line 23, in <module>
    df = df.groupby('CompanyName')[['Column1','Name', 'Birthday', 'Country', 'County']].agg(mode).T.reindex(columns=cols)

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\core\groupby.py", line 676, in agg
    return self.aggregate(func, *args, **kwargs)

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\core\groupby.py", line 2674, in aggregate
    result = self._aggregate_generic(arg, *args, **kwargs)

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\core\groupby.py", line 2722, in _aggregate_generic
    return self._aggregate_item_by_item(func, *args, **kwargs)

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\core\groupby.py", line 2751, in _aggregate_item_by_item
    colg.aggregate(func, *args, **kwargs), data)

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\core\groupby.py", line 2307, in aggregate
    result = self._aggregate_named(func_or_funcs, *args, **kwargs)

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\core\groupby.py", line 2394, in _aggregate_named
    output = func(group, *args, **kwargs)

  File "C:/Users/jbyrusb/Documents/Python Scripts/Tests/tests/TopSixCustomersExecute.py", line 20, in <lambda>
    mode = lambda ts: ts.value_counts(sort=True).index[0]

  File "C:\Users\jbyrusb\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\core\index.py", line 915, in __getitem__
    return getitem(key)

IndexError: index 0 is out of bounds for axis 0 with size 0

Upvotes: 0

Views: 121

Answers (2)

0n10n_
0n10n_

Reputation: 402

Had the same issue i resolved by changing the sep argument from sep='\t' to sep=','.

Hope it saves someone.

Upvotes: 0

Alexander
Alexander

Reputation: 109546

It is difficult without seeing the data causing the error, but try this:

mode = (lambda ts: ts.value_counts(sort=True).index[0] 
                   if len(ts.value_counts(sort=True)) else None)

Upvotes: 1

Related Questions