CODEWITHSUNDEEP
pythonpandasdataframe
Night Walker
Night Walker

Reputation: 21260

How I can speed up row column access to pandas dataframe?

I am having speed issues with my code after I switched to following data frame:

df = pd.DataFrame(data=products, columns=['pk', 'product_name', 'category_name', 'brand_name'])
df.set_index(['pk'], inplace=True)

This is the only place I use the data frame. 'pk' is integer.

            category = self.product_list.iloc[int(prod)-1]['category_name']
            brand = self.product_list.iloc[int(prod)-1]['brand_name']

What I am doing here wrong ?

Upvotes: 1

Views: 398

Answers (1)

jezrael
jezrael

Reputation: 862471

You can use iat:

print product_list.category_name.iat[int(prod)-1]
print product_list.brand_name.iat[int(prod)-1]

Timing (index - string):

Sample:

product_list = pd.DataFrame({'brand_name': {'r': 'r', 'g': 't', 'w': 'i'}, 
                             'category_name': {'r': 's', 'g': 'f', 'w': 'a'}})
print product_list
  brand_name category_name
g          t             f
r          r             s
w          i             a

In [242]: %timeit product_list.iloc[int(prod)-1]['category_name']
The slowest run took 8.27 times longer than the fastest. This could mean that an intermediate result is being cached 
10000 loops, best of 3: 82.7 µs per loop

In [243]: %timeit product_list.brand_name.iat[int(prod)-1]
The slowest run took 16.01 times longer than the fastest. This could mean that an intermediate result is being cached 
100000 loops, best of 3: 9.96 µs per loop

index: int:

product_list = pd.DataFrame({'brand_name': {0: 't', 1: 'r', 2: 'i'}, 
                             'category_name': {0: 'f', 1: 's', 2: 'a'}})
print product_list
  brand_name category_name
0          t             f
1          r             s
2          i             a

In [250]: %timeit product_list.iloc[int(prod)-1]['category_name']
The slowest run took 8.24 times longer than the fastest. This could mean that an intermediate result is being cached 
10000 loops, best of 3: 84.7 µs per loop

In [251]: %timeit product_list.brand_name.iat[int(prod)-1]
The slowest run took 24.17 times longer than the fastest. This could mean that an intermediate result is being cached 
100000 loops, best of 3: 9.86 µs per loop

Upvotes: 1

Related Questions