CODEWITHSUNDEEP
pythonpandasnumpyiteration
striatum
striatum

Reputation: 1598

Efficient nested looping with pandas dataframe

I have a simple panda dataframe like this one:

d = {'col1': ['a','b','c','d','e'], 'col2': [1,2,3,4,5]}
df = pd.DataFrame(d)
df
  col1  col2
0    a     1
1    b     2
2    c     3
3    d     4
4    e     5

And I would need to iterate over it and to get a simple arithmetic results (like a product or so) for all combination of row values. I was thinking to make a matrix and put the values in, like this:

size = df.shape[0]
mtx = np.zeros(shape=(size, size))
mtx
array([[ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.]])

But I somehow 'sense' there is more efficient way to do this than nested looping, like this:

for index1, c11, c12, in df.itertuples():
    for index2, c21, c22 in df.itertuples():
        mtx[index1][index2] = float(c12) * float(c22)

mtx
array([[  1.,   2.,   3.,   4.,   5.],
       [  2.,   4.,   6.,   8.,  10.],
       [  3.,   6.,   9.,  12.,  15.],
       [  4.,   8.,  12.,  16.,  20.],
       [  5.,  10.,  15.,  20.,  25.]])

Any idea will be much appreciated! Thanks!

Upvotes: 1

Views: 194

Answers (1)

Miriam Farber
Miriam Farber

Reputation: 19634

For oprations like *,+,-,/ you can do the following: (this example is for *, but you can just change the operation in the last row if you want +,- or /)

import numpy as np
import pandas as pd
d = {'col1': ['a','b','c','d','e'], 'col2': [1,2,3,4,5]}
df = pd.DataFrame(d)
a=np.array([df.col2.tolist()])
a.T*a

The result is:

array([[ 1,  2,  3,  4,  5],
   [ 2,  4,  6,  8, 10],
   [ 3,  6,  9, 12, 15],
   [ 4,  8, 12, 16, 20],
   [ 5, 10, 15, 20, 25]], dtype=int64)

Change a.T*a to a.T+a for pairwise sum, and to a.T-a for pairwise difference. If you want pairwise division, you can change it into a.T/a, but remember to include the line a=a.astype(float) above the operation.

Upvotes: 3

Related Questions