CODEWITHSUNDEEP
pythonpandasnumpy
sjishan
sjishan

Reputation: 3672

fastest way to create pandas dataframe rows for combination of values from lists

let's say i have three list

listA = ['a','b','c', 'd']
listP = ['p', 'q', 'r']
listX = ['x', 'z']

so the dataframe will have 4*3*2 = 24 rows. now, the simplest way to solve this problem is to do this:

df = pd.DataFrame(columns=['A','P','X'])

for val1 in listA:
   for val2 in listP:
      for val3 in listX:
         df.loc[<indexvalue>] = [val1,val2,val3]

now in the real scenario I will have about 800k rows and 12 columns (so 12 nesting in the loops). is there any way i can create this dataframe much faster?

Upvotes: 3

Views: 1427

Answers (2)

Tarifazo
Tarifazo

Reputation: 4343

Similar discussion here. Apparently np.meshgrid is more efficient for large data (as an alternative to itertools.product.

Application:

v = np.stack(i.ravel() for i in np.meshgrid(listA, listP, listX)).T
df = pd.DataFrame(v, columns=['A', 'P', 'X'])
>>  A  P  X
0   a  p  x
1   a  p  z
2   b  p  x
3   b  p  z
4   c  p  x

Upvotes: 1

Dani Mesejo
Dani Mesejo

Reputation: 61920

You could use itertools.product:

import pandas as pd
from itertools import product

listA = ['a', 'b', 'c', 'd']
listP = ['p', 'q', 'r']
listX = ['x', 'z']

df = pd.DataFrame(data=list(product(listA, listP, listX)), columns=['A','P','X'])
print(df.head(10))

Output

   A  P  X
0  a  p  x
1  a  p  z
2  a  q  x
3  a  q  z
4  a  r  x
5  a  r  z
6  b  p  x
7  b  p  z
8  b  q  x
9  b  q  z

Upvotes: 1

Related Questions