CODEWITHSUNDEEP
pythonpandasstackappendconcatenation
grommit
grommit

Reputation: 177

Pandas DataFrame concatenation in for loop returns empty DataFrame

I am trying to stack several cyclical data elements of a DataFrame on top of each other to change the DataFrame dimensions. E.g. Go from 100x20 to 500x4.

Sample 11x7 input:
    0   1   2   3   4   5   6   7       
0   1   713 1622    658 1658    620 1734        
1   2   714 1623    657 1700    618 1735    
2   3   714 1624    656 1701    617 1736    
3   4   714 1625    655 1702    615 1738    
4   5   714 1626    654 1703    614 1739    
5   6   713 1627    653 1705    612 1740    
6   7   713 1628    651 1706    610 1741    
7   8   713 1629    650 1707    609 1742    
8   9   713 1630    649 1709    607 1744    
9   10  713 1631    648 1710    605 1745    
10  11  712 1632    646 1711    604 1746    

Desired 32x3 output:
0   1   713 1622    
1   2   714 1623    
2   3   714 1624    
3   4   714 1625    
4   5   714 1626    
5   6   713 1627    
6   7   713 1628    
7   8   713 1629    
8   9   713 1630    
9   10  713 1631    
10  11  712 1632    
11  1   658 1658    
12  2   657 1700    
13  3   656 1701    
14  4   655 1702    
15  5   654 1703    
16  6   653 1705    
17  7   651 1706    
18  8   650 1707    
19  9   649 1709    
20  10  648 1710    
21  11  646 1711    
22  1   620 1734        
23  2   618 1735    
24  3   617 1736    
25  4   615 1738    
26  5   614 1739    
27  6   612 1740    
28  7   610 1741    
29  8   609 1742    
30  9   607 1744    
31  10  605 1745    
32  11  604 1746

I have spent an inordinate amount of time checking this, and I cannot find anything better than 

pd.concat([df1, df2], ignore_index=True) 
or 
df1.append(df2, ignore_index=True)

, which should produce identical solutions inthis case. However, whichever option is used, it is going to be placed at the end of a loop that produces temporary DataFrames to be concatenated with or appended to the permanent DataFrame. The temp df's come out fine, but the allegedly straightforward concatenation step fails consistently. I get an empty DataFrame with a proper header...


for l in range(1,13):
    s1 = l * 4 - 4
    s2 = l * 4 
    dft = df0.iloc[:, s1:s2]
    dft.columns = new_col
    #pd.concat([df1, dft], ignore_index=True, axis = 0)
    #df1.append(dft, ignore_index=True)
df1.head()

Either of the commented out lines should produce a stack of 4-wide temp DataFrames... I get an empty DataFrame with a proper header and no error messages...

Upvotes: 0

Views: 1745

Answers (1)

grommit
grommit

Reputation: 177

Solved by @Aryerez in a comment above: Both of pd.concat() and df.append() are by default not in place. See if df1 = pd.concat(etc...) solves it.

Upvotes: 2

Related Questions