CODEWITHSUNDEEP
pythonpandas
benjaminmgross
benjaminmgross

Reputation: 2102

`pandas.DataFrame.apply` in a row by row operation

I would like to return a dataFrame with each row sorted (let's say descending). So if I have the pandas.DataFrame named data:

In [38]: data
Out[38]: 
                  c1        c2        c3        c4        c5        c6
Date                                                                  
2012-10-22  0.973371  0.226342  0.968282  0.872330  0.273880  0.746156
2012-10-19  0.497048  0.351332  0.310025  0.726669  0.344202  0.878755
2012-10-18  0.315764  0.178584  0.838223  0.749962  0.850462  0.400253
2012-10-17  0.162879  0.068409  0.704094  0.712860  0.537545  0.009789

I would like the following returned:

In [39]: sorted_frame
Out[39]: 
                   0         1         2         3         4         5
Date                                                                  
2012-10-22  0.973371  0.968282  0.872332  0.746156  0.273880  0.226342
2012-10-19  0.878755  0.726669  0.497048  0.351332  0.344202  0.310025
2012-10-18  0.850462  0.838223  0.749962  0.400253  0.315764  0.178584
2012-10-17  0.712860  0.704094  0.537545  0.162879  0.068409  0.009789

I've tried DataFrame.sort(axis = 1) however, that doesn't achieve the desired result:

In [40]: data.sort(axis = 1)
Out[43]: 
                  c1        c2        c3        c4        c5        c6
Date                                                                  
2012-10-22  0.973371  0.226342  0.968282  0.872330  0.273880  0.746156
2012-10-19  0.497048  0.351332  0.310025  0.726669  0.344202  0.878755
2012-10-18  0.315764  0.178584  0.838223  0.749962  0.850462  0.400253
2012-10-17  0.162879  0.068409  0.704094  0.712860  0.537545  0.009789

I've created the following function that accomplishes what I'm looking for (using the pandas.TimeSeries.order()):

import numpy

def sorted_by_row(frame, ascending = False):
    vals = numpy.tile(numpy.nan,frame.shape)
    for row in numpy.arange(frame.shape[0]):
        vals[row, :] = frame.ix[row, :].order(ascending = ascending)
    return pandas.DataFrame(vals, index = frame.index)

However, my goal is to be able to use a row-wise function in the DataFrame.apply() method (so I can apply the desired functionality to other functions I build). I've tried:

 #TimeSeries.order() sorts a pandas.TimeSeries object
 data.apply(lambda x: x.order(), axis = 1)

But again, I'm not getting the desired DataFrame above (I've outputted enough DataFrame's so I'll spare the page the real estate).

Your help is greatly appreciated,

-B

Upvotes: 1

Views: 5009

Answers (2)

Wes McKinney
Wes McKinney

Reputation: 105521

Well, it's not too easy to do with pandas out of the box. First, familiarize yourself with argsort:

In [8]: df
Out[8]: 
                   0         1         2         3         4
2012-10-17  1.542735  1.081290  2.602967  0.748706  0.682501
2012-10-18  0.058414  0.148083  0.094104  0.716789  2.482998
2012-10-19  2.396277  0.524733  2.169018  1.365622  0.590767
2012-10-20  0.513535  1.542485  0.186261  2.138740  1.173894
2012-10-21  0.495713  1.401872  0.919931  0.055136  1.358439
2012-10-22  1.010086  0.350249  1.116935  0.323305  0.506086

In [12]: inds = df.values.argsort(1)

In [13]: inds
Out[13]: 
array([[4, 3, 1, 0, 2],
       [0, 2, 1, 3, 4],
       [1, 4, 3, 2, 0],
       [2, 0, 4, 1, 3],
       [3, 0, 2, 4, 1],
       [3, 1, 4, 0, 2]])

These are the indirect sort indices for each row. Now you want to do something like:

new_values = np.empty_like(df)
for i, row in enumerate(df.values):
    # sort in descending order
    new_values[i] = row[inds[i]][::-1]

sorted_df = DataFrame(new_values, index=df.index)

Not that satisfying but it gets the job done:

In [15]: sorted_df
Out[15]: 
                   0         1         2         3         4
2012-10-17  2.602967  1.542735  1.081290  0.748706  0.682501
2012-10-18  2.482998  0.716789  0.148083  0.094104  0.058414
2012-10-19  2.396277  2.169018  1.365622  0.590767  0.524733
2012-10-20  2.138740  1.542485  1.173894  0.513535  0.186261
2012-10-21  1.401872  1.358439  0.919931  0.495713  0.055136
2012-10-22  1.116935  1.010086  0.506086  0.350249  0.323305

More generally you can do:

In [23]: df.apply(lambda x: np.sort(x.values)[::-1], axis=1)
Out[23]: 
                   0         1         2         3         4
2012-10-17  2.602967  1.542735  1.081290  0.748706  0.682501
2012-10-18  2.482998  0.716789  0.148083  0.094104  0.058414
2012-10-19  2.396277  2.169018  1.365622  0.590767  0.524733
2012-10-20  2.138740  1.542485  1.173894  0.513535  0.186261
2012-10-21  1.401872  1.358439  0.919931  0.495713  0.055136
2012-10-22  1.116935  1.010086  0.506086  0.350249  0.323305

But you're responsible for assigning new columns yourself

Upvotes: 2

Aman
Aman

Reputation: 47189

Sorting is a big subject, and I'm sure there are many ways to do this. Here's one.

First creating an example dataframe.

In [31]: rndrange = pd.DatetimeIndex(start='10/17/2012', end='10/22/2012', freq='D')    
In [32]: df = pd.DataFrame(np.random.randn(len(rndrange),5),index=rndrange)
In [33]: df = df.applymap(abs)  #Easier to see sorting if all vals are positive
In [34]: df
Out[34]: 
                   0         1         2         3         4
2012-10-17  1.542735  1.081290  2.602967  0.748706  0.682501
2012-10-18  0.058414  0.148083  0.094104  0.716789  2.482998
2012-10-19  2.396277  0.524733  2.169018  1.365622  0.590767
2012-10-20  0.513535  1.542485  0.186261  2.138740  1.173894
2012-10-21  0.495713  1.401872  0.919931  0.055136  1.358439
2012-10-22  1.010086  0.350249  1.116935  0.323305  0.506086

Sorting: 

In [35]: df.as_matrix().sort(1)    
In [36]: df
Out[36]: 
                   0         1         2         3         4
2012-10-17  0.682501  0.748706  1.081290  1.542735  2.602967
2012-10-18  0.058414  0.094104  0.148083  0.716789  2.482998
2012-10-19  0.524733  0.590767  1.365622  2.169018  2.396277
2012-10-20  0.186261  0.513535  1.173894  1.542485  2.138740
2012-10-21  0.055136  0.495713  0.919931  1.358439  1.401872
2012-10-22  0.323305  0.350249  0.506086  1.010086  1.116935

Upvotes: 1

Related Questions