CODEWITHSUNDEEP
pandas
Jonas
Jonas

Reputation: 1847

get count of entries less or equal in Series

I want to get the count of all elements less or equal to each entry in a pandas.Series eg:

if __name__ == '__main__':
    import pandas as pd
    a = pd.Series(data=[4,7,3,5,2,1,1,6])
    le = pd.Series(data=[a[a <= i].count() for i in a])
    print(le)

Result:

0    5
1    8
2    4
3    6
4    3
5    2
6    2
7    7
dtype: int64

Is there a function in Series or a better way to do this for large data sets?

Upvotes: 3

Views: 65

Answers (3)

Rajat Jain
Rajat Jain

Reputation: 2032

As the problem would be applied on large datasets:

%timeit [(a.values <= x).sum() for x in a]
10000 loops, best of 3: 28.6 µs per loop

%timeit le = pd.Series(data=[a[a <= i].count() for i in a])
100 loops, best of 3: 2.74 ms per loop

%timeit a.apply(lambda x: a[a.le(x)].count())
100 loops, best of 3: 3.09 ms per loop

which implies that apply is slow, as well as OP's way is also not the best.

Upvotes: 0

jezrael
jezrael

Reputation: 862611

Faster is numpy solution - convert Series to numpy array and compare by broadcasting to 2d array, last count True values by sum:

b = a.values
#pandas 0.24+
#b = a.to_numpy()
le = pd.Series((b <= b[:, None]).sum(axis=1), index=a.index)

Details:

print (b <= b[:, None])
[[ True False  True False  True  True  True False]
 [ True  True  True  True  True  True  True  True]
 [False False  True False  True  True  True False]
 [ True False  True  True  True  True  True False]
 [False False False False  True  True  True False]
 [False False False False False  True  True False]
 [False False False False False  True  True False]
 [ True False  True  True  True  True  True  True]]

le = pd.Series([a.le(i).sum() for i in a])

le = a.apply(lambda i: a.le(i).sum())

print(le)
0    5
1    8
2    4
3    6
4    3
5    2
6    2
7    7
dtype: int64

Performance:

np.random.seed(2019)
N = 10**6
s = pd.Series(np.random.randint(100, size=N))
#print (s)

In [173]: %%timeit
     ...: b = a.values
     ...: le = pd.Series((b <= b[:, None]).sum(axis=1), index=a.index)
     ...: 
78.6 µs ± 510 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [174]: %%timeit
     ...: le = pd.Series([a.le(i).sum() for i in a])
     ...: 
3.22 ms ± 136 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [175]: %%timeit
     ...: le = a.apply(lambda i: a.le(i).sum())
     ...: 
3.35 ms ± 290 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [176]: %%timeit
     ...: a.apply(lambda x: a[a.le(x)].count())
     ...: 
     ...: 
5.41 ms ± 457 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [177]: %%timeit
     ...: le = pd.Series(data=[a[a <= i].count() for i in a])
     ...: 
4.91 ms ± 281 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Upvotes: 3

Alex
Alex

Reputation: 7045

You could use apply and a lambda function:

In [4]: a.apply(lambda x: a[a.le(x)].count())
Out[4]: 0    5
        1    8
        2    4
        3    6
        4    3
        5    2
        6    2
        7    7
        dtype: int64

Upvotes: 1

Related Questions