CODEWITHSUNDEEP
pythonpandasdataframepandas-groupby
Carl
Carl

Reputation: 121

How to sum a column over unique values in two other column with Pandas?

I need to find the sum of a column given the unique values of two other columns in a Dataframe.

Example Code to emulate what I'm trying to do.

import numpy as np
import pandas as pd
def makeDictArray(**kwargs):
    data = {}
    size = kwargs.get('size',20)
    strings = 'What is this sentence about? And why do you care?'.split(' ')
    bools = [True,False]
    bytestrings = list(map(lambda x:bytes(x,encoding='utf-8'),strings))
    data['ByteString'] = np.random.choice(bytestrings, size)
    data['Unicode'] = np.random.choice(strings, size)
    data['Integer'] = np.random.randint(0,500,size)
    data['Float'] = np.random.random(size)
    data['Bool'] = np.random.choice(bools,size)
    data['Time'] = np.random.randint(0,1500,size)

    return data

def makeDF(**kwargs):
    size = kwargs.get('size',20)
    data = makeDictArray(size=size)
    return pd.DataFrame(data)

x = makeDF(size=1000000)
x['SUM'] = 0.
xx = x.groupby(['Time','Integer'])['Float'].agg('sum')

Here is xx:

Time  Integer
0     0          0.826326
      1          4.897836
      2          5.238863
      3          6.694214
      4          6.791922

1499  495        5.621809
      496        7.385356
      497        4.755907
      498        6.470006
      499        3.634070
Name: Float, Length: 749742, dtype: float64

What I've tried:

uniqueTimes = pd.unique(x['Time'])
for t in uniqueTimes:
    for i in xx[t].index:
        idx = (x['Time'] == t) & (x['Integer'] == i)
        if idx.any():
            x.loc[idx,'SUM'] = xx[t][i]

Which gives me the correct result, but I want to put the values of the sum back into 'x' in the newly created "SUM" column. I can achieve this by doing a double for loop, however, this is slow and seems to not be the 'pandas way'.

Anyone have any suggestions?

Upvotes: 1

Views: 1197

Answers (1)

Kent Shikama
Kent Shikama

Reputation: 4060

If I understand the problem correctly, you want to do a standard merge across x and xx on ["Time", "Integer"]. 

x = makeDF(size=5)
xx = x.groupby(['Time','Integer'])['Float'].agg('sum')

pd.merge(x, xx.to_frame(name="SUM"), on=["Time", "Integer"])

Output

  ByteString Unicode  Integer     Float   Bool  Time       SUM
0  b'about?'    this      209  0.116809  False  1418  0.116809
1     b'why'      is       12  0.043745   True  1159  0.043745
2   b'care?'   care?      493  0.479680  False   487  0.479680
3  b'about?'  about?      102  0.503759  False   335  0.503759
4     b'And'   care?      197  0.394406  False   207  0.394406

Upvotes: 1

Related Questions