How to overcome the 'ValueError: Wrong number of items passed 2, placement implies 1' error?

Question

I have a time series df:

    Menge     Dates      Time       month
    19.5    2018-01-01  00:00:00     Jan
    19.0    2018-01-01  00:15:00     Jan
    19.5    2018-01-01  00:30:00     Jan
    19.5    2018-01-01  00:45:00     Jan
    21.0    2018-01-01  01:00:00     Jan
    19.5    2018-01-01  01:15:00     Jan
    20.0    2018-01-01  01:30:00     Jan
    23.0    2018-01-01  01:45:00     Jan
    20.5    2018-01-01  02:00:00     Jan
    84.5    2018-01-02  02:00:00     Jan
    80.0    2018-01-02  02:15:00     Jan
    75.5    2018-01-02  02:30:00     Jan
    72.0    2018-01-02  02:45:00     Jan
    70.0    2018-01-02  03:00:00     Jan
    69.0    2018-01-02  03:15:00     Jan
    67.5    2018-01-02  03:30:00     Jan
    67.0    2018-01-02  03:45:00     Jan
    66.0    2018-01-02  04:00:00     Jan
    189.5   2018-02-06  07:00:00     Feb
    188.0   2018-02-06  07:15:00     Feb
    190.5   2018-02-06  07:30:00     Feb
    192.0   2018-02-06  07:45:00     Feb
    185.5   2018-02-06  08:00:00     Feb
    182.5   2018-02-06  08:15:00     Feb
    178.0   2018-02-06  08:30:00     Feb
    189.5   2018-02-06  08:45:00     Feb
    181.0   2018-02-06  09:00:00     Feb
    161.0   2018-02-06  21:00:00     Feb
    159.0   2018-02-06  21:15:00     Feb
    163.5   2018-02-06  21:30:00     Feb
    162.5   2018-02-06  21:45:00     Feb
    163.0   2018-02-06  22:00:00     Feb
    162.5   2018-02-06  22:15:00     Feb
    162.5   2018-02-06  22:30:00     Feb
    162.0   2018-02-06  22:45:00     Feb
    158.5   2018-02-06  23:00:00     Feb

I am trying to calculate the hourly, daily and monthly average for the same.

I have done:

data['month'] = [d.strftime('%b') for d in data.Dates]

to get the month column and following that I am doing:

data_nan_dropped = data.dropna(axis = 0)
data_nan_dropped.Dates = pd.to_datetime(data_nan_dropped.Dates)
data_nan_dropped.Time = pd.to_datetime(data_nan_dropped.Time, format='%H:%M:%S')
hourly_mean = data_nan_dropped.groupby([data_nan_dropped.Dates, data_nan_dropped.Time.dt.hour]).mean()
monthly_mean = data_nan_dropped.groupby(data_nan_dropped.month).mean()
daily_mean = data_nan_dropped.groupby([data_nan_dropped.Dates]).mean()

This code works absolutely fine, but what I want is to add these hourly, monthly, daily means column to my dataframe data_nan_dropped and for that I tried editing the above code as:

data_nan_dropped['hourly_mean'] = data_nan_dropped.groupby([data_nan_dropped.Dates, data_nan_dropped.Time.dt.hour]).transform('mean')
data_nan_dropped['monthly_mean'] = data_nan_dropped.groupby(data_nan_dropped.month).transform('mean')
data_nan_dropped['daily_mean'] = data_nan_dropped.groupby([data_nan_dropped.Dates]).transform('mean')

The data_nan_dropped['hourly_mean'] works perfectly fine and a new column named hourly_mean is created in my dataframe.

But for the monthly_mean & daily_mean, I get the following error:

Traceback (most recent call last):

  File "<ipython-input-5-159d11ea8819>", line 1, in <module>
    data_nan_dropped['daily_mean'] = data_nan_dropped.groupby([data_nan_dropped.Dates]).transform('mean')

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\pandas\core\frame.py", line 3370, in __setitem__
    self._set_item(key, value)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\pandas\core\frame.py", line 3446, in _set_item
    NDFrame._set_item(self, key, value)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\pandas\core\generic.py", line 3172, in _set_item
    self._data.set(key, value)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\pandas\core\internals\managers.py", line 1056, in set
    self.insert(len(self.items), item, value)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\pandas\core\internals\managers.py", line 1158, in insert
    placement=slice(loc, loc + 1))

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\pandas\core\internals\blocks.py", line 3095, in make_block
    return klass(values, ndim=ndim, placement=placement)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\pandas\core\internals\blocks.py", line 87, in __init__
    '{mgr}'.format(val=len(self.values), mgr=len(self.mgr_locs)))

ValueError: Wrong number of items passed 2, placement implies 1

What has to be changed for overcoming this error?

Thanks!

Answer 1

I was playing around with the code again and the following code is giving me the correct answer:

data_nan_dropped.Dates = pd.to_datetime(data_nan_dropped.Dates)
data_nan_dropped.Time = pd.to_datetime(data_nan_dropped.Time, format='%H:%M:%S')
data_nan_dropped['monthly_mean'] = data_nan_dropped.groupby(data_nan_dropped['month'])['Menge'].transform('mean')
data_nan_dropped['hourly_mean'] = data_nan_dropped.groupby([data_nan_dropped['Dates'], data_nan_dropped['Time'].dt.hour])['Menge'].transform('mean')
data_nan_dropped['daily_mean'] =data_nan_dropped.groupby(data_nan_dropped['Dates'])['Menge'].transform('mean')

Answer 2

The problem here is you should point out which columns to get the mean, here I assume it is Menge

data_nan_dropped['hourly_mean'] = data_nan_dropped.groupby([data_nan_dropped.Dates, data_nan_dropped.Time.dt.hour])['Menge'].transform('mean')

Answer 3

Given that both your Dates and Time columns are datetime64[ns] then:

df = data.dropna(axis = 0)
df['Dates'] = pd.to_datetime(df['Dates'])
df['Time'] = pd.to_datetime(df['Time'])

df['month_mean'] = df.groupby(df['Dates'].dt.month).transform('mean')
df['hourbyday_mean'] =df.groupby([df['Dates'].dt.day, df['Time'].dt.hour]).transform('mean')
df['day_mean'] =df.groupby(df['Dates'].dt.day).transform('mean')

Will give you three new columns with their respective mean.