Aggregate time series in numpy

Question

I am trying to aggregate a variable onto time slots, but I am not sure how to do this in a Pythonic way. My plan is to aggregate it every ten seconds. My time data is the first column and it is stated in seconds from midnight. The second column is the variable I want to aggregate. Most of the entries are zero, but the others are actual numbers that I want to sum.

So the idea is to sum column two for every ten seconds elapsed from column 1. The matrix vector would have the time every ten seconds for the first column and the aggregated variable for the second column.

My first idea was to do a while loop on column two and summing all the numbers while we are inside the same time slot. And place it inside a for loop for each 10 second increment. This seems incredibly heavy and slow.

Any suggestions are really appreciated. Thank you.

Sample data:

3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009535350000078e+04    0.000000000000000000e+00
3.420009723610000219e+04    0.000000000000000000e+00
3.420009926809999888e+04    0.000000000000000000e+00
3.420019142339999962e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420025728430000163e+04    0.000000000000000000e+00
3.420030975760000001e+04    0.000000000000000000e+00
3.420031578240000090e+04    0.000000000000000000e+00
3.420033592689999932e+04    0.000000000000000000e+00
3.420046674769999663e+04    0.000000000000000000e+00
3.420046679600000061e+04    0.000000000000000000e+00
3.420074155890000111e+04    3.000000000000000000e+02
3.420074291200000152e+04    0.000000000000000000e+00
3.420076289620000171e+04    -3.000000000000000000e+02
3.420076459099999920e+04    2.000000000000000000e+02
3.420080930030000309e+04    0.000000000000000000e+00
3.420088430179999705e+04    0.000000000000000000e+00
3.420089157010000054e+04    6.000000000000000000e+00
3.420103050939999957e+04    3.000000000000000000e+02
3.420118753419999848e+04    3.000000000000000000e+02
3.420118888490000245e+04    0.000000000000000000e+00
3.420141332990000228e+04    0.000000000000000000e+00
3.420191020060000301e+04    0.000000000000000000e+00
3.420282872450000286e+04    6.000000000000000000e+00
3.420304732930000318e+04    0.000000000000000000e+00
3.420305702279999969e+04    0.000000000000000000e+00
3.420308584690000134e+04    0.000000000000000000e+00
3.420310770000000048e+04    0.000000000000000000e+00
3.420319724010000209e+04    0.000000000000000000e+00
3.420338125820000278e+04    -6.000000000000000000e+00
3.420341430170000240e+04    0.000000000000000000e+00
3.420341743809999753e+04    0.000000000000000000e+00
3.420365752020000218e+04    -1.000000000000000000e+02
3.420377651160000096e+04    0.000000000000000000e+00
3.420377900809999846e+04    0.000000000000000000e+00
3.420394856640000216e+04    0.000000000000000000e+00
3.420477753839999787e+04    0.000000000000000000e+00
3.420477777210000204e+04    0.000000000000000000e+00
3.420509855940000125e+04    0.000000000000000000e+00
3.420509857180000108e+04    0.000000000000000000e+00
3.420509858389999863e+04    0.000000000000000000e+00
3.420509858600000007e+04    0.000000000000000000e+00
3.420510251910000079e+04    3.000000000000000000e+02
3.420540111949999846e+04    0.000000000000000000e+00
3.420560046090000105e+04    0.000000000000000000e+00
3.420650914059999923e+04    3.000000000000000000e+02
3.420677362859999994e+04    0.000000000000000000e+00
3.420677385650000360e+04    0.000000000000000000e+00
3.420680100729999685e+04    0.000000000000000000e+00
3.420680135709999740e+04    0.000000000000000000e+00
3.420740102699999989e+04    0.000000000000000000e+00
3.420760059799999726e+04    0.000000000000000000e+00
3.420827570510000078e+04    0.000000000000000000e+00
3.420827846660000068e+04    -1.000000000000000000e+02
3.420860100809999858e+04    0.000000000000000000e+00
3.420860318459999689e+04    0.000000000000000000e+00
3.420860717129999830e+04    3.000000000000000000e+02
3.420920105229999899e+04    0.000000000000000000e+00
3.420949705469999753e+04    0.000000000000000000e+00
3.420949705469999753e+04    1.000000000000000000e+02
3.420949874789999740e+04    0.000000000000000000e+00
3.420949964329999784e+04    1.000000000000000000e+02
3.420950525599999673e+04    1.000000000000000000e+02
3.420950532879999810e+04    1.000000000000000000e+02
3.420950537430000259e+04    1.000000000000000000e+02
3.420950638910000271e+04    2.000000000000000000e+02
3.420950643239999772e+04    0.000000000000000000e+00
3.420950668009999936e+04    -2.000000000000000000e+02
3.420950796770000306e+04    0.000000000000000000e+00
3.420951590530000249e+04    2.000000000000000000e+02
3.420951602249999996e+04    1.000000000000000000e+02
3.420951673219999793e+04    -2.000000000000000000e+02
3.420951805140000215e+04    2.000000000000000000e+02
3.420951861829999689e+04    -1.000000000000000000e+02
3.420951993930000026e+04    -2.000000000000000000e+02
3.420951995310000348e+04    0.000000000000000000e+00
3.420952149579999968e+04    -1.000000000000000000e+02
3.420953443149999657e+04    0.000000000000000000e+00
3.420955497840000317e+04    -1.000000000000000000e+02
3.420980103039999813e+04    0.000000000000000000e+00
3.420980107300000236e+04    0.000000000000000000e+00
3.420980108509999991e+04    0.000000000000000000e+00
3.420989885620000132e+04    0.000000000000000000e+00
3.420989886560000014e+04    0.000000000000000000e+00
3.420989887540000200e+04    0.000000000000000000e+00
3.421022118190000037e+04    6.000000000000000000e+00
3.421028878499999701e+04    0.000000000000000000e+00
3.421029493570000341e+04    -6.000000000000000000e+00
3.421030143400000088e+04    6.000000000000000000e+00
3.421040105849999964e+04    0.000000000000000000e+00
3.421048329460000241e+04    -6.000000000000000000e+00
3.421048642470000050e+04    6.000000000000000000e+00
3.421085856629999762e+04    0.000000000000000000e+00
3.421091312309999921e+04    2.000000000000000000e+02
3.421091378969999641e+04    -2.000000000000000000e+02
3.421091715290000138e+04    1.000000000000000000e+02
3.421118993179999961e+04    0.000000000000000000e+00
3.421119293469999684e+04    6.000000000000000000e+00
3.421120108150000306e+04    0.000000000000000000e+00
3.421125067030000355e+04    1.000000000000000000e+02
3.421125067030000355e+04    2.000000000000000000e+02
3.421125072869999713e+04    0.000000000000000000e+00
3.421125074440000026e+04    1.000000000000000000e+02
3.421125075079999806e+04    1.000000000000000000e+02
3.421125081070000306e+04    0.000000000000000000e+00
3.421125213139999687e+04    -2.000000000000000000e+02
3.421125223679999908e+04    1.000000000000000000e+02
3.421128031339999870e+04    -1.000000000000000000e+02
3.421128036189999693e+04    -1.000000000000000000e+02
3.421129058039999654e+04    0.000000000000000000e+00
3.421136299729999882e+04    -6.000000000000000000e+00
3.421136675249999826e+04    6.000000000000000000e+00
3.421144104959999822e+04    -6.000000000000000000e+00
3.421144451439999830e+04    6.000000000000000000e+00
3.421180177070000354e+04    1.000000000000000000e+02
3.421180182290000084e+04    1.000000000000000000e+02
3.421180183370000304e+04    1.000000000000000000e+02
3.421180184079999890e+04    0.000000000000000000e+00
3.421180240860000049e+04    -2.000000000000000000e+02
3.421182461470000271e+04    -1.000000000000000000e+02
3.421182466330000170e+04    -1.000000000000000000e+02
3.421183488509999734e+04    0.000000000000000000e+00
3.421209945889999653e+04    7.500000000000000000e+01
3.421213015650000307e+04    1.000000000000000000e+02
3.421250285479999729e+04    2.500000000000000000e+01
3.421250285479999729e+04    3.000000000000000000e+02
3.421250292720000289e+04    1.000000000000000000e+02
3.421250293499999680e+04    1.000000000000000000e+02
3.421250294089999807e+04    0.000000000000000000e+00
3.421250461730000097e+04    2.000000000000000000e+02
3.421250660069999867e+04    2.000000000000000000e+02
3.421250661260000197e+04    0.000000000000000000e+00
3.421250662810000358e+04    -2.000000000000000000e+02
3.421250862849999976e+04    2.000000000000000000e+02
3.421250865109999722e+04    -2.000000000000000000e+02
3.421251071360000060e+04    2.000000000000000000e+02
3.421251073390000238e+04    -2.000000000000000000e+02
3.421251200200000312e+04    2.000000000000000000e+02
3.421251202500000363e+04    -2.000000000000000000e+02
3.421251290870000230e+04    2.000000000000000000e+02
3.421251292259999900e+04    0.000000000000000000e+00
3.421251417010000296e+04    0.000000000000000000e+00
3.421252315950000047e+04    -3.750000000000000000e+02
3.421253324080000311e+04    -1.000000000000000000e+02
3.421253324419999990e+04    -1.000000000000000000e+02
3.421253325169999880e+04    0.000000000000000000e+00
3.421280107870000211e+04    0.000000000000000000e+00
3.421280108050000126e+04    0.000000000000000000e+00
3.421280109909999737e+04    0.000000000000000000e+00
3.421300649369999883e+04    -1.000000000000000000e+02
3.421300649369999883e+04    -5.000000000000000000e+00
3.421301662930000020e+04    0.000000000000000000e+00
3.421301664270000038e+04    0.000000000000000000e+00
3.421308269650000148e+04    -1.000000000000000000e+00
3.421308569990000251e+04    1.000000000000000000e+00
3.421340113990000100e+04    -2.000000000000000000e+02
3.421360080849999940e+04    0.000000000000000000e+00
3.421360081950000313e+04    0.000000000000000000e+00
3.421429604950000066e+04    0.000000000000000000e+00
3.421429604950000066e+04    2.000000000000000000e+02
3.421465075329999672e+04    1.750000000000000000e+02
3.421465535619999719e+04    0.000000000000000000e+00
3.421480113819999679e+04    2.000000000000000000e+02
3.421500744149999809e+04    2.000000000000000000e+02
3.421500745640000241e+04    -2.000000000000000000e+02
3.421501012210000044e+04    0.000000000000000000e+00
3.421501918660000229e+04    2.000000000000000000e+02
3.421506023750000168e+04    2.000000000000000000e+02
3.421506026279999787e+04    0.000000000000000000e+00
3.421506290029999946e+04    0.000000000000000000e+00
3.421529837140000018e+04    -2.000000000000000000e+02
3.421540116690000286e+04    0.000000000000000000e+00
3.421549945029999799e+04    0.000000000000000000e+00
3.421554515879999963e+04    0.000000000000000000e+00
3.421556687119999697e+04    -3.000000000000000000e+02
3.421556832669999858e+04    0.000000000000000000e+00
3.421560137959999702e+04    0.000000000000000000e+00
3.421591907280000305e+04    0.000000000000000000e+00
3.421640112349999981e+04    0.000000000000000000e+00
3.421660092140000052e+04    0.000000000000000000e+00
3.421780116010000347e+04    0.000000000000000000e+00
3.421860098989999824e+04    0.000000000000000000e+00
3.422180125080000289e+04    0.000000000000000000e+00
3.422181661359999998e+04    0.000000000000000000e+00
3.422946705189999921e+04    0.000000000000000000e+00
3.422947095989999798e+04    3.000000000000000000e+02
3.422947237350000069e+04    -2.000000000000000000e+02
3.422980151579999801e+04    0.000000000000000000e+00
3.423008524700000271e+04    0.000000000000000000e+00
3.423035811159999867e+04    0.000000000000000000e+00
3.423036595089999901e+04    0.000000000000000000e+00
3.423315292909999698e+04    -1.000000000000000000e+00
3.423315298620000249e+04    -1.000000000000000000e+02
3.423315299590000359e+04    -1.000000000000000000e+02
3.423315301110000291e+04    0.000000000000000000e+00
3.423315437930000189e+04    0.000000000000000000e+00
3.423315469930000108e+04    -2.000000000000000000e+02
3.423315568000000349e+04    0.000000000000000000e+00
3.423317673620000278e+04    1.000000000000000000e+02
3.423318700569999783e+04    1.000000000000000000e+02

MaxU - stand with Ukraine · Accepted Answer

Consider the following approach:

import pandas as pd

df = pd.read_csv(filename, header=None, names=['ts','val'])

In [31]: df.groupby(df['ts']//10*10)['val'].sum()
Out[31]:
ts
34200.0    1806.0
34210.0     907.0
34220.0     100.0
34230.0    -201.0
Name: val, dtype: float64

Source DF:

In [33]: df
Out[33]:
               ts    val
0    34200.095354    0.0
1    34200.095354    0.0
2    34200.095354    0.0
3    34200.095354    0.0
4    34200.095354    0.0
5    34200.095354    0.0
6    34200.095354    0.0
7    34200.095354    0.0
8    34200.095354    0.0
9    34200.095354    0.0
..            ...    ...
237  34230.365951    0.0
238  34233.152929   -1.0
239  34233.152986 -100.0
240  34233.152996 -100.0
241  34233.153011    0.0
242  34233.154379    0.0
243  34233.154699 -200.0
244  34233.155680    0.0
245  34233.176736  100.0
246  34233.187006  100.0

[247 rows x 2 columns]

df['ts']//10*10 - will round timestamps up to 10 seconds, so we can use it for groupping later on:

In [34]: df['ts']//10*10
Out[34]:
0      34200.0
1      34200.0
2      34200.0
3      34200.0
4      34200.0
5      34200.0
6      34200.0
7      34200.0
8      34200.0
9      34200.0
        ...
237    34230.0
238    34230.0
239    34230.0
240    34230.0
241    34230.0
242    34230.0
243    34230.0
244    34230.0
245    34230.0
246    34230.0
Name: ts, Length: 247, dtype: float64

Aggregate time series in numpy

Answers (1)

Related Questions