CODEWITHSUNDEEP
graphite
knoopx
knoopx

Reputation: 17400

Manually aggregate historical data when importing it into whisper

I'm trying to import a two years old archive of data into whisper and I'm having problems trying to understand the imported data as I think some of the values are getting lost. Do I need to manually aggregate the data for timestamps older than the first schema-defined retention?

Upvotes: 2

Views: 1403

Answers (1)

Valor
Valor

Reputation: 1315

I've not tried that with latest version (0.9.10), but already tried what you say with 0.9.9 and had the same problem.

Let's suppose you have a 30 days one item per minute and then one per day for a year (60s:30d,1d:1y), then when you submit value 50 for a minute two months ago, whisper will see where the timestamp fits, will see it fits in the 1d metric and will store it's value on that item, if you send the next minute, it will get the same point on the whisper file and will just overwrite your last metric.

The workaround I found is to change your schema while importing to the less period you have on your schema for the longest time your schema support, in the example above that would be 60s:1y. Then import all your data, when you end importing, run whisper-resize.py on your whisper file to convert it back to the right schema (60s:1d,1d:1y). That will do the trick, but be aware of choosing the right aggregation method when doing this.

Good Luck!

Guzmán

Upvotes: 3

Related Questions