CODEWITHSUNDEEP
pythondatabaseserializationdatabase-performance
broodrooster
broodrooster

Reputation: 61

Python sensor data storage

I have simple data set (DateTime, int, float) that grows up to 15k items a day and at high points of the day I can have up to 5 additional records a second. Initially I used pickle which obviously is a bad idea (dumping 15k items 5 times a second is too slow). Now i started to look at different data/logging storage options and the sheer amount of choices overwhelmed me a bit.

I see that msgpack and ujson are rather fast but I believe that will leave me with the same problem; I'm appending data to the set a very frequently and the data logging program doesn't really need to have knowledge of the previous data.

My question: I was hoping to get some pointers on what could be a suitable option, I've read about things like redis, mongoDB and noSQL which seem rather quick because of the in-memory property. Or am I overthinking this problem and could I simply have used a traditional database like MySQL or SQLite from the start?

Verdict: overthinking a simple problem, all sloved now.

Upvotes: 0

Views: 787

Answers (1)

holdenweb
holdenweb

Reputation: 37033

If there are no requirements to revise posted data then Cassandra might be a good choice, but that might be a little heavyweight for a Raspberry Pi. Take a look at SQLite, which should be able to handle your storage requirements quite easily and has a driver module (sqliite3) as a part of the standard library.

It sounds as though you are accumulating a bigger and bigger pickle over time, which with a database there's no need to do, and it's quite easy to store datetime values.

Don't be intimidated by the need to learn about database - it sounds complicated but in essence it's just a way to store tables of data.

Upvotes: 1

Related Questions