CODEWITHSUNDEEP
python
erickque
erickque

Reputation: 126

Best way to store data temporarily?

I'm building a sort of monitoring tool in Python where I want to keep certain stats for a short period of time. I only want to keep a maximum of, say, 30 entries for a stat, and for the older entries to be overwritten as new ones come in. This way, only the 30 most recent entries are kept. What sort of file should I use for this (I'll have multiple different stats all of which I would like to only keep their recent history. The stats are updated at regular intervals ~15 seconds).

I want this to be in a file as the data will be handled in another program.

Upvotes: 1

Views: 1646

Answers (2)

larsks
larsks

Reputation: 311248

If you're only keeping a small number of samples (and you don't care about historic data), then the simplest solution is keeping data in memory. You can use a collections.deque object, as described here, to create a fixed-length list that will automatically drop older items as you add newer items.

For situations in which you want to keep data for longer periods (or you simply want it to persist in the event your application restarts, or you want to be able to access the data from multiple applications, etc), people often use a dedicated time series database, such as InfluxDB, Prometheus, Graphite, or any of a number of other solutions.

Upvotes: 2

sniperd
sniperd

Reputation: 5274

You probably want to keep it all in memory. But if you need to keep a file that mimics and a data structure (say a dictionary) I've had great success with pickle. It's easy, and it's fast.

https://pythontips.com/2013/08/02/what-is-pickle-in-python/

Alternatively a more enterprise solution would be to simply store your stats in a database.

Upvotes: 0

Related Questions