CODEWITHSUNDEEP
pythonpython-3.xdatabase
Luca
Luca

Reputation: 1810

Building and accessing a database runtime

My code is acquiring data from a sensor and it's performing some operations based on the last N-minutes of data.

At the moment I just initialize a list at the beginning of my code as

x = []
while running:
    new_value = acquire_new_point()
    x.append(new_value)

    do_something_with_x(x)

Despite working, this has some intrinsic flaws that make it sub-optimal. The most important are:

Some obvious solutions exist as:

I have the feeling, however, that this could be a problem for which something more specific solution has been already created. My first thought went to HDF5, but I'm not sure it's the best candidate for this problem.

So, what is the best strategy/format for a database that needs to be written (appended) run-time and needs to be accessed partially (only the last N-minutes)? Is there any tool that is specific for a task like this one?

Upvotes: 0

Views: 210

Answers (1)

AKX
AKX

Reputation: 168976

I'd simply just use SQLite with a two-column table (timestamp, value) with an index on the timestamp.

This has the additional bonus that you can use SQL for data analysis, which will likely be faster than doing it by hand in Python.

If you don't need the historical data, you can periodically DELETE FROM data WHERE TIMESTAMP < ....

Upvotes: 1

Related Questions