CODEWITHSUNDEEP
pythonsave
crazyduck
crazyduck

Reputation: 147

How to temporary save data in Python?

I read position data from a GPS Sensor in a dictionary, which I am sending in cyclic interval to a server.

If I have no coverage, the data will be saved in a list. If connection can be reestablished, all list items will be transmitted.

But if the a power interruption occurs, all temp data elements will be lost.

What would be the best a pythonic solution to save this data? I am using a SD card as storage, so i am not sure, if writing every element to a file would be the best solution.

Current implementation:

stageddata = []
position = {'lat':'1.2345', 'lon':'2.3455', 'timestamp':'2020-10-18T15:08:04'}
if not transmission(position):
   stageddata.append(position)
else:
   while stageddata:
     position = stageddata.pop()
     if not transmission(position):
        stageddata.append(position)
        return

EDIT: Finding the "best" solution may be very subjective. I agree with zvone, a power outage can be prevented. Perhaps a shutdown routine should save the temporary data. So question may be how to pythonic save a given list to a file?

Upvotes: 1

Views: 6881

Answers (3)

Steuerman
Steuerman

Reputation: 71

I agree with the comment of zvone. In order to know the best solution, we would need more information.

The following would be a robust and configurable solution.

import os
import pickle


backup_interval = 2
backup_file = 'gps_position_backup.bin'


def read_backup_data():
    file_backup_data = []
    if os.path.exists(backup_file):
        with open(backup_file, 'rb') as f:
            while True:
                try:
                    coordinates = pickle.load(f)
                except EOFError:
                    break
                file_backup_data += coordinates

    return file_backup_data


# When the script is started and backup data exists, stageddata uses it
stageddata = read_backup_data()


def write_backup_data():
    tmp_backup_file = 'tmp_' + backup_file
    with open(tmp_backup_file, 'wb') as f:
        pickle.dump(stageddata, f)
    os.replace(tmp_backup_file, backup_file)
    print('Wrote data backup!')


# Mockup variable and method
transmission_return = False
def transmission(position):
    return transmission_return


def try_transmission(position):
    if not transmission(position):
        stageddata.append(position)
        if len(stageddata) % backup_interval == 0:
            write_backup_data()
    else:
        while stageddata:
            position = stageddata.pop()
            if not transmission(position):
                stageddata.append(position)
                return
            else:
                if len(stageddata) % backup_interval == 0:
                    write_backup_data()


if __name__ == '__main__':
    # transmission_return is False, so write to backup_file
    for counter in range(10):
        position = {'lat':'1.2345', 'lon':'2.3455'}
        try_transmission(position)

    # transmission_return is True, transmit positions and "update" backup_file
    transmission_return = True
    position = {'lat':'1.2345', 'lon':'2.3455'}
    try_transmission(position)

I moved your code into some some functions. With the variable backup_interval, it is possible to control how often a backup is written to disk.

Additional Notes:

  • I use the built-in pickle module, since the data does not have to be human readable or transformable for other programming languages. Alternatives are JSON, which is human readable, or msgpack, which might be faster, but needs an extra package to be installed. The tempfile is not a pythonic solution, as it cannot easily be retrieved in case the program crashes.
  • stageddata is written to disk when it hits the backup_interval (obviously), but also when transmission returns True within the while loop. This is needed to "synchronize" the data on disk.
  • The data is written to disk completely new every time. A more sophisticated approach would be to just append the newly added positions, but then the synchronizing part, that I described before, would be more complicated too. Additionally, the safer temporary file approach (see Edit below) would not work.

Edit: I just reconsidered your use case. The main problem here is: Restoring data, even if the program gets interrupted at any time (due to power interruption or whatever). My first solution just wrote the data to disk (which solves part of the problem), but it could still happen, that the program crashes the moment when writing to disk. In that case the file would probably be corrupt and the data lost. I adapted the function write_backup_data(), so that it writes to a temporary file first and then replaces the old file. So now, even if a lot of data has to be written to disk and the crash happens there, the previous backup file would still be available.

Upvotes: 2

Galpaccru
Galpaccru

Reputation: 57

Maybe saving it as a binary code could help to minimize the storage. 'pickle' and 'shelve' modules will help with storing objects and serializing (To serialize an object means to convert its state to a byte stream so that the byte stream can be reverted back into a copy of the object), but you should be carefull that when you resolve the power interruption it does not overwrite the data you have been storing, with open(file, "a") (a== append), you could avoid that.

Upvotes: 0

mpSchrader
mpSchrader

Reputation: 932

A good solution for temporary storage in Python is tempfile.

You can use it, e.g., like the following:

import tempfile

with tempfile.TemporaryFile() as fp:

     # Store your varibale
     fp.write(your_variable_to_temp_store)

     # Do some other stuff 

     # Read file
     fp.seek(0)
     fp.read()

Upvotes: 3

Related Questions