CODEWITHSUNDEEP
pythonfile
Pascal
Pascal

Reputation: 45

What is the best way to store big amount of custom Python objects in a file?

I need some help to find the best way to store a big amount of data (1~2GB). The data source is a raw binary file containing network application packets exchanged between two device.

The packet class is defined in Python by myself (see below).

I would like to store my objects in such a way that I could later read the file packet by packet and not byte by byte

class AppPacket:
    def __init__(self, version=0, command=0, flags=0, seq=0, pldlen=0, pld=[]):
        self.Version = np.uint8(version)
        self.Command = np.uint8(command)
        self.Flags = np.uint16(flags)
        self.SequenceNumber = np.uint16(seq)
        self.PayloadLength = np.uint16(pldlen)
        self.Payload = np.uint8(pld)
        self.CRC8 = np.uint8(0)

Reading byte by byte and parsing the data to rebuild each packet take at least 30minutes for 750MB. I expect to reduce this time as much as possible

Upvotes: 0

Views: 206

Answers (1)

Pascal
Pascal

Reputation: 45

To close this subject, as suggested by @Kris, the best way is to used a database. Since Python is offering native SQLite3 module, I choose to use it with SQLite Studio for DB management.

I use executemany() statement and multithreading to improve the performance during the storing process.

See : https://www.tutorialspoint.com/sqlite/sqlite_python.htm

Thanks to you :)

Upvotes: 1

Related Questions