CODEWITHSUNDEEP
pythonserializationmarshallingpickle
tunarob
tunarob

Reputation: 3038

Python - the best way to serialize data with high performance?

I need to serialize a data which requires high performance. Separated thread will be accessing it each second and must to load data to memory. There will be about 1000 - 10000 dictionary-like entries about user sessions (id, sessid, login date). Some data will be frequently updated because login time has some time.

These data will be shared between python server and Django application. I think to use pickle or its faster version cPickle. I also found marshal.

What is the best way to do that? Is cPickle efficient enough? Or maybe marshal?

EDIT: Very important thing is access time. It's going to be a realtime websocket server so I need very low delays. Is it faster to access cPickle data once a second or to connect to database like Redis?

Upvotes: 2

Views: 3698

Answers (3)

user4966166
user4966166

Reputation:

redis still requires serializing any complex python object, so redis doesn't solve this problem unless you represent all your data as simple keys and simple values. redis is not a deserialization solution, it's just a data store for strings. and in any case, redis is one of the slower options: https://charlesleifer.com/blog/completely-un-scientific-benchmarks-of-some-embedded-databases-with-python/

Upvotes: 5

Ali Afshar
Ali Afshar

Reputation: 41643

Use a real database in memory. Don't use pickle, cPickle or marshal, or anything like that.

Upvotes: 0

Burhan Khalid
Burhan Khalid

Reputation: 174622

A better approach may be to use some in-memory cache - memcached, if your needs are simple - or something with a bit more of a feature set, like redis.

Upvotes: 4

Related Questions