CODEWITHSUNDEEP
pythonjsonsqlitepickle
Malcolm
Malcolm

Reputation: 5323

Best way to save complex Python data structures across program sessions (pickle, json, xml, database, other)

Looking for advice on the best technique for saving complex Python data structures across program sessions.

Here's a list of techniques I've come up with so far:

Pickle is the easiest and fastest technique, but my understanding is that there is no guarantee that pickle output will work across various versions of Python 2.x/3.x or across 32 and 64 bit implementations of Python.

Json only works for simple data structures. Jsonpickle seems to correct this AND seems to be written to work across different versions of Python.

Serializing to XML or to a database is possible, but represents extra effort since we would have to do the serialization ourselves manually.

Thank you, Malcolm

Upvotes: 10

Views: 3444

Answers (4)

denis
denis

Reputation: 21947

What are your criteria for "best" ?

  • pickle can do most Python structures, deeply nested ones too
  • sqlite dbs can be easily queried (if you know sql :)
  • speed / memory ? trust no benchmarks that you haven't faked yourself.

(Fine print:
cPickle.dump(protocol=-1) compresses, in one case 15M pickle / 60M sqlite, but can break.
Strings that occur many times, e.g. country names, may take more memory than you expect; see the builtin intern().
)

Upvotes: 2

rnicholson
rnicholson

Reputation: 4588

Have you looked at PySyck or pyYAML?

Upvotes: 2

SpliFF
SpliFF

Reputation: 38976

You left out the marshal and shelve modules.

Also this python docs page covers persistence

Upvotes: 4

Ned Batchelder
Ned Batchelder

Reputation: 375654

You have a misconception about pickles: they are guaranteed to work across Python versions. You simply have to choose a protocol version that is supported by all the Python versions you care about.

The technique you left out is marshal, which is not guaranteed to work across Python versions (and btw, is how .pyc files are written).

Upvotes: 15

Related Questions