Python embedded data persistence that's simple, modifiable, and reliable

Question

Looking for something for Python 2.7 that is:

• Simple - Get up and running quickly without a lot of overhead
• Modifiable - Can handle the changes that come with prototyping and frequent changes to data schema
• Reliable - Able to recover no matter what, i.e. if there's a power outage I'm not going to lose the entire data store because it's corrupt.

Some candidates and comments:

• Text / CSV / JSON file - Seems simple, could be somewhat reliable if you keep a backup copy of the file frequently, or implement a simple backup-plus-journal setup (make a backup of the file when the program starts, write all changes to a journal file)
• SQLite - Simple and reliable, but handling frequent changes seems a little cumbersome.
• Shelve / Marshal / cPickle - Simple and modifiable, not sure how reliable it's going to be. If the file ends up corrupted, seems like you're SOL. At least with a human-readable file you could piece it back together.
• ZODB - Might be a winner. I'm unclear, does it require your objects to inherit? I'd like to keep a good separation of the persistence layer since this is for initial prototyping and may change once the schema is hashed out.
• SQLAlchemy - For a small project, seems like a lot of overhead, and you might as well just use SQLite. Maybe it's better with Elixir?

If MongoDB was embedded that might be perfect. SQLite would be great if I could solve the frequently changing schema problem. What fits the bill based on your experiences?

Answer 1

ZODB offers full ACID compliance, it fits the reliability requirements.

Any object that does inherit from Persistent becomes it's own separate record (changes recorded as a unit), but you don't have to. Using Persistent is not required, but it is recommended.

The ZODB heavily relies on the pickle module, so __setstate__ hooks can be used for schema upgrades.

Use the ZODB if your data structure fits the inherent tree structure that ZODB data naturally builds. Indexing is an add-on operation, one that you need to handle yourself. Zope / Plone uses events and a dedicated catalog to index interesting information from objects, then lets you efficiently find those objects again in a large hierarchy.

If your data is more table-like in nature (piles of different information with complex inter-relations), stick with a SQL solution. Text / CSV / JSON / shelve / plain pickle is not going to be reliable enough.

SQLAlchemy is easy enough to use if you know your SQL; it is an excellent ORM. Elixir is no longer maintained, SQLAlchemy itself now offers all the functionality that project offered, natively. To manage schema migrations you could look into sqlalchemy-migrate.

For simple projects using SQLite directly is easy enough, but migrating schema is a little more cumbersome. I've used the schema_version pragma before to detect when a schema migration was needed.

Answer 2

I'd recommend Google Protocol Buffers

• Simple: yes, there's a complete library.
• Modifiable: yes, if the schema evolves, versions using an older schema will ignore what's not in their schema.
• Reliable: I guess this depends on your requirements here, you will likely need extra care, yes.

Bonus:

• Supported in lots of languages, which gives you freedom to use your data with different tools.

---

If reliability is a major blocker, I think you want to go with sqlite.