CODEWITHSUNDEEP
serializationdata-structures
Harish
Harish

Reputation: 495

Adding durability to in-memory data structures

What are some of the popular techniques you can adopt to add durability to your in-memory data structures (ie) if the process crashes, you can preserve all previously executed operations on that data structure?

If my data structure involves just a list of tuples, then I would just store them in a SQL DB and that would give me durability for free. But what if my data structure was a graph or a tree?

The one thing I could think of is to explicitly log all operations to disk (append-only log) and in the event of a crash, replay the log to preserve the previous state. If the log becomes too big, then there will be a compaction step. I'm guessing this is what a database engine does internally for durability (checkpointing is what this process is called)?

Btw note that this is not a scenario where the entire dataset doesn't fit in memory.

Upvotes: 9

Views: 725

Answers (6)

Tim Cooper
Tim Cooper

Reputation: 10478

I've implemented the "Mrjb" technology in 2 companies' products, which is basically exactly what you've suggested in your question: a "Memory Resident Journal Backed" database, an in-memory data-structure where every change is logged to disk as it happens. And it works great for us!

http://www.edval.biz/memory-resident-programming-object-databases

I'd be happy to share our real-world experiences with using this in a production context. I love being able to replay an exact sequence of events or roll back to any point in time.

Upvotes: 1

yfeldblum
yfeldblum

Reputation: 65455

You might want to try an object prevalence engine. For .NET, you might want to try Bamboo.Prevalence, which is a port of a similar engine called Prevayler for Java.

Upvotes: 4

Jeff C
Jeff C

Reputation: 445

Any answer to your question will entail doing something like what an ACID database system does. So I would say your best bet is to use an RDBMS to store your application state, updating whenever you have an (application) transaction that must not be lost.

Upvotes: 0

Chris Marasti-Georg
Chris Marasti-Georg

Reputation: 34690

Yes, you would want to serialize the data to some format - xml, binary, whatever. Depending on the programming languagem this may be built in for you. Java has ObjectStreams, .NET has XmlSerializer and also a BinaryFormatter.

Upvotes: 0

pkaeding
pkaeding

Reputation: 37673

You could come up with some way to serialize your structure, whether with XML, YAML, JSON, etc. Then you could either store that in the DB, or perhaps put one big try/catch around the main execution point to the program. Then if some uncaught exception happens, which will cause the program to crash, you could serialize your data, ans well as log any error messages, stack traces, etc.

Upvotes: 0

Rik
Rik

Reputation: 29243

The word you're looking for is "serialization".

Upvotes: 0

Related Questions