CODEWITHSUNDEEP
javac++network-programmingreplication
Justin Breitfeller
Justin Breitfeller

Reputation: 13801

Are there any frameworks to synchronize data generated on one peer with all other peers in an unreliable network?

We are developing a system with the following requirements.

To put it another way, each system needs to replicate its data to N peer systems. Ideally, this will be done in an intelligent manner.

I have considered looking into database synchronization frameworks, but I am concerned that it is overkill for this problem. I don't think there is any possibility for row conflicts because each system's data is entirely independent of other systems.

The question is, do you know of any frameworks that could help solve this problem? Or possibly a way to phrase this issue that might help me down a path to discover a solution.

Finally, ideally, this framework would be in C++ (and potentially, java).

Upvotes: 11

Views: 3950

Answers (5)

Lesto
Lesto

Reputation: 2299

What you are looking for is called a "distributed database", and they are extensively used even in production system; http://www.project-voldemort.com/ for example, is used by linkedin

As p2p network like DHT and Kadmelia ARE key->value database, there are also some P2P database, where new node are automatically added and the failure resistence of any node is strong, as those network resistance and scalability is proven

So just look on your preferred search engine for "p2p database" and "distributed database", and you will find a lot of implementation.

Upvotes: 0

Austin Brougher
Austin Brougher

Reputation: 526

SymmetricDS.org

The solution you are looking for sounds a lot like the open source software SymmetricDS.

"SymmetricDS is an asynchronous data replication software package that supports multiple subscribers and bi-directional synchronization. It uses web and database technologies to replicate tables between relational databases, in near real time if desired. The software was designed to scale for a large number of databases, work across low-bandwidth connections, and withstand periods of network outage."
-SymmetricDS.org

Symmetric was designed to be used as a Java library, as well as a stand alone application. Used with a lightweight database like H2, you could avoid your overkill scenario. H2 can optionally be run embedded within an application and can store data in memory or to disk.

Disclaimer: I recently started working for JumpMind, the company that develops this software.

Upvotes: 2

UmNyobe
UmNyobe

Reputation: 22910

It seems you want to implementing a reliable broadcast for your peer communication. Check out the library J.N. provided, and if it is not sufficient (or you want to modify it) there are some algorithms in this book.

Check Causal Order Broadcast and Total Order Broadcast.

My teacher at the univ did implement such a library, I will update when I find it.

Upvotes: 0

MattDavey
MattDavey

Reputation: 9017

Interesting problem. Many of the issues you've described lend themselves particularly well to the BitTorrent protocol.

Upvotes: 1

J.N.
J.N.

Reputation: 8431

0mq. It is a C framework with a C++ interface. It notably supports EPGM (reliable multicast over UDP) and N-to-N connections. Though, there will be work to do for your special use case.

Upvotes: 2

Related Questions