CODEWITHSUNDEEP
zeromq
mmehra
mmehra

Reputation: 51

How to push all data to late subscribers?

I would like to know if zmq already solves following problem (or) the application sitting on top of zmq needs to take care of it.

1) A central publisher which publishes data to all subscribers. This data is static in nature, something like configuration. The data can be modified at any point in time.
2) Multiple subscribers subscribe to messages from this publisher. The publisher can join at any point in time.
3) If data changes, publisher should just publish the diff to the existing subscribers.
4) If a subscriber joins later, the publisher should publish all the data (current configuration) to the new subscriber.

Zeromq guide suggests following for solving Slow Joiner syndrome but this does not solve above problem. http://zguide.zeromq.org/page:all#Slow-Subscriber-Detection-Suicidal-Snail-Pattern

Upvotes: 3

Views: 1182

Answers (2)

Pieter Hintjens
Pieter Hintjens

Reputation: 6669

The Clone pattern from the Guide does precisely what you want.

Upvotes: 3

Timothy Shields
Timothy Shields

Reputation: 79461

The problem I'm seeing with your setup is that it requires all the subscribers to have the same state. If all subscribers are at version 7 and you publish the 7-to-8 diff, then they all update to version 8. But this requires a tightly-coupled state synchronization between nodes. How would you handle the case when subscribers get out of sync?

Consider this alternative setup:

  • the "publisher" has a single ROUTER socket that it binds
  • each "subscriber" has a single DEALER socket that connects to the ROUTER
    • can't use a REQ socket because that would prohibit the sending of "update-hints" (details to follow)
  • when a subscriber i joins the network, it sends an "update" request to the publisher, so that the publisher is aware of the subscriber's identity and his current version version[i]
    • the publisher responds with the diffs necessary to bring subscriber i up to date
  • if data changes on the publisher (i.e., a new version) it sends an "update-hint" to all of the known subscribers
    • when a subscriber receives an "update-hint," it performs an "update" request
  • (optional) subscribers periodically send an "update" request (infrequent polling)

This approach has the following benefits:

  • the publisher is server; the subscribers are clients
  • the publisher never initiates the sending of any actual data - it only responds to requests from clients (that is, the "update-hints" don't count as sending actual data)
  • the subscribers are all independently keeping themselves up to date (eventual consistency) even though they may be out of sync intermittently

Upvotes: 2

Related Questions