CODEWITHSUNDEEP
creplicationzeromq
jrichemont
jrichemont

Reputation: 61

zmq pattern for reliable multicast

I am struggling to work out how to use zmq to implement the architecture I need. I have a classic publish/subscribe situation except that once client x has subscribed to a topic I need the topic data to be sent to it to be cached if the client dies and resent on reconnect. The data order is important and I can't miss messages should the client be offline for a while.

The PUB/SUB pattern doesn't seem to know about individual clients and will just stop sending to client x if it dies. Plus I can't find out this has happened and cache the messages, or know when it reconnects.

To try to get around this I used the REQ/REP pattern so the clients can announce themselves and have some persistence but this is not ideal for a couple of reasons:

1) The clients must constantly ask "got any data for me?" which offends my sensibilities

2) What happens if there's no data to send to client x but there is to client y? Without zmq I'd have had a thread per client and simply block the one with no data but I can't block client x without also blocking client y in a single thread.

Am I trying to shove a round peg in a square hole, here? Is there some way I can get feedback from PUB saying 'failed to send to client x'? so I can cache the messages instead? Or is there some other pattern I should be using?

Otherwise it's back to low level tcp for me...

Many thanks;

Jeremy

Upvotes: 3

Views: 2801

Answers (3)

jorgen
jorgen

Reputation: 737

In some sense you are trying to shove a round peg in a square hole. You have choosen the tool - PUB/SUB - and are trying to solve a problem it are not designed to solve, at least not without some additional design.

The PUB/SUB is an unreliable broadcast. The client can miss messages for several reasons:

  • Subscribers join late, so they miss messages the server already sent.
  • Subscribers can fetch messages too slowly, so queues build up and then overflow.
  • Subscribers can drop off and lose messages while they are away.
  • Subscribers can crash and restart, and lose whatever data they already received.
  • etc...

For REQ/RSP the client do not have to constantly ask "got any data for me?", instead the client should probably acknowledge every data so that the server can send correct data next time. If the server has nothing to send, it is just quite. eg.

client               server
Hello ---------------->
                     (wait until something exist to send)
<-------------------- Msg 1
Ack 1 ---------------->
                     (wait ...)
<-------------------- Msg 2
                      ...

There are several good ways to do what you want with zmq. First of all you should try to design your protocol. What shall happen when I connect? Should I get any old messages then? If so, how old? If i miss a message when I am connected, should I be able to get it? If the client restarts, should I then get any old messages?

I strongly recommend the very good zmq guide http://zguide.zeromq.org/page:all that have a lot of very good information regardning different ways to get reliability in a protocol. Read the complete guide, including the chapters 4 and 5 whih discuss different techniques on getting a reliable transport. Based on your problem discussion: the Chapter 5 seems like a good start. Try out some of the examples. Then design your protocol.

Upvotes: 1

AShelly
AShelly

Reputation: 35600

What about adding an Archiver process. Part of a Client's subscription process would be to also notify the Archiver to start archiving the same subscription(s). The Archiver would keeps all the messages received in an ordered list.

The Clients would record the time or id of the last published message they received. When they started after a crash, they would first contact the Archiver and say "Give me all messages since X". And they would resubscribe with the Publisher. When a client receives the same message from both the Publisher and the Archiver, it tells the Archiver to stop replaying.

The Archiver could purge messages older then the max expected down time for an offline client. Or alternately, Clients could periodically check in to say "I am up to date with message Y", allowing purging of all older items.

Upvotes: 0

Steve Emmerson
Steve Emmerson

Reputation: 7842

This is an area of active research.

I'm currently working on something similar. Our solution is to have a TCP "back channel" on which to receive missed data and have the subscribers know what the last successfully received publication was so that when they reconnect, they can ask for publications since that one.

Upvotes: 1

Related Questions