CODEWITHSUNDEEP
design-patternspublish-subscribezeromqdata-distribution-service
user1259642
user1259642

Reputation: 701

Event-driven publish subscribe model

I am working on a requirement whereby a process (say producer) needs to send out one-way messages to a variable number of processes (say consumers).

The publish-subscribe model seemed good for this because the consumers will subscribe to messages from the producer. I tried using ZeroMQ to achieve this.

However, I have a few problems with it:

  1. The consumers have to continuously poll for messages. I would have the consumers to be notified when there is a new message.

  2. There is a possibility of the producer queue being filled up. I would have liked the producer to remove messages from the queue based on some condition (say remove messages older than 5 seconds, or remove messages that have been read 5 times).

  3. Since the consumers are polling and the messages are not removed from the queue, the consumers see duplicate messages till a new message comes in. I want the consumer to be notified only once per new message.

I understand I may be using a wrong model (publish-subscribe may not be suitable). I have thought about using request-reply, but that doesn't work since the producer does not want to keep track of the number of consumers.

Can anyone suggest a good alternative?

Upvotes: 1

Views: 3039

Answers (4)

Ertan D.
Ertan D.

Reputation: 832

DDS (Data Distribution Service) middleware supports exactly what you are trying to achieve and much more easily.

Directly answering your questions:

  1. DDS supports a listener mechanism, your subscribers does not need to continuously poll.

  2. DDS has good QoS settings to prevent publisher queues being filled up. You can use History QoS to say "keep only the latest 10 samples in the queue", or you can use Lifespan QoS to say "keep only the samples published in the last 10 seconds".

  3. Again, you can use the DDS listener mechanism and you will be notified only once for each new sample. No need to poll.

There are currently two open source implementations.

Upvotes: 1

Madhusudanan
Madhusudanan

Reputation: 11

I suggest to go for Push-Pull model with Broker between Producer and Consumer.

  1. Broker should be notified for any new message.
  2. Consumers will listen for brokers to notify(Keep table to track success/failure.So duplication will be avoided during retry)
  3. Once #2 is done then Consumer can pull the data from Producer(source) and send the ack to broker for the success/failure

Hope this helps

Upvotes: 1

scaganoff
scaganoff

Reputation: 1850

Try using a JMS provider or an AMQP provider. These have some of the things you are looking for with Topics:

  1. Push notification to subscribers.

  2. A time-to-live attribute on messages which allows messages to be deleted or put onto a dead-letter queue if it is not consumed within the TTL.

  3. Once-only notification - depending on your configuration.

Note that once-only messaging does have edge conditions in the event of network failuer which can result in either a lost message or a duplicate message...you choose.

In terms of which providesr to use. RabbitMQ is popular for AMQP. For JMS there are any number of proprietary products or open-source implementations.

Upvotes: 0

Dalton Barreto
Dalton Barreto

Reputation: 601

Do you need more than one producer? If not you could use PUSH/PULL instead of PUB/SUB.

With PUSH/PULL you can have as many consumers as you want (They are the PULL side of the model). All messages written to the PUSH endpoint are distributed round-robin style among all connected consumers. This also assures that two consumers do not receive the same message.

As you described, having consumers as the SUB endpoint you can end up delivering the same message to more than one consumer (assuming this would be a problem in your model) if two or more consumers are subscribed to the same "prefix".

Assuming "prefix" is the string you pass to sock.setsockopt(ZMQ_SUBSCRIBE, "prefix", ...);

Upvotes: 0

Related Questions