CODEWITHSUNDEEP
publish-subscribezeromq
leothelegend
leothelegend

Reputation: 11

Slow Subscriber

I am newbie to ZMQ ZMQ Version - 2.2.1 Ubuntu - 10.04 I am using the PUB-SUB pattern for communication between multiple publishers and multiple subscribers. A forwarder is used to subscribe data from multiple publishers and the same is published to all the subscribers. Currently, if three publishers are running and if each publisher sends 1000 messages in 1second via the PUB channel. The subscriber receives the data, stores it and writes to a database every 1second. Because of the involvement of database, the rate at which subscriber receives the data is getting delayed, as a result the memory usage (RAM) increases by 6-7MB every 1second. Finally the subscriber gets killed by OS due to OOM I tried using the options ZWQ_HWM & ZMQ_SWAP on both the sockets of forwarder. But still the issue persists. Is there any solution for this???

Upvotes: 1

Views: 349

Answers (1)

Pieter Hintjens
Pieter Hintjens

Reputation: 6669

Overall your problem is that your database cannot keep up with your publisher. 0MQ cannot solve this for you. You need an architectural solution based on changing the behavior of your system, presumably the way you do inserts.

You have a few options:

  • Use a faster database
  • Use a faster database insert method
  • Write to a log which is processed asynchronously by another process
  • Change to a socket pattern that lets the receivers tell the senders that they are backed up, so the senders pause (if that's possible)

I think in your case the spool-to-disk-file option is the best.

Upvotes: 1

Related Questions