CODEWITHSUNDEEP
design-patternsredispublish-subscribegoogle-cloud-pubsub
big
big

Reputation: 1938

Redis pub sub for multiple producers and multiple consumers

Lets say there are N producers and M users which subscribes to these N producers. Here N producer produce N different types of messages e.g 

producer1 produces  messageType1, 
producer2 produces  messageType2,
producer3 produces  messageType3,
.
.
. 
producerN produces  messageTypeN.

M users can subscribe to these messages. One user can subscribe to multiple types of messages. E.g.

user1 consumes (messageType1, messageType2, messageType10)
user2 consumes (messageType14, messageType5)
.
.
userM consumes (messageType21, messageType22, messageType23, .... messageTypeN)

Users may consume same or distinct message types. My questions is how to design this scenario. It looks like pub sub pattern. For this scenario, do I have to create channels per user in redis. If yes, there is a limitation on number of redis channel one can create (10K). In that case how to handle millions of user? Any help would be appreciated.

Upvotes: 0

Views: 4566

Answers (1)

for_stack
for_stack

Reputation: 22886

In the pub/sub scenario, you should create channels per producer. Each user subscribes channels of the corresponding producers.

User Side

// user1
subscribe producer1 producer2
// user2
subscribe producer2

Producer Side

// producer1
publish producer1 message1
// producer2
publish producer2 message2

The limitation is NOT the number of channels you can create, but the number of client connections. You CANNOT have millions of users connect to a single Redis instance at the same time.

A possible solution

In order to achieve that, you have to create multiple Redis instances, and shard users into shardings. Each Redis instance creates a full list of producers, and handles connections from only one sharding of users.

When producing messages, you can publish the message on the corresponding channel of each Redis instances, so that users who subscribed the channel, can receive the message.

Upvotes: 1

Related Questions