CODEWITHSUNDEEP
queuepartitioningdistributed-system
Farzin Nasiri
Farzin Nasiri

Reputation: 730

How to Partition a Queue in a distributed system

This problem accrued to me a while ago, unfortunately, I could not find the answer I was looking for on the web. Here is the problem statement:

Consider a simple producer-consumer environment where we only have one producer writing to a queue and one consumer reading from it. Now since the objects written on the queue are quite large in size and our available resources are not much on our current machine, we decided to implement a distributed queue system where the data inside the queue is partitioned among multiple nodes. It is important to us that the total ordering is conserved while pushing and poping the data, meaning that from the point of a user this distributed queue acts just like a single unified queue.

Before giving a solution to this problem we have to ask if high availability is more important to us or portion tolerance. I believe in both versions, there are interesting challenges to tackle and I thought that such a question must surely be raised before, however, after searching for existing solutions I could not find a complete and well-thought-out answer from an algorithmic or scientific point of view. Most of what I found were engineering and high-level approaches, leveraging tools like Kafka, RabitMQ, Redis etc.

So the problem remains and I would be thankful if you could share with me your designs, algorithms and thoughts on this problem or point me to some scientific journal or article etc that has already tackled such a problem.

Upvotes: 0

Views: 549

Answers (1)

satyaprakash
satyaprakash

Reputation: 151

enter image description here

This can be one of the ways in which the above can be achieved. Here the partitioning is achieved in the round-robin fashion.

To achieve high availability, you can have partition replicas.

Pros:-

  1. By adding replicas system becomes highly available.
  2. Multi-consumer groups can be implemented

Cons:- route table becomes the single source of failure, hence redundancy can be achieved via using dynamo DB & consistent read here.

Upvotes: 1

Related Questions