CODEWITHSUNDEEP
redismicroservicesredis-streams
Jovan Perovic
Jovan Perovic

Reputation: 20193

Order of processing different Redis Streams messages

Given that I have a service P (Producer) and C (Consumer) my P service needs to:

From C's perspective, is there a way to guarantee that it had all the info it needed for successful persistence?

I know that services like Kafka and Redis Streams are not generally built for this stuff, but how does one overcome this?

Idea 1:

Idea 2:

Is there any obvious way to handle object dependencies?

Upvotes: 1

Views: 1172

Answers (2)

Hi computer
Hi computer

Reputation: 1121

I believe IDEA 2 is a better solution cuz I think keep the whole message in one data structure is a good idea.

And probably you can try to use multiple keys.

For example:

On Service P

def now_timestamp = datetime.currentstamp # let`s say it is 1515151551

redis sadd not_processed_timestamp 1515151551
redis set X_1515151551 INFO_OF_X
redis set Y_1515151551 INFO_OF_Y
redis set Z_1515151551 INFO_OF_Z

On service C, create a new thread

def new_task_timestamp = redis spop not_processed_timestamp # let`s say it is 1515151551
redis blocking-get X_1515151551
redis blocking-get Y_1515151551
redis blocking-get Z_1515151551

# process the rest

Upvotes: 3

Mahdi Yusuf
Mahdi Yusuf

Reputation: 21028

This is a good question covered in this note about Redis Streams.

We could say that schematically the following is true:

If you use 1 stream -> 1 consumer, you are processing messages in order.

If you use N streams with N consumers, so that only a given consumer hits a subset of the N streams, you can scale the above model of 1 stream -> 1 consumer.

If you use 1 stream -> N consumers, you are load balancing to N consumers, however in that case, messages about the same logical item may be consumed out of order, because a given consumer may process message 3 faster than another consumer is processing message 4.

So basically Kafka partitions are more similar to using N different Redis keys, while Redis consumer groups are a server-side load balancing system of messages from a given stream to N different consumers.

So if you want to process them in order and use more than one consumer you will need to deal with that yourself.

Upvotes: 3

Related Questions