CODEWITHSUNDEEP
redisstream
Drew Dara-Abrams
Drew Dara-Abrams

Reputation: 8054

joining Redis streams

I am very intrigued by Redis streams. (Looks like the potential to build little systems powered by append-logs, like Kafka, but without all the overhead of Kafka.)

It looks straightforward to XADD to a log/stream and to consume an entry from a log/stream. But what about if you want to join across two streams?

Kafka Streams, Flink, Spark, etc. provide means for doing this. Is there an equivalent in the Redis universe?

If not, I guess I'll just need to implement my own thing that consumes from two streams, does its own join logic from the messages, and publishes back out to a new stream. If others have experience doing this with Redis Streams, please do share your pointers or warnings.

Upvotes: 8

Views: 895

Answers (2)

Cr4zyTun4
Cr4zyTun4

Reputation: 765

I have attended the Redis University Stream course and they does not show any built in method to join messages from two streams.

I guess one work-around would be to handle the IDs in a proper way. Let's make an example:

You produce messages to stream A with even IDs.
You prodce messages to stream B with odd IDs.
You consume from stream A and stream B and produce to a new stream C being careful on respecting the order because it is not allowed to produce an ID equal or smaller than the larger already present in the stream.

In this whay you achieve the join.
Looking forward to better answer not using external libraries

Upvotes: 0

Dennis Jaheruddin
Dennis Jaheruddin

Reputation: 21563

If I am correct, you are looking for a way to join two Redis streams.

It seems there is a connector available for Spark, that allows you to consume the streams https://github.com/RedisLabs/spark-redis/blob/master/doc/streaming.md

From here Spark logic for the join should be easy to use.

Upvotes: 1

Related Questions