CODEWITHSUNDEEP
amazon-web-servicesstreamamazon-kinesis
Rahul Gupta
Rahul Gupta

Reputation: 1802

How do I combine multiple kinesis data streams into 1 data stream?

I have data coming in from 3 different servers(3 data streams). Is it possible to merge these data (kind of update/upsert) in kinesis consumer application and get the updated data as output?

The data I have from stream 2,3 is dependent on stream 1. For example

Stream 1 (ItemDetails) - {itemId, itemName, itemdescription},
stream 2 (ItemtoCart) - {ItemId},
stream 3 (Itemordered) - {ItemId}.

Final Stream output I am expecting is

OutputStream - {itemId, itemName, itemdescription, itemtoCart_flag, itemOrdered_flag}

Stream 1 is receiving the data at the rate of 10K records/sec.

Upvotes: 0

Views: 1786

Answers (1)

prayagupadhyay
prayagupadhyay

Reputation: 31252

Say there are three streams as below,

stream                   event in stream
stream1(ItemPurchased) - {"item" : 1, "totalQuantity": 100}

stream2(ItemOrdered)   - {"item" : 1, "sold": 1}

stream3(ItemCancelled) - {"item" : 1, "orderCancelled": 1}

The streams are for Item purchase, then sold and or cancelled.

Say, I wanna build a final state of item available quantity from these events. What I would do is,

  • consume each event in stream / kinesis has lambda functionality but not sure how easily it talks to non aws datastores like MongoDB, Cassandra
  • and have a logic to build a final state based on event

State transition table

stream                   events                                  consumer/onEvent          state (could be MongoDB, Cassandra)
stream1(ItemPurchased) - {"item" : 1, "totalQuantity": 100}   -> create new state      -> {"item" : 1, "availableQuantity": 100} 

stream2(ItemOrdered)   - {"item" : 1, "sold": 1}              -> decrease the quantity -> {"item" : 1, "availableQuantity": 100 - 1} 

stream3(ItemCancelled) - {"item" : 1, orderCancelled: 1}      -> increase the quantity -> {"item" : 1, "availableQuantity": 99 + 1}

Hope that answers your question, but unlike you asked it is the final state table not a stream.

Upvotes: 1

Related Questions