Reputation: 1802
I have data coming in from 3 different servers(3 data streams). Is it possible to merge these data (kind of update/upsert) in kinesis consumer application and get the updated data as output?
The data I have from stream 2,3 is dependent on stream 1. For example
Stream 1 (ItemDetails) - {itemId, itemName, itemdescription},
stream 2 (ItemtoCart) - {ItemId},
stream 3 (Itemordered) - {ItemId}.
Final Stream output I am expecting is
OutputStream - {itemId, itemName, itemdescription, itemtoCart_flag, itemOrdered_flag}
Stream 1 is receiving the data at the rate of 10K records/sec.
Upvotes: 0
Views: 1782
Reputation: 31252
Say there are three streams as below,
stream event in stream
stream1(ItemPurchased) - {"item" : 1, "totalQuantity": 100}
stream2(ItemOrdered) - {"item" : 1, "sold": 1}
stream3(ItemCancelled) - {"item" : 1, "orderCancelled": 1}
The streams are for Item purchase, then sold and or cancelled.
Say, I wanna build a final state of item available quantity from these events. What I would do is,
State transition table
stream events consumer/onEvent state (could be MongoDB, Cassandra)
stream1(ItemPurchased) - {"item" : 1, "totalQuantity": 100} -> create new state -> {"item" : 1, "availableQuantity": 100}
stream2(ItemOrdered) - {"item" : 1, "sold": 1} -> decrease the quantity -> {"item" : 1, "availableQuantity": 100 - 1}
stream3(ItemCancelled) - {"item" : 1, orderCancelled: 1} -> increase the quantity -> {"item" : 1, "availableQuantity": 99 + 1}
Hope that answers your question, but unlike you asked it is the final state table not a stream.
Upvotes: 1