Reputation: 749
Scenario is slightly different than the classic stream-stream join
streamA: stream of transactions: transTS, userid, productid,...
streamB: stream of new products created: productid, productname, createTS, ...)
I want to join transactions with productIds, but I can't find a combination of watermarks/join conditions to make that happen.
streamA_wm = streamA.withWatermark("transTS", "3 minutes")
streamB_wm = streamB.withWatermark("createTS", "1 day")
streamA_wm
.join(streamB_wm, "productId AND transTS >= createTS", "leftOuter")
The result is empty.
What am I doing wrong?
Upvotes: 1
Views: 265
Reputation: 18128
I think you may have the wrong approach here. Whilst products are transactional when created and updated, they are metadata relative the other Transaction Stream.
I would suggest the following:
But may be there is a reason for a Stream for Products, but ... What happens if no more updates to Product made and you get data for that Product again via the Stream for Transactions?
Upvotes: 1