CODEWITHSUNDEEP
hazelcaststream-processinghazelcast-jetevent-stream-processing
siddhadev
siddhadev

Reputation: 16631

Are Hazelcast Jet Reliable Topic Sinks idempotent? (Hazelcast fault-tolerance of a websocket source)

I cannot find this in the Hazelcast Jet 5.0 (or 4.x) documentation, so I hope someone can answer this here - can a reliable topic be used as an idempotent sink, for example to de-duplicate events coming from two identical unreliable sources (like a websocket). Or should I use an explicit event de-duplication as suggested at https://hazelcast.com/blog/stream-deduplication-with-hazelcast-jet/? Or is there a better way to cope with unreliable sources like websockets (I mean for the case I don't want to miss events ingested over a websocket, and there is non-zero chance that a single websocket instance might fail)?

Upvotes: 0

Views: 50

Answers (2)

siddhadev
siddhadev

Reputation: 16631

I ended up using a putIfAbsent() on an IMap journal - I thinks that's much simpler (and somewhat obvious) for my use case than the de-duplication solution linked above.

Upvotes: 0

Oliv
Oliv

Reputation: 10812

Any queue can't in general be used for de-duplication. If you offer the same item twice, it has no means to ignore such call, for that it would have to store the identifiers from the entire history, or you have to specify storage limits like in the example you linked where the TTL attribute of filterStateful is used.

Upvotes: 0

Related Questions