CODEWITHSUNDEEP
mongodbapache-sparkspark-streaming
Faaiz
Faaiz

Reputation: 685

How to send mongodb collection updates in Spark streaming context?

How to send the data from a mongodb collection whenever it gets updated to spark streaming context. I have seen socketTextStream to write the data to mongodb. But is there any way to read the stream when the collection gets updated? Is there any way to avoid to implement Custom Receiver for mongodb ? Or If anyone has already implemented, then it will be nice, if someone can share.

Thanking you for any input on this.

Upvotes: 0

Views: 172

Answers (1)

dumitru
dumitru

Reputation: 2108

One way to achieve this is using the CDC pattern where on one side you have the reactive mongo library that can hook onto a collection and receive events like update, insert, delete from the mongodb and on the other side you have a spark streaming application that will listen on this events.

The transport engine of the events can be any publish/subscriber system (e.g Kafka)

Upvotes: 1

Related Questions