CODEWITHSUNDEEP
cassandraapache-spark
tesnik03
tesnik03

Reputation: 1359

how can I keep my spark RDD updated?

I am working with apache Spark with cassandra. I am able to create RDDs and get the results. This I am doing using Calliope.

I tried finding a lot about how I can keep my RDD updated without manually refreshing it. I found few examples that can save data in Cassandra using calliope cassandra steraming but I didn't find anything to fetch the data every few seconds and keep my RDD updated. Let me know if anyone has idea about it.

Upvotes: 0

Views: 264

Answers (1)

rs_atl
rs_atl

Reputation: 8985

You wouldn't want to frequently requery your data using Spark, because it has no choice but to reload the entire table every time (it has no way to know what was updated). If you want to handle constant updates, you should send your updates into a queue, then use Spark streaming to process them.

Upvotes: 1

Related Questions