CODEWITHSUNDEEP
apache-sparkspark-streaming
tomek
tomek

Reputation: 771

Pausing/throttling spark/spark-streaming application

I wrote a simple spark-streaming application which basically reads stream of events from Kafka and stores these events to Cassandra in a table allowing efficient queries over these data. The main purpose of this job is to process current real-time data. But there are also historical events stored in hdfs.

I want to reuse the code processing RDDs (a part of the streaming job) in a historical job and I am wondering what is the best solution for reading historical data according to the following requirements:

  1. Historical events are stored in daily rolled up files in hdfs (I want to run the job on a range of historical files)
  2. It would be nice to have a possibility of pausing the job (inserts to cassandra are idempotent, so I need at-least-once processing)
  3. I want to have some throttling mechanism allowing to define maximum number of events which can be processed (in a period of time: e.g every 1min)

I've considered two approaches so far:

Am I right that regular batch spark cannot meet my requirements? I am waiting for your advice regarding to spark streaming solution.

Upvotes: 3

Views: 1946

Answers (1)

Tathagata Das
Tathagata Das

Reputation: 1808

For Batch Spark job, 1. You can give comma separated file names in sc.***File operations 2, 3. Since you will be able to

For Streaming job, 1. You could define the RDDs for the files and insert them using queueStream. 2. Depends on what you mean by pausing. You could simply stop the streaming context gracefully when you want to pause. 3. Yes, that is it.

But stepping back, you can do a lot of code sharing in the RDD and DStream transformation. Whatever you do for RDDs in your batch part, could be reused within DStream.transform() in your streaming part.

Upvotes: 2

Related Questions