Beniamino Del Pizzo
Beniamino Del Pizzo

Reputation: 893

Combination of Spark context and streaming context

I would like to perform some batch computation after the stop of the streaming context. (strContext.stop(true,true)

I think that the stop of the streaming context will cause the stop of the associated spark context in fact I'm receiving an errore about the spark listener bus. It seems that even the call to strContext.stop(false,true) will cause the stop of the execution of the spark context.

Can anyone provide an example of batch computation after a streaming ?

Upvotes: 0

Views: 258

Answers (1)

Raju Bairishetti
Raju Bairishetti

Reputation: 354

def stop(stopSparkContext: Boolean, stopGracefully: Boolean): Unit = {
    ...
    if (stopSparkContext) sc.stop()
}

From the above code, spark Context should not be stopped if you pass strContext.stop(false).

Are you stopping some where else in you code. Can you paste your code here?

Upvotes: 1

Related Questions