CODEWITHSUNDEEP
apache-sparkspark-streaming
AKC
AKC

Reputation: 1023

How to Stop running Spark Streaming application Gracefully?

How Do I stop Spark Streaming? My Spark Streaming job is running continuously. I want to stop in a graceful manner.

I have seen the below option to shut down the streaming application.

sparkConf.set("spark.streaming.stopGracefullyOnShutdown","true")

Spark configuration: available properties

But, how do I update this parameter on a running application?

Upvotes: 12

Views: 16637

Answers (1)

Glennie Helles Sindholt
Glennie Helles Sindholt

Reputation: 13154

Have a look at this blogpost. It it the "nicest" way to gracefully terminate a streaming job I have come across.

How to pass Shutdown Signal :

Now we know how to ensure graceful shutdown in spark streaming. But how can we pass the shutdown signal to spark streaming. One naive option is to use CTRL+C command at the screen terminal where we run driver program but obviously its not a good option. One solution , which i am using is , grep the driver process of spark streaming and send a SIGTERM signal . When driver gets this signal, it initiates the graceful shutdown of the application. We can write the command as below in some shell script and run the script to pass shutdown signal :

ps -ef | grep spark | grep | awk '{print $2}' | xargs kill -SIGTERM

e.g. ps -ef | grep spark | grep DataPipelineStreamDriver | awk '{print $2}' | xargs kill -SIGTERM

Upvotes: 18

Related Questions