CODEWITHSUNDEEP
apache-sparkbatch-processingspark-streaming
keypoint
keypoint

Reputation: 2318

Spark Streaming: long queued/active batches

could anyone please point out what's the cause of this active batches hanging there for many weeks and never being processed? Thanks a lot.

My guess is not enough executors, and more workers/executors will solve the problem? Or Spark assign priority on different batches within its task scheduler?

But the situation here is, very recent batches (end of June) got processed successfully, but batches in May still being queued.

I just checked my Spark setting, scheduler policy is FIFO

spark.scheduler.mode    FIFO

enter image description here

Upvotes: 5

Views: 2077

Answers (2)

Hassan Ahmadkhani
Hassan Ahmadkhani

Reputation: 31

in spark-submit set --diver-memory and --executor-memory to appropriate values base on your events count such that 10000M

Upvotes: 1

keypoint
keypoint

Reputation: 2318

It turns out that the master node is the bottleneck.

Master node is short of memory, and then maybe scheduler cannot process fast enough.

Solution: change master node to be a more powerful EC2 instance

Upvotes: 2

Related Questions