CODEWITHSUNDEEP
apache-sparkspark-streaminghadoop-yarndstream
scalacode
scalacode

Reputation: 1106

Spark streaming error during job runtime in cluster (yarn resource manager)

I am facing the following error:
I wrote an application which is based on Spark streaming (Dstream) to pull messages coming from PubSub. Unfortunately, I am facing errors during the execution of this job. Actually I am using a cluster composed of 4 nodes to execute the spark Job.

After 10 minutes of the job running without any specific error, I get the following error permanently:

ERROR org.apache.spark.streaming.CheckpointWriter:
Could not submit checkpoint task to the thread pool executor java.util.concurrent.RejectedExecutionException: Task org.apache.spark.streaming.CheckpointWriter$CheckpointWriteHandler@68395dc9 rejected
from java.util.concurrent.ThreadPoolExecutor@1a1acc25
[Running, pool size = 1, active threads = 1, queued tasks = 1000, completed tasks = 412]

Upvotes: 6

Views: 338

Answers (0)

Related Questions