CODEWITHSUNDEEP
apache-sparkbigdataspark-streamingdynamic-allocation
Amar AttilaZz
Amar AttilaZz

Reputation: 242

Spark Streaming - dynamic allocation do not remove executors in middle of windows interval

I have a spark streaming job with batch Interval of 10 minutes and slides/window of 1 hour, I have activated the dynamic allocation with Spark.

But executors get removed after 1 hours and not after each 10 min of Batch Interval, So I have to wait 1 hour to see executors removed (in case of no data to process).

Is there a way to tell spark to revise excecutors removal after each batch interval instead of Window interval ?

Thanks.

Upvotes: 1

Views: 1228

Answers (1)

Ravikumar
Ravikumar

Reputation: 1131

Please check executor idle timeout property settings in job level or default config (spark-defaults.conf)

  • spark.dynamicAllocation.executorIdleTimeout
  • spark.dynamicAllocation.cachedExecutorIdleTimeout

Refer this link for more details.

Upvotes: 3

Related Questions