CODEWITHSUNDEEP
scalaapache-sparkhadoop-yarn
Ruxi Zhang
Ruxi Zhang

Reputation: 323

spark-shell ERROR YarnScheduler: Lost executor

Inside of spark-shell I keep getting the following error message in between my comment submitting. What are these possibly about?

scala> 16/04/12 12:49:14 ERROR YarnScheduler: Lost executor 2 on XXXXXXXX: remote Rpc client disassociated 16/04/12 12:49:14 ERROR YarnScheduler: Lost executor 5 on XXXXXXXX: remote Rpc client disassociated 16/04/12 12:49:16 ERROR YarnScheduler: Lost executor 4 on XXXXXXXX: remote Rpc client disassociated

Upvotes: 1

Views: 4383

Answers (1)

J Maurer
J Maurer

Reputation: 1044

If you have dynamic resource allocation configured, these errors will show up when unneeded resources are released to Yarn. ERROR level seems misleading to me, perhaps INFO would be more appropriate. http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation

edit: This should be fixed in Spark 1.6 https://issues.apache.org/jira/browse/SPARK-4134

Upvotes: 1

Related Questions