CODEWITHSUNDEEP
apache-spark
worldterminator
worldterminator

Reputation: 3076

spark Failed to get block(s) error

I run a very large job with spark. I got error like

15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_18444_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_18504_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_18692_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_18779_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_18818_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_18877_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_18923_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_19109_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_19125_7412, and will not retry (0 retries)
15/07/15 18:58:17 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from bhdp4005.prod.bdd.jp.local:53161
15/07/15 18:58:17 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block shuffle_7_19312_7412, and will not retry (0 retries)

I don't know what cause this kind of error. I am using Spark 1.4.0 with environment: --driver-memory 60G --num-executors 130 --master yarn-cluster --executor-memory 45G

Upvotes: 3

Views: 5706

Answers (1)

ArunV
ArunV

Reputation: 11

ShuffleBlockFetcherIterator failed to fetch exception is noticed when a running executor tries accessing a block which is no more available. This can be because the previous executor got killed or the block is now inaccessible.

Upvotes: 1

Related Questions