CODEWITHSUNDEEP
apache-sparkjupyter-notebookazure-eventhub
Alon Catz
Alon Catz

Reputation: 2530

How to troubleshoot package loading error in spark

I'm using spark in HDInsight with Jupyter notebook. I'm using the %%configure "magic" to import packages. Every time there is a problem with the package, spark crashes with the error:

The code failed because of a fatal error: Status 'shutting_down' not supported by session..

or

The code failed because of a fatal error: Session 28 unexpectedly reached final status 'dead'. See logs:

Usually the problem was with me mistyping the name of the package, so after a few attempts I could solve it. Now I'm trying to import spark-streaming-eventhubs_2.11 and I think I got the name right, but I still receive the error. I looked at all kinds of logs but still couldn't find the one which shows any relevant info. Any idea how to troubleshoot similar errors? 

%%configure -f
{ "conf": {"spark.jars.packages": "com.microsoft.azure:spark-streaming-eventhubs_2.11:2.0.5" }}

Additional info: when I run 

spark-shell --conf spark.jars.packages=com.microsoft.azure:spark-streaming-eventhubs_2.11:2.0.5

The shell starts fine, and downloads the package

Upvotes: 1

Views: 7785

Answers (1)

Alon Catz
Alon Catz

Reputation: 2530

I finally was able to find the log files which contain the error. There are two log files which could be interesting

  • Livy log: livy-livy-server.out
  • Yarn log

On my HDInsight cluster, I found the livy log by connecting to one of the Head nodes with SSH and downloading a a file at this path (this log didn't contain useful info):

/var/log/livy/livy-livy-server.out

The actual error was in the yarn log file accessible from YarnUI. In HDInsight Azure Portal, go to "Cluster dashboard" -> "Yarn", find your session (KILLED status), click on "Logs" in the table, find "Log Type: stderr", click "click here for full log".

The problem in my case was Scala version incompatibility between one of the dependencies of spark-streaming_2.11 and Livy. This is supposed to be fixed Livy 0.4. More info here

Upvotes: 2

Related Questions