CODEWITHSUNDEEP
azureapache-sparktensorflowmachine-learningazure-hdinsight
Rafael
Rafael

Reputation: 671

Azure HDInsights Spark Cluster Install External Libraries

I have a HDInsights Spark Cluster. I installed tensorflow using a script action. The installation went fine (Success).

But now when I go and create a Jupyter notebook, I get:

import tensorflow

Starting Spark application
The code failed because of a fatal error:
    Session 8 unexpectedly reached final status 'dead'. See logs:
YARN Diagnostics:
Application killed by user..

Some things to try:
a) Make sure Spark has enough available resources for Jupyter to create a Spark context. For instructions on how to assign resources see http://go.microsoft.com/fwlink/?LinkId=717038
b) Contact your cluster administrator to make sure the Spark magics library is configured correctly.

I don't know how to fix this error... I tried some things like looking at logs but they are not helping.

I just want to connect to my data and train a model using tensorflow.

Upvotes: 1

Views: 349

Answers (1)

maxiluk
maxiluk

Reputation: 381

This looks like error with Spark application resources. Check resources available on your cluster and close any applications that you don't need. Please see more details here: https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-resource-manager#kill-running-applications

Upvotes: 1

Related Questions