CODEWITHSUNDEEP
apache-sparkcloudera-cdhcloudera-manager
Chandan
Chandan

Reputation: 764

Spark History Server not starting

I have installed CDH 5.4.7 on my 3 node cluster. After running the first job on Spark I checked for application history page. It was written as following

Event log directory: hdfs://****:8020/user/spark/applicationHistory

No completed applications found!

Did you specify the correct logging directory? Please verify your setting of 
spark.history.fs.logDirectory and whether you have the permissions to access 
it. It is also possible that your application did not run to completion or 
did not stop the SparkContext.

I checked the HDFS and found that /user/spark/applicationHistory was already there. But there was no entry inside that directory. That means no log has been written. I searched cloudera documentation page and found the article Managing the Spark History Server on the following link

Managing Spark History Server

As described I have added a Spark History Server and started it. Executed the following two commands for my user

$ sudo -u hdfs hadoop fs -chown -R spark:spark /user/spark
$ sudo -u hdfs hadoop fs -chmod 1777 /user/spark/applicationHistory

However, when I tried to execute the following command it gives no such file or directory error

$ cp /etc/spark/conf/spark-defaults.conf.template /etc/spark/conf/spark-defaults.conf

So, I went to the path /etc/spark and listed the files inside that. It showed something like this

conf -> /etc/alternatives/spark-conf

Neither I could create dir named conf because it is already there nor I can change directory to /etc/spark/conf

Also service spark-history-server start command gives unrecognized service error.

Kindly help! Thanks in advance

Upvotes: 2

Views: 6003

Answers (1)

Shrey
Shrey

Reputation: 41

I was facing the same issue on Cloudera Quickstart VM 5.12.0 image and able to sort out the issue by following below steps:

  • Stop the History Server:

$ sudo service spark-history-server stop

  • Set ownership and permissions on /user/spark/applicationHistory/ directory in HDFS and as follows:

$ sudo -u hdfs hadoop fs -chown -R spark:spark /user/spark

$ sudo -u hdfs hadoop fs -chmod 777 /user/spark/applicationHistory

  • Add below lines to /etc/spark/conf/spark-defaults.conf file to log the events:

spark.eventLog.enabled=true

spark.eventLog.dir=hdfs://quickstart.cloudera:8020/user/spark/applicationHistory

  • Add below line to /etc/spark/conf/spark-defaults.conf file to link YARN ResourceManager directly to the Spark History Server:

spark.yarn.historyServer.address=http://quickstart.cloudera:18088

  • Start the History Server:

$ sudo service spark-history-server start

Hope that it will be useful to others.

Upvotes: 2

Related Questions