CODEWITHSUNDEEP
apache-sparkapache-spark-sqlsparks-pakage-management
Ritika Garg
Ritika Garg

Reputation: 87

How to access spark history server

I am running my spark application on small dataset just for functional testing. But I also wanted to see how many the executors are being created and how the data is being partitioned. For this, I tried to access the spark UI application but the problem with accessing spark UI application is that the connection is lost as soon as the application completes.

How to access the spark history server to monitor past spark applications. I am running the spark application using intellij IDE and I am not able to find the option to access such server.

Upvotes: 1

Views: 8258

Answers (2)

sathya
sathya

Reputation: 2072

From the Apache Spark Docs, The endpoints are mounted at /api/v1. Eg., for the history server, they would typically be accessible at http://<server-url>:18080/api/v1, and for a running application, at http://localhost:4040/api/v1.

to start: ./sbin/start-history-server.sh

This creates a web interface at http://<server-url>:18080 by default, listing incomplete and completed applications and attempts.

The spark jobs themselves must be configured to log events, and to log them to the same shared, writable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:

Please refer the Monitoring and Instrumentation from https://spark.apache.org/docs/latest/monitoring.html for more Spark version specific options details.

Upvotes: 1

Ghislain Fourny
Ghislain Fourny

Reputation: 7279

How about adding a sleep call at the end of the program to delay its completion?

Upvotes: 1

Related Questions