CODEWITHSUNDEEP
apache-spark
pg2455
pg2455

Reputation: 5178

How to open Spark UI when working on a server?

I am using Spark on my server with Ubuntu. I use the shell only and there is no UI for the server other than that. There is always an option of looking at the stats of Spark like jobs, cpus, memory etc when working on local computer on a web browser.

Is there a way that I can view all that information locally on the server?

Upvotes: 9

Views: 15705

Answers (5)

oma0256
oma0256

Reputation: 103

You can use SSH tunnelling to achieve this, for example if the Spark Web UI on your remote server is exposed on port 4040 you can run the command ssh -N -L 7000:127.0.0.1:4040 [email protected] on your local machine terminal in order to access the Spark Web UI using http://127.0.0.1:7000/ on your browser.

Upvotes: 0

Arjun Bhattacharya
Arjun Bhattacharya

Reputation: 101

Suppose your spark master web ui is running on remote.example.com on port 8082, you can use SSH tunneling to open the web ui on your local machine -

On your local terminal type -

ssh -L 8082:remote.example.com:8082 [email protected]

Now, in your local web browser open http://localhost:8082

You should see the web ui.

Upvotes: 3

szentesmarci
szentesmarci

Reputation: 29

I think a python should be running on the server, so you can use this command: 

python -m webbrowser -n http://host:port

(for example the application detail ui: http://localhost:4040). It gives a shell based browser if the server has no graphical interface

Upvotes: 2

Daniel Darabos
Daniel Darabos

Reputation: 27455

spark-shell always starts a web UI on port 4040 by default. If the server has no graphical interface, you can either use a text-based browser, such as Links, or access port 4040 from another machine, such as your laptop. You may need to run ufw allow 4040 to unblock the port in the firewall.

Upvotes: 8

naveenmad
naveenmad

Reputation: 31

In local mode, you can only view the web UI during the duration of the job. You can also start a history server which stores all the information which you need. I would say try running on standalone mode. Follow the below link to run Spark on standalone mode.

http://spark.apache.org/docs/latest/spark-standalone.html

Basically you need to start three services,

start-master.sh

start-slave.sh

stop-history-server.sh

This will start the history server on 18080 port.

Upvotes: 1

Related Questions