CODEWITHSUNDEEP
apache-spark
Sudo
Sudo

Reputation: 651

Spark Clusters: worker info doesn't show on web UI

I have installed spark standalone on a set of clusters. And I tried to launch clusters through the cluster launch script. I have added cluster's IP address into conf/slaves file. The master connects to all slaves through password-less ssh. After running ./bin/start-slaves.sh script, I get the following message:

starting org.apache.spark.deploy.worker.Worker, logging to /root/spark-0.8.0-incubating/bin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-jbosstest2.out

But the webUI of the master (localhost:8080) is not showing any information about the worker. But when I add localhost entry onto my conf/slaves file the worker info of localhost is shown.

There are no error messages, the message on terminal says the worker is started, but the WebUI is not showing any workers.

Upvotes: 12

Views: 14594

Answers (5)

Tyler Zhang
Tyler Zhang

Reputation: 1

I met the same issue and finally solved by adding the following line in $SPARK_HOME/conf/spark-env.sh:

SPARK_MASTER_HOST=your_master_ip_address

Upvotes: 0

helseth
helseth

Reputation: 1

You can use: ./start-master.sh --host 192.168.x.x instead of changing the file: /etc/hosts

Upvotes: 0

ramisetty.vijay
ramisetty.vijay

Reputation: 31

When you run the cluster, check command $jps in worker nodes, whether its up correctly and check it in the logs with the worker's PID.

or

set the following: run the cluster and check if the ports are up or not with your configured ports

export SPARK_MASTER_WEBUI_PORT=5050
export SPARK_WORKER_WEBUI_PORT=4040

Upvotes: 2

Jimmy
Jimmy

Reputation: 2280

check your /etc/hosts and see the bindings for master

If your master is binding to localhost as well as ip address (eg 192.168.x.x), remove localhost. if you have local host intact master will be mapped to localhost which wont allow slaves to connect to master Ip address

Upvotes: 0

davilj
davilj

Reputation: 101

I had the same problem. I noticed when I could not telnet master:port from the slaves. In my etc/hosts file (on master) I had a 127.0.0.1 master entry (before my 192.168.0.x master). When I removed the 127.0.0.1 entry from my etc/hosts file I could telnet and when I start-slaves.sh (from the master) my slaves connected

Upvotes: 6

Related Questions