CODEWITHSUNDEEP
hadoopapache-spark
Pavel Bernshtam
Pavel Bernshtam

Reputation: 4444

Spark listens on localhost

I installed spark on a cluster of machines w/o public DNS (just created machines on a cloud). Hadoop looks to be installed and worked correctly, but Sparks listens on 7077 and 6066 as 127.0.0.1 instead of public ip so worker nodes can't connect to it.

What is wrong?

My /etc/hosts on the master node looks like:

127.0.1.1 namenode namenode
127.0.0.1 localhost
XX.XX.XX.XX namenode-public
YY.YY.YY.YY hadoop-2
ZZ.ZZ.ZZ.ZZ hadoop-1

My $SPARK_HOME/conf/spark-env.sh looks like:

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export SPARK_PUBLIC_DNS=namenode-public
export SPARK_WORKER_CORES=6
export SPARK_LOCAL_IP=XX.XX.XX.XX

sudo netstat -pan|grep 7077

tcp        0      0 127.0.1.1:7077          0.0.0.0:*               LISTEN      6670/java

Upvotes: 0

Views: 2161

Answers (1)

cfg
cfg

Reputation: 26

You should specify SPARK_MASTER_HOST in spark-env.sh (it must be the address of your machine that is visible to the slave nodes). Moreover, you may need to add rules for ports 7077 and 6066 in iptables.

Upvotes: 1

Related Questions