CODEWITHSUNDEEP
apache-sparkpysparksuse
Claude Falbriard
Claude Falbriard

Reputation: 935

Spark Master and Workers not Connecting via Localhost Addresses

After install of the Spark package at Linux (SuSE SLES 12) I see the following connectivity error ("failed to connect"), which beside the Spark slave process also impacts the "pyspark" examples, rejecting connections. Any hint how to activate the port 7077 connectivity via localhost addresses is welcome. Part of the problem might be the default Linux firewall settings.

Firewall Commands to Open Localhost addresses: 

sudo iptables -A INPUT -s 127.0.0.1 -d 127.0.0.1 -j ACCEPT
sudo iptables -A INPUT -s 127.0.0.1 -d zbra2016 -j ACCEPT

Starting the Spark Master - commands: 

export SPARK_LOCAL_IP=zbra2016  
./sbin/stop-master.sh
./sbin/start-master.sh



16/04/19 10:12:29 INFO Master: Registered signal handlers for [TERM, HUP, INT]
16/04/19 10:12:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/19 10:12:29 INFO SecurityManager: Changing view acls to: linux1
16/04/19 10:12:29 INFO SecurityManager: Changing modify acls to: linux1
16/04/19 10:12:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(linux1); users with modify permissions: Set(linux1)
16/04/19 10:12:30 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
16/04/19 10:12:30 INFO Master: Starting Spark master at spark://zbra2016:7077
16/04/19 10:12:30 INFO Master: Running Spark version 1.6.1
16/04/19 10:12:30 WARN Utils: Service 'MasterUI' could not bind on port 8080. Attempting port 8081.
16/04/19 10:12:30 INFO Utils: Successfully started service 'MasterUI' on port 8081.
16/04/19 10:12:30 INFO MasterWebUI: Started MasterWebUI at http://localhost:8081
16/04/19 10:12:30 INFO Utils: Successfully started service on port 6066.
16/04/19 10:12:30 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
16/04/19 10:12:31 INFO Master: I have been elected leader! New state: ALIVE

Starting the Spark Worker - commands: 

./sbin/stop-slave.sh 
./sbin/start-slave.sh spark://zbra2016:7077

Logfile displays a "Failed to Connect Error Message": 

 /data/spark/spark/logs/spark-linux1-org.apache.spark.deploy.worker.Worker-1-zbra2016.out
16/04/19 10:15:46 INFO Worker: Retrying connection to master (attempt # 1)
16/04/19 10:15:46 INFO Worker: Connecting to master zbra2016:7077...
16/04/19 10:15:47 WARN Worker: Failed to connect to master zbra2016:7077
java.io.IOException: Failed to connect to zbra2016/127.0.0.1:7077

Testing connectivity of alias: zbra2016 = localhost

linux1@zbra2016:/data/spark/spark> ping zbra2016
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.022 ms

Upvotes: 4

Views: 2274

Answers (2)

Chris
Chris

Reputation: 1

You may be able to change the settings allowing port 7077 through your firewall. Try: sudo ufw allow 7077

Upvotes: 0

Claude Falbriard
Claude Falbriard

Reputation: 935

We just found a solution for it in the setup of the Linux iptables firewall. I used the following command to open localhost traffic: 

iptables -I INPUT 1 -p all -s localhost -d localhost -j ACCEPT

Now the worker process is able to connect to the master through the localhost ports.

Upvotes: 1

Related Questions