CODEWITHSUNDEEP
scalanetwork-programmingraspberry-piapache-spark
blue-sky
blue-sky

Reputation: 53806

Change bound IP running on port 7077 - Apache Spark

Can Spark be configured so that instead of binding to address 127.0.1.1 for port 7077, can instead be bound to 0.0.0.0 . In same way as port 8080 is bound :

netstat -pln
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.1.1:7077          0.0.0.0:*               LISTEN      2864/java
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      2864/java
tcp        0      0 127.0.1.1:6066          0.0.0.0:*               LISTEN      2864/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -
udp        0      0 192.168.192.22:123      0.0.0.0:*                           -
udp        0      0 127.0.0.1:123           0.0.0.0:*                           -
udp        0      0 0.0.0.0:123             0.0.0.0:*                           -
udp        0      0 0.0.0.0:21415           0.0.0.0:*                           -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     7195     -                   /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     SEQPACKET  LISTENING     405      -                   /run/udev/control

Reason I'm asking this is that I'm unable to connect workers to master node and I think the issue is that the master ip is not discoverable.

Error when try to connect slave to master : 

15/04/02 21:58:18 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@raspberrypi:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: raspberrypi/192.168.192.22:7077
15/04/02 21:58:18 INFO RemoteActorRefProvider$RemoteDeadLetterActorRef: Message [org.apache.spark.deploy.DeployMessages$RegisterWorker] from Actor[akka://sparkWorker/user/Worker#1677101765] to Actor[akka://sparkWorker/deadLetters] was not delivered. [10] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

Upvotes: 2

Views: 5036

Answers (1)

Marius Soutier
Marius Soutier

Reputation: 11274

In spark-env.sh you can set SPARK_MASTER_IP=<ip>.

A hostname would also work fine (via SPARK_STANDALONE_MASTER=<hostname>), just make sure the workers connect to exactly the same hostname as the master binds to (i.e. the spark:// address that is shown in Spark master UI).

Upvotes: 3

Related Questions