CODEWITHSUNDEEP
amazon-ec2apache-spark
Wayne Song
Wayne Song

Reputation: 151

Getting java.net.BindException when attempting to start Spark master on EC2 node with public IP

I am trying to start a Spark master for a standalone cluster on an EC2 node. The CLI command I'm using looks like this:

JAVA_HOME=<location of my JDK install> \ java -cp <spark install dir>/sbin/../conf/:<spark install dir>/lib/spark-assembly-1.4.0-hadoop2.6.0.jar:<spark install dir>/lib/datanucleus-core-3.2.10.jar:<spark install dir>/lib/datanucleus-api-jdo-3.2.6.jar:<spark install dir>/lib/datanucleus-rdbms-3.2.9.jar \ -Xms512m -Xmx512m -XX:MaxPermSize=128m \ org.apache.spark.deploy.master.Master --port 7077 --webui-port 8080 --host 54.xx.xx.xx

Note that I'm specifying the --host argument; I want my Spark master to be listening on a specific IP address. The host that I'm specifying (i.e. 54.xx.xx.xx) is the public IP for my EC2 node; I've confirmed that nothing else is listening on port 7077 and that my EC2 security group has all ports open. I've also double-checked that the public IP is correct.

When I use --host 54.xx.xx.xx, I get the following error message:

15/07/27 17:04:09 ERROR NettyTransport: failed to bind to /54.xx.xx.xx:7093, shutting down Netty transport Exception in thread "main" java.net.BindException: Failed to bind to: /54.xx.xx.xx:7093: Service 'sparkMaster' failed after 16 retries! at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389) at scala.util.Success$$anonfun$map$1.apply(Try.scala:206) at scala.util.Try$.apply(Try.scala:161) at scala.util.Success.map(Try.scala:206) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

This does not occur if I leave out the --host argument and it doesn't occur if I use --host 10.0.xx.xx, where 10.0.xx.xx is my private EC2 IP address.

Why would Spark fail to bind to a public EC2 address?

Upvotes: 3

Views: 435

Answers (2)

Evan Choo
Evan Choo

Reputation: 11

I had the same problem using Oracle Cloud Instance. My private IP is like 10.x.x.2 and my public IP is like 140.x.x.238.

Here are some steps you can follow:

  1. Check your private IP address

    Using command ifconfig to find out the address of your network card

ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 10.x.x.2  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::17ff:fe00:7cf9  prefixlen 64  scopeid 0x20<link>
        ether 02:00:17:00:7c:f9  txqueuelen 1000  (Ethernet)
        RX packets 146457  bytes 61901565 (61.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 142865  bytes 103614447 (103.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
  1. Set conf/spark-env.sh
SPARK_LOCAL_IP=127.0.0.1
SPARK_MASTER_IP=YOUR_HOST_NAME

  1. Change hosts file

    In Ubuntu 18.04, modify /etc/hosts

    Remove something like127.0.1.1 YOUR_HOST_NAME

    Change 140.x.x.238 YOUR_HOST_NAME to 10.x.x.2 YOUR_HOST_NAME in my case

Upvotes: 1

Taterhead
Taterhead

Reputation: 5951

Try setting the environment variable SPARK_LOCAL_IP=54.xx.xx.xx

reference the first SO answer to a similar problem here.

Upvotes: 0

Related Questions