CODEWITHSUNDEEP
amazon-web-servicesapache-sparkamazon-ec2
user3086871
user3086871

Reputation: 721

How to bind Public IP to spark nodes in Amazon EC2?

I am trying to create a Spark cluster between two instances in two different regions. As they are not in same VPC/security group, I am having trouble to connect Master from one region to Slave from another region (and vice versa). So far I have done the following:

  1. Edited /etc/hosts file to add public IP of both Master and Slaves

    54.208.204.190 master 13.113.105.113 slave01

  2. Added slave01 to $SPARK_HOME/conf/slaves file

  3. In $SPARK_HOME/conf/spark-env.sh added the following:

    export JAVA_HOME=/home/ubuntu/jdk1.8.0_151 export SPARK_WORKER_CORES=8 export SPARK_MASTER_HOST=ec2-54-208-204-190.compute-1.amazonaws.com

    I have assigned Public DNS of master in SPARK_MASTER_HOST because assigning public IP of master was not working. It was showing me the following error:

    MasterUI' could not bind on port 8080.

So, the above configuration worked for me and I can see slave01 successfully registered with master, and in Spark WebUI one worker was showing as intended. But when I tried to run SparkPi example, it could not add an executor. In logs from slave01 I have found the following:

`Caused by: java.io.IOException: Failed to connect to /172-31-23-69:48441`

172-31-23-69 is the private IP of the master. In my understanding, the slave01 wanted to connect to master by this private IP of master, but as they are not in the same vpc slave01 is failing to connect to master. I am not sure why slave01 will want to use private IP of master in the first place because I have given both Public DNS and IP of the master in spark-env.sh and hosts file. Also, how slave01 came to know the private IP of master is another interesting question.

I have tried to set SPARK_LOCAL_IP variable to public IP in both instances respectively, but that does not work either. So if anyone can show me any kind of direction here I will be very grateful. Thanks in advance.

Upvotes: 0

Views: 2353

Answers (1)

Michael - sqlbot
Michael - sqlbot

Reputation: 178966

When an EC2 instance has a public IPv4 address associated with it, you can't bind a socket to the public IP address, because of the way public IP addresses are handled in EC2.

The public IP is statically NAT-ed to the private IP by the Internet Gateway -- the instance itself is not aware of the public IP address.

(See the output from ifconfig -- the public IP is not there, and isn't supposed to be there -- only the private IP).

VPC peering allows you to interconnect the networks of multiple VPCs together, giving instances access to each other across account boundaries and even AWS region boundaries.

There may be an alternate solution specific to what you're doing, but keeping the traffic all within the bounds of private IP space seems like a good workaround and best practice.

Note that interconnected VPCs must have unique, non-overlapping CIDR blocks. Peering Isn't transitive, so peering VPC A to B and then peering VPC B to C does not allow VPCs A and C to communicate. Any two VPCs that have instances needing to communicate must be directly peered.

https://docs.aws.amazon.com/AmazonVPC/latest/PeeringGuide/Welcome.html

Upvotes: 2

Related Questions