CODEWITHSUNDEEP
apache-spark
Jagadeesh
Jagadeesh

Reputation: 31

Spark uses Random Port even after defining executor port

I have a small cluster setup for my development purpose, which contains 3 VMs with spark 2.3 installed on all the VMs. I have started the master in VM1 and slaves with master Ipaddress in other 2 Vms. we have Firewall up in all the 3 Vms and opened the port range from 38001:38113 in the firewall

Before starting the VMs we have the following configurations Done.

In Master, Worker 1 & Worker 2 Nodes

Spark-default.conf file was added with the following properties:

In Worker 1 & Worker 2 Nodes

Spark-env.sh file was added with the following properties:

When we started the Spark-shell and executed a sample csv file read, the executor started on the Worker is starting with a random port for spark driver.

E.g:

Spark Executor Command: "/usr/java/jdk1.8.0_171-amd64/jre/bin/java" "-cp" "/opt/spark/2.3.0/conf/:/opt/spark/2.3.0/jars/*" "-Xmx1024M" "-Dspark.driver.port=34573" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:34573" "--executor-id" "1" "--hostname" "293.72.146.384" "--cores" "4" "--app-id" "app-20180706072052-0000" "--worker-url" "spark://[email protected]:38112"

As you can see in the above command the executor started with Spark.driver.port with 34573. And this is always starting randomly. Because of this my program fails as it is unable to communicate with the driver.

Can anyone help me with this configuration which can be used to execute in network tight environment where All the ports are blocked.

Thanks in advance.

Upvotes: 1

Views: 2786

Answers (1)

DataDataData
DataDataData

Reputation: 19

Start worker:

./start-slave.sh spark://hostname:port -p [Worker Port]

Options: -c CORES, --cores CORES Number of cores to use -m MEM, --memory MEM Amount of memory to use (e.g. 1000M, 2G) -d DIR, --work-dir DIR Directory to run apps in (default: SPARK_HOME/work) -i HOST, --ip IP Hostname to listen on (deprecated, please use --host or -h) -h HOST, --host HOST Hostname to listen on -p PORT, --port PORT Port to listen on (default: random) --webui-port PORT Port for web UI (default: 8081) --properties-file FILE Path to a custom Spark properties file. Default is conf/spark-defaults.conf.

Upvotes: -1

Related Questions