CODEWITHSUNDEEP
apache-spark
DevEx
DevEx

Reputation: 4571

Setting up spark cluster on Virtual Machines

I installed 2 vm machines and also enabled port forwarding so I can ssh from host to guests machines. I started spark standalone Master on host machine, my question is how to set up the 2 vm machines so I can use them as standalone Workers? (FYI: VM network adapter is attached to NAT.)

Upvotes: 1

Views: 2146

Answers (1)

DevEx
DevEx

Reputation: 4571

My solution:

On Worker nodes(vm machines)

  • Use two network adapters: NAT(for internet), Host-Only(to access resources). See how to setup host-only on vm

  • Setup vm machines with same user as host machine.

On Master node(Host)

  • Add this entry, SPARK_MASTER_HOST={IP ADDRESS} in the conf/spark-env.sh to bind spark master to host ip address.
  • Add ip addresses of worker nodes(vm machines) to conf/slaves

  • Configure Master to passwordlessly ssh into Workers:

    ssh-keygen
.ssh/ssh-copy-id -i ./id_rsa.pub {Worker IP Address}

  • Start spark cluster: ./sbin/start-all.sh

Upvotes: 2

Related Questions