Spark - How to run a standalone cluster locally

Question

Is there the possibility to run the Spark standalone cluster locally on just one machine (which is basically different from just developing jobs locally (i.e., local[*]))?.

So far I am running 2 different VMs to build a cluster, what if I could run a standalone cluster on the very same machine, having for instance three different JVMs running?

Could something like having multiple loopback addresses do the trick?

Answer 1

More simply,

./sbin/start-all.sh

On your local machine there will be master and one worker launched.

./bin/spark-submit  \
--class org.apache.spark.examples.SparkPi   \
--master spark://localhost:7077  \
examples/jars/spark-examples_2.12-3.0.1.jar 10000

A sample application is submitted. For monitoring via Web UI:

Master UI:      http://localhost:8080
Worker UI:      http://localhost:8081
Application UI: http://localhost:4040  

Answer 2

If you can't find the ./sbin/start-master.sh file on your machine, you can start the master also with

./bin/spark-class org.apache.spark.deploy.master.Master

Answer 3

A small update as for the latest version (the 2.1.0), the default is to bind the master to the hostname, so when starting a worker locally use the output of hostname:

./bin/spark-class org.apache.spark.deploy.worker.Worker  spark://`hostname`:7077 -c 1 -m 512M

And to run an example, simply run the following command:

bin/run-example SparkPi

Answer 4

yes you can do it, launch one master and one worker node and you are good to go

launch master

./sbin/start-master.sh

launch worker

./bin/spark-class org.apache.spark.deploy.worker.Worker  spark://localhost:7077 -c 1 -m 512M

run SparkPi example

./bin/spark-submit  --class org.apache.spark.examples.SparkPi   --master spark://localhost:7077  lib/spark-examples-1.2.1-hadoop2.4.0.jar 

Apache Spark Standalone Mode Documentation