CODEWITHSUNDEEP
apache-sparkapache-spark-standalone
user4473195
user4473195

Reputation:

In spark standalone mode is the master and executors located in a single machine?

Does spark standalone mode means that the executors and master are run on a single machine?If yes how can it attend parallelism. Is the value passed to set local function of spark conf set as one in standalone mode to indicate that spark application is running on single machine?

Upvotes: 2

Views: 868

Answers (1)

Jason Heo
Jason Heo

Reputation: 10236

"Does spark standalone mode means that the executors and master are run on a single machine?" => No.

"standalone" in "spark standalone mode" does not mean a single machine. In Spark Standalone mode, typically there are multiple slave nodes. Spark Standalone mode is another cluster mode like Mesos and Yarn. Please refer to https://spark.apache.org/docs/latest/spark-standalone.html in detail.

I guess you are asking "Local Mode". Spark runs in Local Mode if you execute like spark-shell --master=local[4]. In this case, spark driver and executors run on a single JVM and has multiple threads.

You can find many answers about "Spark Local Mode" if you google it.

Upvotes: 2

Related Questions