CODEWITHSUNDEEP
apache-spark
Finkelson
Finkelson

Reputation: 3013

Spark cluster with two master nodes. Is it possible?

Suppose we have a Spark cluster of 22 nodes. Is it possible to have in the cluster 2 master nodes which would be coordinated for example by Zookeeper?

One master node is a standby node.

Upvotes: 3

Views: 1586

Answers (1)

user6022341
user6022341

Reputation:

It is possible. Use following conf options:

Configuration

In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spark-env using this configuration:

System property Meaning

spark.deploy.recoveryMode Set to ZOOKEEPER to enable standby Master recovery mode (default: NONE).

spark.deploy.zookeeper.url The ZooKeeper cluster url (e.g., 192.168.1.100:2181,192.168.1.101:2181).

spark.deploy.zookeeper.dir The directory in ZooKeeper to store recovery state (default: /spark).

Standby Masters with ZooKeeper

Upvotes: 3

Related Questions