CODEWITHSUNDEEP
apache-sparkcluster-computing
Tiffany
Tiffany

Reputation: 273

Difference between YARN deployment modes - CLUSTER and CLIENT modes on the machine with a node manager

For me the only difference would be that in the first case the Driver would be inside the App Master JVM whereas it would be next to it the above client mode configuration. What would be different in those two cases ?

Upvotes: 1

Views: 607

Answers (1)

Raktotpal Bordoloi
Raktotpal Bordoloi

Reputation: 1057

Spark deployment mode - client vs cluster

Client mode

  • It's interactive. If you want to get a job result (dynamic analysis) at your machine (client - the Driver program), client deployment mode is quite useful.
  • Easier for developing/debugging.
  • End user have the control where the Driver Program is running
  • Always up application: expose your Spark job launcher as REST service or a Web UI.

Cluster mode

  • Easier for resource allocation (let the master decide): Fire and forget
  • Due to this Fire-N-Forget behavior, It's good to submit long, resource hungry job in cluster mode.
  • Monitoring Driver Program is done from Master Web UI like other workers. (not from client machine)
  • Stop at the end: once job is finished, allocated resources are freed automatically.
  • In cluster mode, if driver program has some issue, YARN restarts the driver without killing the executors.
  • In client mode, YARN automatically kills all executors if your driver is killed.

Upvotes: 3

Related Questions