Why shouldn't local mode in Spark be used for production?

Question

The official documentation and all sorts of books and articles repeat the recommendation that Spark in local mode should not be used for production purposes. Why not? Why is it a bad idea to run a Spark application on one machine for production purposes? Is it simply because Spark is designed for distributed computing and if you only have one machine there are much easier ways to proceed?

Answer 1

I agree that this is largely ignored in official documentation but there are actually some benefits of running Spark even in local mode (e.g. instead of pure python, scala etc). There's a great resource and benchmark with more details here.

In summary the main advantages:

• A single, unified API that scales from “small data” on a laptop to “‘big data” on a cluster.
• Spark can often be faster, due to parallelism, than single-node PyData tools.
• Spark can have lower memory consumption and can process more data than laptop ’s memory size, as it does not require loading the entire data set into memory before processing.
• Offers a number of abstractions as well as fault tolerance for parallel computing.
• Offers a number of algorithms and functions that are optimized for parallel computing.

Answer 2

Local mode in Apache Spark is intended for development and testing purposes, and should not be used in production because:

• Scalability: Local mode only uses a single machine, so it cannot handle large data sets or handle the processing needs of a production environment.
• Resource Management: Spark’s standalone cluster manager or a cluster manager like YARN, Mesos, or Kubernetes provides more advanced resource management capabilities for production environments compared to local mode.
• Fault Tolerance: Local mode does not have the ability to recover from failures, while a cluster manager can provide fault tolerance by automatically restarting failed tasks on other nodes.
• Security: Spark’s cluster manager provides built-in security features such as authentication and authorization, which are not present in local mode.

Therefore, it is recommended to use a cluster manager for production environments to ensure scalability, resource management, fault tolerance, and security.

Answer 3

I have the same question. I am certainly not an authority on the subject, but because no-one has answered this question, I'll try to list the reasons I've encountered while using Spark local mode in Java. So far:

• Spark uses System.exit() calls in certain occassions, such as an out of memory error or when the local dir does not have write permissions, so if such a call is triggered, the entire JVM shuts down (including your own application from within which Spark runs, see e.g., https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala#L45, https://github.com/apache/spark/blob/b22946ed8b5f41648a32a4e0c4c40226141a06a0/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala#L63). Moreover, it circumvents your own shutdownHook, so there is no way to gracefully handle such system exits in your own application. On a cluster, it is usually fine if a certain machine restarts, but, if all Spark components are contained in a single JVM, the assumption that we can shut down the entire JVM upon a Spark failure does not (always) hold.