CODEWITHSUNDEEP
javaapache-spark
FreeMemory
FreeMemory

Reputation: 8634

Spark standalone cluster on custom JVM

I'm trying to deploy a Spark standalone cluster using a custom built JVM. Running the Spark master and Spark worker processes is fine, but once the driver starts and sends work to the master, the worker crashes because it tries to fork an Executor by looking in JAVA_HOME/bin/java. Is there a way to customize how the executor gets started so that it uses a custom JVM?

Upvotes: 1

Views: 153

Answers (1)

FreeMemory
FreeMemory

Reputation: 8634

I figured out the easiest way of doing this was to inherit org.apache.spark.deploy.worker.Worker. I did that in Java, then created a PartialFunction that handled only the LaunchExecutor message; when I received that message I created another custom instance of a class I inherited from ExecutorRunner.

Inside my CustomExecutorRunner I built up my own special Java command and executed that rather than letting ExecutorRunner do it.

It was quite messy and required a fair bit of reflection to access private fields that I needed to get everything running smoothly, but it worked.

Upvotes: 0

Related Questions