CODEWITHSUNDEEP
apache-sparkclasspathapache-spark-standalone
LiranBo
LiranBo

Reputation: 2126

Prepend spark.jars to workers classpath

My use case is pretty simple, I want to override a few classes that are part of the Hadoop distribution, to do so I created a new jar that I serialize from the driver to the worker nodes using spark.jars properties.

To make sure my new jar takes precedence in the workers classpath, I want to add them to spark.executor.extraClassPath property.

However, since I'm serializing these jars with spark.jars, their path in the workers is dynamic and includes the app-id & executor-id - <some-work-dir>/<app-id>/<executor-id>.

Is there a way around it? is it possible to add a dir inside the app dir to be first in classpath?

Working with Spark 2.4.5 Standalone client mode - Docker.

p.s I'm aware of the option to add the jar to the workers image, and then add it to the classpath, but then I'll have to keep updating the image with every code change.

Upvotes: 0

Views: 762

Answers (1)

diogoramos
diogoramos

Reputation: 86

You can enable this option on spark submit:

spark.driver.userClassPathFirst=True

Check here the spark-submit options documentation

Upvotes: 1

Related Questions