CODEWITHSUNDEEP
apache-sparkhere-olp
InherentlyNuts
InherentlyNuts

Reputation: 199

How to override spark conf in OLP

How can I use the following flags to run a spark job in OLP?

--conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true

Upvotes: 1

Views: 164

Answers (2)

InherentlyNuts
InherentlyNuts

Reputation: 199

Using a shaded jar did help, and yes it was a dependency issue, since I was using a higher version of the commons lib.

I had to put my commons lib dependency before any other conflicting dependency would be included, incase anyone else faces this issue, they could try that to resolve conflicts.

Upvotes: 0

ichow
ichow

Reputation: 141

OLP currently do not support setting custom Spark properties. If you are running into dependency conflicts (often what triggers this question), you will need to find another way to resolve them such as shading (when building the deployed fat jar).

Upvotes: 0

Related Questions