CODEWITHSUNDEEP
apache-sparkooziespark-submitoozie-coordinatoroozie-workflow
Zaid Khan
Zaid Khan

Reputation: 826

Oozie workflow arguments interprets double quotes weirdly for spark-submit command parameters

I have a Spark job which takes a bunch of configurable parameters. I am facing an issue specifically in this portion:

--conf spark.executor.extraJavaOptions="-Duser.timezone=PST -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35"

For some reason, Oozie interprets this as:

--conf 'spark.executor.extraJavaOptions="-Duser.timezone=PST' -XX:+UseG1GC '-XX:InitiatingHeapOccupancyPercent=35"'

instead of:

--conf spark.executor.extraJavaOptions="-Duser.timezone=PST -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35"

I have tried various combinations, but most of them give wrong results. Used <![CDATA[--conf ...]]> as well but to no avail. Any help appreciated

Upvotes: 0

Views: 17

Answers (0)

Related Questions