CODEWITHSUNDEEP
oozieoozie-workflow
Gopal Kumar
Gopal Kumar

Reputation: 311

File as command line argument to spark action in Oozie workflow

How to pass a file as command line argument to spark job in Oozie workflow? My spark job is expecting a file as command line argument, but when I am pass that file in the workflow as /file/location it is not picking up that file.

Upvotes: 0

Views: 516

Answers (1)

Gopal Kumar
Gopal Kumar

Reputation: 311

I got one workaround, if we put the file in a custom-directory in ozzie shared library with a few additional change in job.properties

  • oozie.use.system.libpath=true
  • oozie.action.sharelib.for.spark=spark,custom-directory
  • oozie.libpath=true

Then we need to update the shared lib using below command:

oozie admin -auth SIMPLE -sharelibupdate

After that we can directly pick up the file by just using the name of the file, which we placed in custom-directory, in the oozie workflow.

Upvotes: 1

Related Questions