CODEWITHSUNDEEP
amazon-s3oozie
cloud
cloud

Reputation: 41

How to let oozie access S3 to get the workflow.xml and ***.jar

I want to trigger a oozie job by：

oozie job -oozie http://***.***.***.***:11000/oozie -config /tmp/test1/job.properties -run

My job.properties:

oozie.wf.application.path=s3a://my-bucket/job1
user.name=hdfs
jobTracker=cm001:8032
nameNode=hdfs://cm001:8020
mainClass=****

My ***.jar and workflow.xml is all located in s3a://my-bucket/job1, I know i have to config the S3 endpoint and the AK/SK to let oozie access S3, but where can i config these configuration? How to config this configuration? I didn't find any document in oozie official document.

Upvotes: 0

Views: 559

Answers (1)

cloud
cloud

Reputation: 41

My scenario is use Cloudera to manage Oozie and HDFS. And i want let Oozie to use Ceph as the filesystem. My configuration is:

  1. Add S3 related configuration to oozie-site.xml in oozie server and core-site.xml in HDFS.

Add following configuration:

    fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
    fs.s3a.path.style.access=true
    fs.s3a.endpoint=***
    fs.s3a.access.key=***
    fs.s3a.secret.key=***

  1. The path of workflow.xml shouldn't located in root folder.

    Good example: 's3a://bucket/test/' Bad example: 'sa3://bucket'

Upvotes: 1

Related Questions