CODEWITHSUNDEEP
hadoopworkflowapache-pigoozie
Angelik
Angelik

Reputation: 99

How to launch N times a script Pig using oozie?

I need to process a lot of files with some specifically date. I find only one solution, which is to launch a job N times with each time a different dataset. The partitions used is based on yyyy, mm, dd. I have a java action which generate the good partition to use for each data.

My question is, how can I create a loop to launch my script N times ? I work today with oozie workflow.

Thanks

Upvotes: 0

Views: 76

Answers (1)

Frederic
Frederic

Reputation: 3284

This sounds like a use case for coordinators.

You can declare Datasets and let oozie automatically start a workflow when a specfic dataset instance is available.

Upvotes: 1

Related Questions