CODEWITHSUNDEEP
javaoozie
Davis Broda
Davis Broda

Reputation: 4135

Generating Oozie Workflows using Java Code

Looking through Oozie examples and documentation, it looks like you need a workflow file in order to run an oozie job from Java code. Is ther any way to submit a job directly fro Java code, without needing a workflow file? Is there any pre-existing way to dynamically generate these files through java code? Are there any pre-existing tools that will make generating them easier? Or will I have to write the entirety of the code to generate the file?

Current Situation

    OozieClient wc = new OozieClient("http://bar:8080/oozie");

    Properties conf = wc.createConfiguration();
    conf.setProperty(OozieClient.APP_PATH, "workflow file path");
    // set other properties
    ...

    // submit and start the workflow job
    wc.run(conf);

Ideal situation is something vaguely like this.

OozieAction action = new OozieAction("actionName");
action.setOkDestination("nextAction");
action.setErrorDestination("errorDestination");
//Rest of config for action

OozieWorkflow workflow = new Oozieworkflow();
workfow.setStartAction(action);
workflow.addAction(otherAction);
//rest of conf

OozieClient wc = new OozieClient("http://bar:8080/oozie");
wc.runWorkflow(workflow);

Another situation that would be desirable if the former is impossibble is:

OozieAction action = new OozieAction("actionName");
action.setOkDestination("nextAction");
action.setErrorDestination("errorDestination");

//Rest of config for action

OozieWorkflow workflow = new Oozieworkflow();
workfow.setStartAction(action);
workflow.addAction(otherAction);

//rest of conf

workflow.writeToFile("some localFile")


//load file to HDFS

//This would also work
// workflow.writeToHDFS("someHdfsLocation");

OozieClient wc = new OozieClient("http://bar:8080/oozie");

//run with created workflow

Upvotes: 4

Views: 2676

Answers (4)

asalamon74
asalamon74

Reputation: 6170

Oozie 5.1.0 added support for Fluent Job API which makes it possible to write java code instead of workflow XML files (under the hood, Oozie will generate the XML file for you).

Simple example for the java code which creates a workflow similar to the shell action demo of Oozie:

public class MyFirstWorkflowFactory implements WorkflowFactory {

    @Override
    public Workflow create() {
        final ShellAction shellAction = ShellActionBuilder.create()
                .withName("shell-action")
                .withResourceManager("${resourceManager}")
                .withNameNode("${nameNode}")
                .withConfigProperty("mapred.job.queue.name", "${queueName}")
                .withExecutable("echo")
                .withArgument("my_output=Hello Oozie")
                .withCaptureOutput(true)
                .build();

        final Workflow shellWorkflow = new WorkflowBuilder()
                .withName("shell-workflow")
                .withDagContainingNode(shellAction).build();

        return shellWorkflow;
    }
}

More detailed documentation can be found here: https://oozie.apache.org/docs/5.1.0/DG_FluentJobAPI.html

Upvotes: 1

Prabakaran
Prabakaran

Reputation: 1

have a static oozie workflow in your HDFS which just takes 2 parameters and writes the content of parameter1(say content which user enters) to parameter2(say writing to HDFS). Now call the oozie CLI and specify app.path as the location created by workflow1

Upvotes: 0

Ahmed Moawad
Ahmed Moawad

Reputation: 381

There's a graphical tool to generate Oozi workflows via an eclipse plugin. Find it here Eclipse marketplace: https://marketplace.eclipse.org/content/oozie-eclipse-plugin

It looks like this: enter image description here

enter image description here

Upvotes: 1

foo_bar
foo_bar

Reputation: 41

I have been in a similar situation.

What I would suggest is to use the oozie schema definition (xsd) and generating the java equivalent objects through xjc. Given these objects you can probably create the workflow (not trivial though)

There are scala based DSL's you can use https://github.com/klout/scoozie does something similar with Scala->oozie generation

Upvotes: 3

Related Questions