CODEWITHSUNDEEP
google-cloud-dataflow
sag
sag

Reputation: 5451

Custom parameters from PipelineOptions

I have a CustomSource and CustomSink. These Source and Sink require specific parameters like from where the data has to be read and where the result has to be written.

I want to pass these parameters as args to PipelineOption. How to get these parameter from PipelineOption?

Is writing my own Option extending PipelineOption the only way?

Thanks, Sam.

Upvotes: 1

Views: 552

Answers (1)

Pieter De Maeyer
Pieter De Maeyer

Reputation: 48

To create your own options you first extend the PipelineOptions interface:

public static interface Options extends PipelineOptions {
    String getInput();
    void setInput(String value);

    String getOutput();
    void setOutput(String value);
}

Then when creating your PipelineOptions you do the following:

public static void main(String[] args) {
     Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);
     ...
}

When you want to retrieve your options:

Options options = pipeline.getOptions().as(Options.class);
options.getInput();

You can also take a look at the word count example here.

Upvotes: 2

Related Questions