CODEWITHSUNDEEP
google-cloud-dataflowapache-beam
rmahfoud
rmahfoud

Reputation: 41

How to get PipelineOptions in composite PTransform in Beam 2.0?

After upgrading to Beam 2.0 the Pipeline class doesn't have getOptions() class anymore. I have a composite PTransform that relies on getting the options in the its expand method:

public class MyCompositeTransform extends PTransform<PBegin, PDone> {
    @Override
    public PDone expand(PBegin input) {
        Pipeline pipeline = input.getPipeline();
        MyPipelineOptions options = pipeline.getOptions().as(MyPipelineOptions.class);
        ...
    }
}

In Beam 2.0 there doesn't seem to be a way to access the PipelineOptions at all in the expand method.

What's the alternative?

Upvotes: 4

Views: 3934

Answers (1)

Kenn Knowles
Kenn Knowles

Reputation: 6033

Pablo's answer is right on. I want to also clarify that there is a major change in how PipelineOptions are managed.

You can use them to parse and pass around the arguments to your main program (or whatever code builds your pipeline) but these are technically independent from the PipelineOptions that configure how your pipeline is run.

In Beam, the Pipeline is fully constructed, and only afterwards you choose a PipelineRunner and PipelineOptions to control how the pipeline is run. The pipeline itself does not actually have options.

If you do want the behavior of your PTransform (not its expansion) to use some option that is obtained dynamically, you should make your PTransform accept a ValueProvider like this example in WriteFiles and you can define a pipeline option that returns a ValueProvider like here in ValueProviderTest

Upvotes: 3

Related Questions