CODEWITHSUNDEEP
google-cloud-dataflowapache-beam
Norio Akagi
Norio Akagi

Reputation: 725

Google Cloud Dataflow (Apache Beam) - Can I use SideInputs with TextIO.write?

It looks so far we can use sideInputs only for DoFn to use side input when processing each input line.

Is there any way to utilize the input with TextIO.Write ? In my case,

    outputTuple.get(deadletterTag)
        .apply("Write to deadletter files",
            TextIO.write().to(String.format("gs://dataflow-xxx/deadletter/%s/", uniqueID))
                .withSuffix(".csv")
                .withHeader(csvHeader)
        ).withSideInputs(csvHeaderView);

I have a csv header info as a PCollectionView<String> (obtained in a prior process by using Combine.globally and AsSingletonView) and want to somehow extract the header string and assign it into withHeader, but not sure if it's possible because there is no ProcessContext appears.

Thanks.

Upvotes: 0

Views: 295

Answers (1)

Lukasz Cwik
Lukasz Cwik

Reputation: 1731

Unfortunately there is no easy way to do this as of Apache Beam 2.2.0 and earlier.

I have created BEAM-2771 which tracks a general feature which could enable this.

Upvotes: 1

Related Questions