CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowapache-beam
Scott Wegner
Scott Wegner

Reputation: 7493

Dataflow Display Data missing from composite transform

I'm trying the new display data functionality in Dataflow to make additional details show up in the Google Cloud Dataflow UI. However, the display data for custom PTransform's doesn't show up. In my Dataflow pipeline, I have a transform like:

Pipeline p = // ..
p.apply(new PTransform<PCollection<Integer>, PCollection<Integer>>() {
  @Override
  public PCollection<Integer> apply(PCollection<Integer> input) {
    return input
      .apply(/* .. */)
      .apply(/* .. */)
      .apply(/* .. */);
  }

  @Override
  public void populateDisplayData(DisplayData.Builder builder) {
    builder.add(DisplayData.item("foo", "bar"));
  }
});

When I run the Dataflow job, the UI doesn't seem to show the foo=bar display data.

Upvotes: 1

Views: 382

Answers (1)

Scott Wegner
Scott Wegner

Reputation: 7493

Display data is not supported on composite transforms. Instead, you can attach display data to any user-defined function which your transform executes. For example, if one of the inner transforms in the above composite uses a ParDo, you could re-write the display data methods as:

Pipeline p = // ..
p.apply(new PTransform<PCollection<Integer>, PCollection<Integer>>() {
  @Override
  public PCollection<Integer> apply(PCollection<Integer> input) {
    return input
      .apply(/* .. */)
      .apply(/* .. */)
      .apply(ParDo.of(new DoFn<Integer, Integer>() {
        @Override
        public void processElement(ProcessContext c) { /* .. */ } 

        @Override
        public void populateDisplayData(DisplayData.Builder builder) {
          builder.add(DisplayData.item("foo", "bar"));
        }            
      }));
  }
});

There is a JIRA issue to extend display data functionality to composite transforms in Beam and Dataflow SDKs.

Upvotes: 2

Related Questions