CODEWITHSUNDEEP
google-cloud-dataflow
Graham Polley
Graham Polley

Reputation: 14791

Latest SDK version 0.4.150414 giving never seen before warnings

What do these warnings mean when running our pipeline?

1497 [main] WARN com.google.cloud.dataflow.sdk.Pipeline - Transform AsIterable2 does not have a stable unique name. In the future, this
will prevent reloading streaming pipelines

Upvotes: 0

Views: 77

Answers (1)

Ben Chambers
Ben Chambers

Reputation: 6130

The warning in question indicates that the specified transform -- AsIterable2 -- isn't uniquely named. A likely cause of this is that there are two applications of an AsIterable transform at the top-level.

You can get rid of the warning by using the PTransform#setName method on the transform in question.

We attempt to infer names from the class names being applied. The only times it should be necessary to set an explicit name are:

  1. When using an anonymous PTransform or DoFn.

  2. When the same named transform is used multiple times within the same PTransform.

Specifically, requirement #2 means that if you use a named PTransform multiple times within an outer PTransform you need to make sure that each application has a different name. For instance:

input.apply(someTransform)
    .apply(View.<TableRow>asIterable().withName("iterable1"));
input.apply(someOtherTransform)
    .apply(View.<TableRow>asIterable().withName("iterable2"));

instead of:

View.AsIterable<TableRow> iterable = View.TableRow>asIterable().setName("aName");
input.apply(someTransform).apply(iterable);
input.apply(someOtherTransform).apply(iterable);

Upvotes: 1

Related Questions