CODEWITHSUNDEEP
google-cloud-dataflowapache-beam
supahot
supahot

Reputation: 31

cannot transform PCollection containing type Map - "wrong number of arguments"

I'm trying to apply a transform to a PCollection of Map objects in Apache beam with ParDo. I specified Map as the output type of the first DoFn, and specified Map as the input type of the next DoFn. However, I'm getting the following error:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam: An exception occured while executing the Java class. error when invoking Coder factory method public static org.apache.beam.sdk.coders.MapCoder org.apache.beam.sdk.coders.MapCoder.of(org.apache.beam.sdk.coders.Coder,org.apache.beam.sdk.coders.Coder): wrong number of arguments -> [Help 1]

Here's my code:

  static class GetProductName extends DoFn<String, Map> {
    @ProcessElement
    public void processElement(ProcessContext c) {
      {...}
      Map<String, String> properties = new HashMap<>();
      properties.put("name", "test_name");
      properties.put("sku", "test_sku");

      // Use ProcessContext.output to emit the output element.
      c.output(properties);
    }
  }

  static class FormatForDatastore extends DoFn<Map, Entity> {
    @ProcessElement
    public void processElement(ProcessContext c) {
      Map<String, String> product = new HashMap<>();
      product = c.element();
    {...}
    }
  }

  public static void main(String[] args) {
    WordCountOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(WordCountOptions.class);
    Pipeline p = Pipeline.create(options);

    // apply transforms
    p.apply("ReadJSONLines", TextIO.read().from(options.getInputFile()))
     .apply(ParDo.of(new GetProductName()))
     .apply(ParDo.of(new FormatForDatastore()))
     {...}
   }

Any ideas what the issue is?

I'm not sure how to interpret the error. I did some testing by editing the code to transform PCollection of type String instead of type Map, and it works without issues. However, I would like a way to pass multiple key value pairs between these transforms. Suggestions for alternatives are welcome.

Upvotes: 1

Views: 1353

Answers (1)

jkff
jkff

Reputation: 17913

You're using the raw type Map. Try using the proper generic type Map<String,String>.

Upvotes: 1

Related Questions