CODEWITHSUNDEEP
spark-streamingapache-spark-1.4
dexter
dexter

Reputation: 462

Compile error while calling updateStateByKey

Compile Error :

The method updateStateByKey(Function2<List<Integer>,Optional<S>,Optional<S>>) in the type JavaPairDStream<String,Integer> is not applicable for the arguments (Function2<List<Integer>,Optional<Integer>,Optional<Integer>>)

In a simple word count example , mapping the words with 1 

JavaPairDStream<String, Integer> wordCounts = words.mapToPair(s -> new Tuple2<>(s,1));

And then applying updateStateByKey on wordCounts 

 JavaPairDStream<String, Integer> finalcount =  wordCounts.updateStateByKey(updateFunction);

The updateFunction is defined as follows:

 final Function2<List<Integer>, Optional<Integer>, Optional<Integer>> updateFunction =
                    new Function2<List<Integer>, Optional<Integer>, Optional<Integer>>() {
                      @Override
                      public Optional<Integer> call(List<Integer> values, Optional<Integer> state) {
                        Integer newSum = state.orElse(0);
                        for (Integer value : values) {
                          newSum += value;
                        }
                        return Optional.of(newSum);
                      }
                    };

The updateStateByKey has following recommended signatures available:

enter image description here

Upvotes: 1

Views: 167

Answers (1)

user5276310
user5276310

Reputation: 21

Please check which package you import for using Optional. Spark use com.google.common.base.Optional not jdk default package java.util.Optional.

Upvotes: 2

Related Questions