CODEWITHSUNDEEP
javascalaapache-spark
Pinch
Pinch

Reputation: 2888

Error calling updateStateByKey in Spark Streaming

I have this generic method in Scala

def updateStateByKey[S](updateFunc: JFunction2[JList[V], Optional[S],
Optional[S]])   : JavaPairDStream[K, S] = { ... }

When I call it in Java, both of these does not compile:

1

JavaPairDStream<String, Integer> stateDstream =
pairs.<Integer>updateStateByKey(...);

2

JavaPairDStream<String, Integer> stateDstream =
pairs.updateStateByKey(...);

How do I invoke the method correctly?

Error messages:

The method updateStateByKey(Function2<List<Integer>,Optional<S>,Optional<S>>,
int) in the type JavaPairDStream<String,Integer> is not applicable for
the arguments
(Function2<List<Integer>,Optional<Integer>,Optional<Integer>>,
HashPartitioner, JavaPairRDD<String,Integer>)

Edited: The whole function call (Java 8):

final Function2<List<Integer>, Optional<Integer>, Optional<Integer>> updateFunction =
    (values, state) -> {
      Integer newSum = state.or(0);
      for (Integer value : values) {
        newSum += value;
      }
      return Optional.of(newSum);
    };



JavaPairDStream<String, Integer> stateDstream = pairs.updateStateByKey(
    updateFunction
    ,
    new HashPartitioner(context.defaultParallelism()), initialRDD);

Edited: It turned out that generics is not the issue, but the parameters do not match the method signature.

Upvotes: 1

Views: 550

Answers (1)

Justin Pihony
Justin Pihony

Reputation: 67135

The problem is that you are passing in an initialRDD, while the method updateStateByKey does not have that as a parameter.

The closest signature is:

updateStateByKey[S](updateFunc: Function2[List[V], Optional[S], Optional[S]], 
  partitioner: Partitioner): JavaPairDStream[K, S]

Upvotes: 1

Related Questions