CODEWITHSUNDEEP
javaapache-spark
TheCyberliem
TheCyberliem

Reputation: 1368

Java FlatMapFunction in Spark: error: is not abstract and does not override abstract method call(String) in FlatMapFunction

I was trying to run the first example.

The source code is:

/*keyWordCount.java */
            import org.apache.spark.*;
            import org.apache.spark.api.java.*;
            import org.apache.spark.api.java.function.*;
            import org.apache.spark.rdd.*;
            import org.apache.spark.api.java.JavaRDD;
            import java.util.*;

            public class keyWordCount {

                public static void main(String[] args) {
                    SparkConf conf = new SparkConf().setAppName("keyWordCount");
                    JavaSparkContext sc = new JavaSparkContext(conf);
                    JavaRDD<String> textFile = sc.textFile("output");
                    JavaRDD<String> dictFile = sc.textFile("keyword");
                    JavaRDD<String> words = textFile.flatMap(new FlatMapFunction<String, String>() {
                        @Override public Iterable<String> call(String s) { return Arrays.asList(s.split(" ")); }
                    });
                }
            }

When I compile using mvn compile package, the following error keep showing up:

[ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/cyberliem/TestSpark/src/main/java/keyWordCount.java:[16,81] error: is not abstract and does not override abstract method call(String) in FlatMapFunction

[ERROR] /home/cyberliem/TestSpark/src/main/java/keyWordCount.java:[17,39] error: call(String) in cannot implement call(T) in FlatMapFunction

[ERROR] T extends Object declared in interface FlatMapFunction R extends Object declared in interface FlatMapFunction /home/cyberliem/TestSpark/src/main/java/keyWordCount.java:[17,5] error: method does not override or implement a method from a supertype

I'm not sure how to fix this, can anyone give me an ideal why it goes wrong?

Upvotes: 7

Views: 4427

Answers (3)

bigbro
bigbro

Reputation: 41

I guess that the version of spark is a newer one, we can find that the function call(T t) we should override returns Iterator instead of Iterable,see http://spark.apache.org/docs/latest/api/java/index.html.

As the answer above,what we should do is :

  1. import java.util.Iterator;

  2. change your code to :

JavaRDD words = textFile.flatMap(new FlatMapFunction() { @Override public Iterator call(String s) { return Arrays.asList(s.split(" ")).iterator(); }

Upvotes: 1

Somum
Somum

Reputation: 2422

Just for clarification there looks like a version mismatch when using FlatMapFunction on my ellipse and compiling via maven

On ellipse, I had this

public Iterable<String> call(String s)

On the other hand, when compiling using maven I had to use

public Iterator<Rating> call(String s)

This caused agony but now it has resolved using latter.

Upvotes: 1

Ivan Shulak
Ivan Shulak

Reputation: 100

Try this one:

JavaRDD<String> words = textFile.flatMap(new FlatMapFunction<String, String>() {
    @Override public Iterator<String> call(String s) { return Arrays.asList(s.split(" ")).iterator(); }
});

or even simpler using lambdas:

JavaRDD<String> words = textFile.flatMap(l -> Arrays.asList(l.split(" ")).iterator());

Upvotes: 6

Related Questions