CODEWITHSUNDEEP
javascalaapache-sparkpmml
Kunal
Kunal

Reputation: 55

Spark- Exception in thread java.lang.NoSuchMethodError

I am trying to score a model from pmml file using pmml4s library. Every time I submit the job in Spark I get the following error:

20/05/13 23:30:10 ERROR SparkSubmit: org.apache.spark.sql.types.StructType.names(). 
[Ljava/lang/String;
   java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.names(). 
    [Ljava/lang/String;
    at org.pmml4s.spark.ScoreModel.transform(ScoreModel.scala:56)
    at com.aexp.JavaPMML.main(JavaPMML.java:24)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

Following is my code sample:

ScoreModel model = ScoreModel.fromFile(args[0]);
SparkConf conf = new SparkConf();
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();
Dataset<?> df = spark.read().format("csv")
                     .option("header", "true")
                     .option("inferSchema", "true")
                     .load(args[1]);

Dataset<?> scoreDf = model.transform(df);

Following is the pom file that I am using:

<dependencies>
    <dependency>
        <groupId>org.pmml4s</groupId>
        <artifactId>pmml4s-spark_2.11</artifactId>
        <version>0.9.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
          <artifactId>spark-core_2.11</artifactId>
          <version>2.3.2</version>
        </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.11</artifactId>
        <version>2.3.2</version>
    </dependency>
</dependencies>

I have edited my pom file and made the spark version similar still I face the same issue. When I am using Scala, I am facing the same problem. Is there any dependency that I am missing?

Upvotes: 2

Views: 1892

Answers (2)

PredictFuture
PredictFuture

Reputation: 226

The error is caused by the PMML4S-Spark used the method names of StructType, which is introduced since Spark 2.4. Now it has been fixed in the latest PMML4S-Spark 0.9.5. Please, update your pom file to use the new version:

    <dependency>
        <groupId>org.pmml4s</groupId>
        <artifactId>pmml4s-spark_2.11</artifactId>
        <version>0.9.5</version>
    </dependency>

Upvotes: 1

s.polam
s.polam

Reputation: 10382

Try to use same version of spark libraries. If spark versions are not matching we will be getting NoSuchMethodError issue in many places as those methods might have modified or removed in latest versions.

Upvotes: 3

Related Questions