CODEWITHSUNDEEP
scalaapache-sparkintellij-ideavectorapache-spark-ml
Magg_rs
Magg_rs

Reputation: 321

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/ml/feature/VectorAssembler in IntelliJ

I am trying to build a model in Spark ML using Linear Regression in IntellijIDEA.

Before fitting the model, i am supposed to create a VectorAssembler having feature column. 

import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.linalg.Vectors

//creating features column
val assembler = new VectorAssembler()
  .setInputCols(Array("col4","col5","col6","col7"))
  .setOutputCol("features")

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/ml/feature/VectorAssembler at energydata$.main(energydata.scala:35) at energydata.main(energydata.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.ml.feature.VectorAssembler at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 2 more

But this gives error in Intellij. When i tried the same in spark-shell, it works.

can anybody suggest where I might be going wrong here?? 

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.6"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.1.0" % "provided"

Upvotes: 0

Views: 2402

Answers (2)

Vishal rana
Vishal rana

Reputation: 11

I have also face same error while running spark application. i have just remove (%%"provide") scope of dependency spark sql

Upvotes: 0

Mith Coriel
Mith Coriel

Reputation: 29

I had a similar problem with other classes that it couldn't find. And it was because, in the maven dependency, it said "<scope>provided</scope>". I removed that, pressed reload, and now it works.

Upvotes: 2

Related Questions