CODEWITHSUNDEEP
scalaapache-spark
Joji Ddan
Joji Ddan

Reputation: 61

run-main-0) scala.ScalaReflectionException: class java.sql.Date in JavaMirror with ClasspathFilter(

Hi I have a file given to by my teacher. It is about Scala and Spark. When I run the code it gives me this exception:

  (run-main-0) scala.ScalaReflectionException: class java.sql.Date in 
  JavaMirror with ClasspathFilter

The file itself looks like this:

import org.apache.spark.ml.feature.Tokenizer
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._
object Main {
   type Embedding       = (String, List[Double])
   type ParsedReview    = (Integer, String, Double)
   org.apache.log4j.Logger getLogger "org"  setLevel 
   (org.apache.log4j.Level.WARN)
   org.apache.log4j.Logger getLogger "akka" setLevel 
  (org.apache.log4j.Level.WARN)
   val spark =  SparkSession.builder
     .appName ("Sentiment")
     .master  ("local[9]")
     .getOrCreate

import spark.implicits._

val reviewSchema = StructType(Array(
        StructField ("reviewText", StringType, nullable=false),
        StructField ("overall",    DoubleType, nullable=false),
        StructField ("summary",    StringType, nullable=false)))

// Read file and merge the text abd summary into a single text column

def loadReviews (path: String): Dataset[ParsedReview] =
    spark
        .read
        .schema (reviewSchema)
        .json (path)
        .rdd
        .zipWithUniqueId
        .map[(Integer,String,Double)] { case (row,id) => (id.toInt, s"${row getString 2} ${row getString 0}", row getDouble 1) }
        .toDS
        .withColumnRenamed ("_1", "id" )
        .withColumnRenamed ("_2", "text")
        .withColumnRenamed ("_3", "overall")
        .as[ParsedReview]

 // Load the GLoVe embeddings file

 def loadGlove (path: String): Dataset[Embedding] =
     spark
         .read
         .text (path)
    .map  { _ getString 0 split " " }
    .map  (r => (r.head, r.tail.toList.map (_.toDouble))) // yuck!
         .withColumnRenamed ("_1", "word" )
         .withColumnRenamed ("_2", "vec")
         .as[Embedding]

def main(args: Array[String]) = {

  val glove  = loadGlove ("Data/glove.6B.50d.txt") // take glove 

  val reviews = loadReviews ("Data/Electronics_5.json") // FIXME

  // replace the following with the project code



   glove.show
   reviews.show

        spark.stop
   }

 }

I need to keep the line import org.apache.spark.sql.Dataset because some code depends on it but it is exactly because of it I have an exception throw.

My build.sbt file looks like this:

  name := "Sentiment Analysis Project"

  version := "1.1"

  scalaVersion := "2.11.12"

  scalacOptions ++= Seq("-unchecked", "-deprecation")

  initialCommands in console := 
  """
  import Main._
  """

   libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"

   libraryDependencies += "org.apache.spark" %% "spark-mllib" % 
   "2.3.0"

    libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.5"

    libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % 
    "test"

Upvotes: 6

Views: 1198

Answers (1)

Gonzalo
Gonzalo

Reputation: 3764

The Scala guide recommends you compile with Java8:

We recommend using Java 8 for compiling Scala code. Since the JVM is backward compatible, it is usually safe to use a newer JVM to run your code compiled by the Scala compiler for older JVM versions.

Although it's only a recommendation, I found it to fix the issue you mention.

In order to install Java 8 using Homebrew, it's best to use jenv which will help you handle multiple Java versions should you need to.

brew install jenv

Then run the following to add a tap (repository) of alternative versions of casks, since Java 8 is not in the default tap anymore:

brew tap homebrew/cask-versions

To install Java 8:

brew cask install homebrew/cask-versions/adoptopenjdk8

Run the following to add the previously installed Java version to jenv's list of versions:

jenv add /Library/Java/JavaVirtualMachines/<installed_java_version>/Contents/Home

Finally run

jenv global 1.8

or

jenv local 1.8

to use Java 1.8 globally or locally (in the current folder).

Fore more information, follow the instructions at jenv's website

Upvotes: 1

Related Questions