CODEWITHSUNDEEP
scalaapache-sparkjupyteralmond
Vadim
Vadim

Reputation: 825

SparkSession is not initialized with almond Jupyter

I'm trying to launch Almond Jupiter through docker or on line from https://almond.sh. In the spark.ipynb image, an error appears on the line with NotebookSparkSession

import $ivy.`org.apache.spark::spark-sql:2.4.0`
import $ivy.`sh.almond::almond-spark:0.3.0`

import org.apache.log4j.{Level, Logger}
Logger.getLogger("org").setLevel(Level.OFF)

import org.apache.spark.sql._

val spark = {
  NotebookSparkSession.builder()
    .master("local[*]")
    .getOrCreate()
}

There is the exception with docker:

java.lang.NoSuchMethodError: coursier.package$Resolution$.apply$default$13()Lscala/collection/immutable/Map;
  org.apache.spark.sql.ammonitesparkinternals.SparkDependencies$.sparkJars(SparkDependencies.scala:134)
  org.apache.spark.sql.ammonitesparkinternals.AmmoniteSparkSessionBuilder.getOrCreate(AmmoniteSparkSessionBuilder.scala:234)
  org.apache.spark.sql.almondinternals.NotebookSparkSessionBuilder.getOrCreate(NotebookSparkSessionBuilder.scala:62)

I tried to use on line version with the same spark.ipynb but there is an exception

java.lang.AssertionError: assertion failed: 
  NotebookSparkSession.builder()
     while compiling: cmd3.sc
        during phase: superaccessors
     library version: version 2.12.8
    compiler version: version 2.12.8
  reconstructed args: -nowarn -Yresolve-term-conflict:object

  last tree to typer: This(class cmd3)
       tree position: line 19 of cmd3.sc
            tree tpe: cmd3.this.type
              symbol: final class cmd3 in package $sess
   symbol definition: final class cmd3 extends Serializable (a ClassSymbol)
      symbol package: ammonite.$sess
       symbol owners: class cmd3
           call site: class Helper in class cmd3 in package $sess

Upvotes: 2

Views: 829

Answers (1)

Ram Ghadiyaram
Ram Ghadiyaram

Reputation: 29165

Clear indication of scala version (2.12.8) mismatch issue with spark 2.4.0 . see the release notes of spark 2.4

try to lower the version i.e 2.11 (Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.4.0 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).)

it has experimental support to 2.12 scala version

or I think you need to use spark 2.4.3 for support of scala 2.12.8 as said by documentation.

Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.4.3 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).

Upvotes: 1

Related Questions