CODEWITHSUNDEEP
scalaapache-sparkcassandraspark-jobserver
Jakob Olsson
Jakob Olsson

Reputation: 31

Context error on spark-jobserver cluster

we are working on launching a spark-jobserver on a spark cluster.

To deploy the server we follow the documentation at github.com/spark-jobserver/spark-jobserver#deployment. We deploy with by running ./server_deploy.sh local.

I've uploaded our local.conf and local.sh to pastebin -

local.conf - http://pastebin.com/DWJEuX11

local.sh - http://pastebin.com/S2hjXb8J

We then launch our cluster master and worker running the following commands from the spark root folder

./sbin/master-start.sh  
./bin/spark-class org.apache.spark.deploy.worker.Worker   spark://IP:PORT

Checking localhost:8080 it appears to be working fine.

We then start the server by running ./server_start.sh --master spark://IP:PORT --deploy-mode cluster, again checking localhost:8080 we can see that the server is up and running on one core on the worker. Checking localhost:8090 we can see that the spark-jobserver is also up and running.

We then create a custom context which includes CassandraSQLContext (github.com/datastax/spark-cassandra-connector/blob/master/spark-cassandra-connector/src/main/scala/org/apache/spark/sql/cassandra/CassandraSQLContext.scala) by running

curl -d "" 'localhost:8090/contexts/cassandra-context?context-factory=spark.jobserver.context.CassandraContextFactory&num-cpu-cores=4&memory-per-node=512m'

We then receive the following error

{
    "status": "CONTEXT INIT ERROR",
    "result": {
        "errorClass": "java.lang.NoClassDefFoundError",
        "cause": "org.apache.spark.sql.cassandra.CassandraSQLContext",
        "stack":  ["java.net.URLClassLoader.findClass(URLClassLoader.java:381)"...
    }
}

We've tried fixing this error by including a dependency jar in local.conf 

dependent-jar-uris = ["file:///path/to/jar.jar"]

This gave the same error.

We have also tried including the entire spark-cassandra-connector src folder in our job-server-extras, which gave the error File line length exceeds 110 characters on almost every file when running ./server_deploy.sh local.

We would appreciate any help possible.

Upvotes: 1

Views: 764

Answers (1)

RussS
RussS

Reputation: 16576

I would just add the --packages line to the spark-submit used to start the JobServer. Probably the easiest way to to work around that.

See

  1. https://github.com/spark-jobserver/spark-jobserver/blob/master/bin/server_start.sh
  2. http://spark-packages.org/package/datastax/spark-cassandra-connector

You can just add --packages to your server start script ala

./server_start.sh --packages datastax:spark-cassandra-connector:VersionYouWant

Upvotes: 0

Related Questions