CODEWITHSUNDEEP
scalarestapache-spark
vdep
vdep

Reputation: 3590

Executing simple Spark Jobs in Hydrosphere Mist

I created spark job similar to this: https://github.com/Hydrospheredata/mist/blob/master/examples-spark2/src/main/scala/SimpleContext.scala

changed the import statments to:

import io.hydrosphere.mist.lib.spark2.{ContextSupport, MistJob}

as existing import import io.hydrosphere.mist.api.{ContextSupport, MistJob} cannot resolve.

Created jars using sbt package and added it to router.conf, which looks like:

first-job = {
  path = "path to jar"
  className = "SimpleContext$"
  namespace = "foo"
}

The curl command to run the above job fails returning

The requested resource could not be found.

However curl commands when pointed to the existing mist-examples-spark2.jar runs fine. Does this mean the problem lies while jar creation?. Has anyone able to run Mist Jobs?

The logger in console where i ran mist-master showed:

17-07-11 21:56:52 [mist-akka.actor.default-dispatcher-18] ERROR phere.mist.master.JobEndpoints:44 Job's loading failed for first-job
java.lang.NoClassDefFoundError: io/hydrosphere/mist/lib/spark2/MistJob
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
.
.

Is there a problem with import statments?

Upvotes: 1

Views: 485

Answers (1)

Vadim Chelyshov
Vadim Chelyshov

Reputation: 26

Currently, we are working to make our library better and it leads to that it can have binary incompatibility between versions. It means that you should ensure that your local mist installation( or version of docker image) is same as the version of mist-lib that you use to build job.

Upvotes: 1

Related Questions