CODEWITHSUNDEEP
scalaapache-spark
Vinny
Vinny

Reputation: 134

Lost task in 1.0 in stage 0.0

I'm very new to Spark and Scala. What I'm trying to do is to make a Simple WordCount work.

package demo
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.SparkContext._

object WordCount extends App {
  // Create a Scala Spark Context.
  val conf = new SparkConf().setAppName("wordCount").setMaster("spark://MyHost")
  val sc = new SparkContext(conf)
  // Load our input data.
  val input = sc.textFile("words.txt")
  // Split it up into words.
  val words = input.flatMap(line => line.split(" "))
  // Transform into word and count.
  val counts = words.map(word => (word, 1)).reduceByKey{case (x, y) => x + y}
  // Save the word count back out to a text file, causing evaluation.
  counts.saveAsTextFile("wordsCount.txt")
}

I have setup a Master and started one Worker that is connected to the Master.

Spark Master

When I run the example I get the following error:

Lost task 1.0 in stage 0.0 (TID 1, MyHost): java.lang.ClassNotFoundException: demo.WordCount$$anonfun$2

And I really don't know what it means.

My build.sbt looks like this:

name := "SimpleWordCount"

version := "0.0.1"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.2.0" % "provided"
)

Upvotes: 0

Views: 2536

Answers (2)

David
David

Reputation: 11

You need set the necessary jars into spark context

val conf = new SparkConf().setAppName("wordCount").setMaster("spark://MyHost").setJars(Seq("$HOME/WordCount.jar",...))

Upvotes: 1

uberwach
uberwach

Reputation: 1109

The code looks fine, the error tells you that he cannot find one of the closures defined in the method (which are represented as anonymous classes right now, thus the "ClassNotFound" error). It seems to be either a version and/or a deployment problem. My guess is that the jar code was not deployed to the spark nodes.

First of all in build.sbt use version "1.2.1" for spark (as your master node does), secondly check http://spark.apache.org/docs/1.2.1/submitting-applications.html on how to submit your jar to the server, and third you can leave out the "setMaster" in your code as the submit script will deal with it for now.

Upvotes: 2

Related Questions