CODEWITHSUNDEEP
scalaapache-spark
c0rp
c0rp

Reputation: 501

Why saveAsTextFile do nothing?

I'm trying to implement simple WordCount in Scala + Spark. Here is my code

object FirstObject {
  def main(args: Array[String]) {
    val input = "/Data/input"
    val conf = new SparkConf().setAppName("Simple Application")
                              .setMaster("spark://192.168.1.162:7077")
    val sparkContext = new SparkContext(conf)
    val text = sparkContext.textFile(input).cache()
    val wordCounts = text.flatMap(line => line.split(" "))
                         .map(word => (word,1))
                         .reduceByKey((a,b) => a+b)
                         .sortByKey()
    wordCounts.saveAsTextFile("/Data/output")

  }

This job is working for 54s, and finally do nothing. Is is not writing output to /Data/output

Also if I replace saveAsTextFile with forEach(println) it is produce desired output.

Upvotes: 0

Views: 258

Answers (1)

psisoyev
psisoyev

Reputation: 2168

You should check your user rights for /data/output folder. This folder should have writing rights for your specific user.

Upvotes: 1

Related Questions