CODEWITHSUNDEEP
javaapache-spark
Kabir
Kabir

Reputation: 61

Read contents of a directory in Spark

I am new to spark just wanted to know how can we read contents of a directory and iterate over them. C# corresponding code

Foreach(var path in Directory. EnumerateFiles(directory, *,.. ) {} ;

Upvotes: 3

Views: 69

Answers (1)

Amit Kumar
Amit Kumar

Reputation: 178

  JavaSparkContext jsc = new JavaSparkContext(sc);
  JavaPairRDD<String,String> rdd = jsc.wholeTextFiles(path);
          for(Tuple2<String, String> str : rdd.toArray()) {           System.out.println("+++++++++++++++++++++++++++++++++++++++++++");
      System.out.println("File name " + str._1);
      System.out.println("+++++++++++++++++++++++++++++++++++++++++++");
      System.out.println();
      System.out.println("-------------------------------------------");
      System.out.println("content " + str._2);
      System.out.println("-------------------------------------------");
  }

Hope it helps, I had the same question.

Upvotes: 1

Related Questions