CODEWITHSUNDEEP
scalaapache-sparkrdd
oikonomiyaki
oikonomiyaki

Reputation: 7951

Invert map and reduceByKey in Spark-Scala

I'm have a CSV dataset that I want to process using Spark, the second column is of this format:

yyyy-MM-dd hh:mm:ss

I want to group each MM-dd 

val days : RDD = sc.textFile(<csv file>)

val partitioned = days.map(row => {

    row.split(",")(1).substring(5,10)

}).invertTheMap.groupOrReduceByKey

The result of groupOrReduceByKey is of form:

("MM-dd" -> (row1, row2, row3, ..., row_n) )

How should I implement invertTheMap and groupOrReduceByKey?

I saw this in Python here but I wonder how is it done in Scala?

Upvotes: 1

Views: 639

Answers (1)

Till Rohrmann
Till Rohrmann

Reputation: 13346

This should do the trick

val testData = List("a, 1987-09-30",
  "a, 2001-09-29",
  "b, 2002-09-30")

val input = sc.parallelize(testData)

val grouped = input.map{
  row =>
    val columns = row.split(",")

    (columns(1).substring(6, 11), row)
}.groupByKey()

grouped.foreach(println)

The output is

(09-29,CompactBuffer(a, 2001-09-29))
(09-30,CompactBuffer(a, 1987-09-30, b, 2002-09-30))

Upvotes: 1

Related Questions