marshall
marshall

Reputation: 11

Spark RDD: Get row number

How do I get the row number out of an RDD that I'm currently processing:

val rdd2 = rdd1
  .filter(row => {
          // get row number
      }
      true
  })

Upvotes: 1

Views: 2830

Answers (1)

Shyamendra Solanki
Shyamendra Solanki

Reputation: 8851

val rdd2 = rdd1.zipWithIndex.filter{ 
    case (row, index) => {
      // row number is index. (but is not fixed, unless RDD is sorted)
}

Upvotes: 4

Related Questions