Reputation: 11
How do I get the row number out of an RDD that I'm currently processing:
val rdd2 = rdd1
.filter(row => {
// get row number
}
true
})
Upvotes: 1
Views: 2830
Reputation: 8851
val rdd2 = rdd1.zipWithIndex.filter{
case (row, index) => {
// row number is index. (but is not fixed, unless RDD is sorted)
}
Upvotes: 4