York
York

Reputation: 83

How to transform a item in RDD(Spark) to many items in RDD?

I get a RDD with 'Scan' action from 'HBase'. Each item in this RDD is like: x1, y1, y2, y3... So the items in this RDD are like(each line is a row result of scan action):

  1. x1, y1, y2, y3
  2. x2, y1, y4, y8, y9
  3. x3, y5
  4. ......
  5. xn, y1, y6, y100

I want transform this RDD to another RDD like:

  1. x1, y1
  2. x1, y2
  3. x1, y3
  4. x2, y1
  5. x2, y4
  6. ...
  7. xn, y1
  8. xn, y6
  9. xn, y100

How can I do this transform?

Upvotes: 0

Views: 77

Answers (1)

Justin Pihony
Justin Pihony

Reputation: 67115

"flatMap that sh*t":

rdd.flatMap(x => {
  val key = x.head
  x.tail.map(y=>(key,y))
})

Upvotes: 2

Related Questions