CODEWITHSUNDEEP
apache-sparkhivehiveql
Parthiban Thee
Parthiban Thee

Reputation: 21

How to convert two columns in a row having a range value to multiple incrementing rows based on the range

I have two columns in a row which have range values. eg: from_series: 100 and to_series: 110. Now I need to convert these to multiple rows.

name value from_series to_series
 aaa  32      100        110

I need to convert as shown below:

name value series
 aaa  32     100
 aaa  32     101
 aaa  32     102 until 110

Upvotes: 2

Views: 40

Answers (1)

Shivansh
Shivansh

Reputation: 3544

I think this code snippet should work where df is your dataFrame. 

  df.rdd.flatMap{row=>
    val name = row.getAs[String]("name")
    val value = row.getAs[Int]("value")
    val fromSeries = row.getAs[Int]("from_series")
    val toSeries = row.getAs[Int]("to_series")
    (fromSeries to toSeries).map(seriesValue=>(name,value,seriesValue))
    }

Upvotes: 1

Related Questions