CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqlexplode
dreddy
dreddy

Reputation: 463

Spark dataframe add a row for every existing row

I have a dataframe with following columns:

groupid,unit,height
----------------------
1,in,55
2,in,54

I want to create another dataframe with additional rows where unit=cm and height=height*2.54.

Resulting dataframe:

groupid,unit,height
----------------------
1,in,55
2,in,54
1,cm,139.7
2,cm,137.16

Not sure how I can use spark udf and explode here. Any help is appreciated. Thanks in advance.

Upvotes: 6

Views: 12231

Answers (1)

Ramesh Maharjan
Ramesh Maharjan

Reputation: 41987

you can create another dataframe with changes you require using withColumn and then union both dataframes as 

import sqlContext.implicits._
import org.apache.spark.sql.functions._

val df = Seq(
  (1, "in", 55),
  (2, "in", 54)
).toDF("groupid", "unit", "height")

val df2 = df.withColumn("unit", lit("cm")).withColumn("height", col("height")*2.54)

df.union(df2).show(false)

you should have 

+-------+----+------+
|groupid|unit|height|
+-------+----+------+
|1      |in  |55.0  |
|2      |in  |54.0  |
|1      |cm  |139.7 |
|2      |cm  |137.16|
+-------+----+------+

Upvotes: 11

Related Questions