CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Erik Barajas
Erik Barajas

Reputation: 193

How to append List[String] to every row of DataFrame?

After a series of validations over a DataFrame,

I obtain a List of String with certain values like this:

List[String]=(lvalue1, lvalue2, lvalue3, ...)

And I have a Dataframe with n values:

dfield 1  | dfield 2  | dfield 3
___________________________
dvalue1   | dvalue2   | dvalue3
dvalue1   | dvalue2   | dvalue3

I want to append the values of the List at the beggining of my Dataframe, in order to get a new DF with something like this:

dfield 1  | dfield 2  | dfield 3 | dfield4 | dfield5 | dfield6
__________________________________________________________
lvalue1   | lvalue2   | lvalue3  | dvalue1 | dvalue2 | dvalue3
lvalue1   | lvalue2   | lvalue3  | dvalue1 | dvalue2 | dvalue3

I have found something using a UDF. Could be this correct for my purpose?

Regards.

Upvotes: 0

Views: 1326

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74709

TL;DR Use select or withColumn with lit function.

I'd use lit function with select operator (or withColumn).

lit(literal: Any): Column Creates a Column of literal value.

A solution could be as follows.

val values = List("lvalue1", "lvalue2", "lvalue3")
val dfields = values.indices.map(idx => s"dfield ${idx + 1}")

val dataset = Seq(
  ("dvalue1", "dvalue2", "dvalue3"),
  ("dvalue1", "dvalue2", "dvalue3")
).toDF("dfield 1", "dfield 2", "dfield 3")

val offsets = dataset.
  columns.
  indices.
  map { idx => idx + colNames.size + 1 }

val offsetDF = offsets.zip(dataset.columns).
  foldLeft(dataset) { case (df, (off, col)) => df.withColumnRenamed(col, s"dfield $off") }

val newcols = colNames.zip(dfields).
  map { case (v, dfield) => lit(v) as dfield } :+ col("*")

scala> offsetDF.select(newcols: _*).show
+--------+--------+--------+--------+--------+--------+
|dfield 1|dfield 2|dfield 3|dfield 4|dfield 5|dfield 6|
+--------+--------+--------+--------+--------+--------+
| lvalue1| lvalue2| lvalue3| dvalue1| dvalue2| dvalue3|
| lvalue1| lvalue2| lvalue3| dvalue1| dvalue2| dvalue3|
+--------+--------+--------+--------+--------+--------+

Upvotes: 3

Related Questions