CODEWITHSUNDEEP
scalaapache-spark
syv
syv

Reputation: 3608

Add new Column with the constant value as list

Im creating list like

var transactionList = result.select(col("transaction_id")).distinct().collect().map(_(0)).toList

I want to insert "transactionList" into a Dataframe and latter explode it

I tried it like 

df.withColumn("transactionList" , ArrayType(for (id <- transactionList) lit(id))

But it is not working

Upvotes: 0

Views: 470

Answers (1)

koiralo
koiralo

Reputation: 23119

You should also replace .map(_(0)) with .map(_.getString(0)) 

result.select(col("transaction_id")).distinct().collect().map(.getString(0))

You can use lit to convert the literal value to Column 

df.withColumn("transactionList", lit(transactionList))

If you have transactionList = List("a", "b")

This will add a new Column transactionList as array with value (a, b) in all row.

/**
   * Creates a [[Column]] of literal value.
   *
   * The passed in object is returned directly if it is already a [[Column]].
   * If the object is a Scala Symbol, it is converted into a [[Column]] also.
   * Otherwise, a new [[Column]] is created to represent the literal value.
   *
   * @group normal_funcs
   * @since 1.3.0
   */

Upvotes: 1

Related Questions