CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqlkey-value
Arvinth
Arvinth

Reputation: 70

Error while saving map value in Spark dataframe(scala) - Expected column, actual Map[int,string]

I have key values pair in Map[int,string]. I need to save this value in Hive table using spark dataframe. But i am getting error - Expected column, actual Map[int,string]

Code:

val dbValuePairs = Array(2019,10)
val dbkey = dbValuePairs.map(x => x).zipWithIndex.map(t => (t._2, t._1)).toMap

val dqMetrics = spark.sql("select * from dqMetricsStagingTbl")
  .withColumn("Dataset_Name", lit(Dataset_Name))
  .withColumn("Key", dbkey)
dqMetrics.createOrReplaceTempView("tempTable")

spark.sql("create table if not exists hivetable AS select * from tempTable")

dqMetrics.write.mode("append").insertInto(hivetable)

Please help! Error in withColumn("Key", dbkey) line

Upvotes: 0

Views: 297

Answers (1)

Boris Azanov
Boris Azanov

Reputation: 4501

Look at Spark function withColumn signature:

def withColumn(colName: String, col: Column): DataFrame

it takes two arguments: colName as String and col as Column.

your dbkey type is Map[Int, Int] that is not Column:

val dbkey: Map[Int, Int] = dbValuePairs.map(x => x).zipWithIndex.map(t => (t._2, t._1)).toMap

if you want store Map in your table column you can use map function which takes a sequence of Column:

// from object org.apache.spark.sql.functions
def map(cols: Column*): Column

so you can convert your dbkey to Seq[Column] and pass it to withColumn function:

val dbValuePairs = Array(2019,10)
val dbkey: Map[Int, Int] = dbValuePairs.map(x => x).zipWithIndex.map(t => (t._2, t._1)).toMap
val dbkeyColumnSeq: Seq[Column] = dbkey.flatMap(t => Seq(lit(t._2), lit(t._1))).toSeq

val dqMetrics = spark.sql("select * from dqMetricsStagingTbl")
  .withColumn("Dataset_Name", lit(""))
  .withColumn("Key", map(dbkeyColumnSeq:_*))

Upvotes: 2

Related Questions