CODEWITHSUNDEEP
scaladataframeapache-spark
Tomasz
Tomasz

Reputation: 610

Add mapType column to existing DataFrame

I have got probably easy and quick question regarding the DataFrames in the Scala in Spark.

I have an existing Spark DataFrame (operate with Scala 2.10.5 and Spark 1.6.3) and would like to add a new column with the ArrayType or MapType, but don't know how to achieve that. But don't know how to deal with that. I would not like to create multiple columns with the 'single' values, but store them in one column. It would shorten my code and make it more changes prone.

import org.apache.spark.sql.types.MapType

...

// DataFrame initial creation
val df = ...

// adding new columns
val df_new = df
   .withColumn("new_col1", lit("something_to_add") // add a literal
   .withColumn("new_col2"), MapType("key1" -> "val1", "key2" -> "val2")) // ???

Upvotes: 0

Views: 1257

Answers (1)

dumitru
dumitru

Reputation: 2108

You could try something like 

val df_new = df
   .withColumn("new_col1", lit("something_to_add") // add a literal
   .withColumn("new_col2"), typedLit[Map[String, String]](Map("key1" -> "val1"), ("key2" -> "val2")))

Upvotes: 1

Related Questions