CODEWITHSUNDEEP
apache-spark
hawarden_
hawarden_

Reputation: 2170

Spark : put hashmap into Dataset column?

I have a dataset Dataset<Row> which comes from reading a parquet file. Knowing that one column inside InfoMap is of type Map.

Now I want to update this column, but when I use withColumn, it tells me that I cannot put a hashmap inside because it's not a litteral.

I want to know what is the correct way to update a column of type Map for a dataset ?

Upvotes: 1

Views: 552

Answers (1)

Nir Hedvat
Nir Hedvat

Reputation: 870

Try using typedLit instead of lit

typedLit

"...The difference between this function and lit() is that this function can handle parameterized scala types e.g.: List, Seq and Map"

data.withColumn("dictionary", typedLit(Map("foo" -> 1, "bar" -> 2)))

Upvotes: 3

Related Questions