CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Frank
Frank

Reputation: 4417

Spark: select with key in map

I need to select with the key geo.cc in a map in a DataFrame:

 |-- params: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

but the value of the key, with its dot in the middle, seems to confuse Spark. If I write:

X.filter(X("params.geo.cc") === "us")

I get the error:

org.apache.spark.sql.AnalysisException: Can't extract value from params#3[geo];

What can I do? (needless to say, I do not control the key, i.e. I cannot change that geo.cc string to e.g. geo_cc.

Upvotes: 8

Views: 6663

Answers (2)

M. Alexandru
M. Alexandru

Reputation: 624

Try this :

X.filter(col("params")("key") === "geo.cc" && col("params")("value") === "us")

Upvotes: 6

Alper t. Turker
Alper t. Turker

Reputation: 35249

You should use apply:

val df = Seq((1L, Map("geo.cc" -> "US"))).toDF("id", "params")

df.select($"params"("geo.cc") === "US").show
// +-----------------------+
// |(params['geo.cc'] = US)|
// +-----------------------+
// |                   true|
// +-----------------------+

or getItem 

df.select($"params".getItem("geo.cc") === "US").show
// +-----------------------+
// |(params['geo.cc'] = US)|
// +-----------------------+
// |                   true|
// +-----------------------+

on a specific column, not DataFrame.

Upvotes: 6

Related Questions