CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Vikhyat Srivastava
Vikhyat Srivastava

Reputation: 13

Multiple parameter in IN clause of Spark SQL from parameter file

I am trying to run spark query where I am creating curated table from a source table based upon values in parameter file.

properties_file.properties contains below key values:

substatus,allow,deny

SparkQuery is

//Code to load property file in parseConf

spark.sql(s"""insert into curated.table from source.table where 
substatus='${parseConf.substatus}'""")

Above works with single value in substatus. But Can someone help what shall i do if I need to use substatus in ${parseConf.substatus} for multiple values from param as below.

spark.sql(s"""insert into curated.table from source.table where substatus in '${parseConf.substatus}'""")

Upvotes: 0

Views: 630

Answers (2)

Divyaansh Bajpai
Divyaansh Bajpai

Reputation: 232

Equalto operator expects 1 value to be passed other than directly reading the value from parameter file who make in pass a one string. You need to break the values and then use IN clause inplace of equalto(=).

Upvotes: 0

Vikhyat Srivastava
Vikhyat Srivastava

Reputation: 13

To resolve my problem, I updated my property file as:
substatus,'allow'-'deny'
Then in scala code, I implemented below logic:
val subStatus=(parseConf.substatus).replace('-',',')
spark.sql(s"""insert into curated.table from source.table where substatus in ('${subStatus}')""")
Above strategy helped in breaking the values in string to muliple parameters of IN clause.

Upvotes: 1

Related Questions