CODEWITHSUNDEEP
pythonsqlapache-sparkpyspark
Victor
Victor

Reputation: 17107

Cannot pass variables to a spark sql query in pyspark

I have a python variable (I am using pyspark) of date datatype: The variable value is 2016-10-31

print type(load_dt)

 >> <type 'datetime.date'>

I am having difficulty passing this to a sparksql query:

    hive_context.sql("select * from  tbl t1 where cast (substring(t1.dt,1,10) as date) ={0}".format(load_dt));

    Error:

    u"cannot resolve '(cast(substring(dt,1,10) as date) = ((2016 - 10) - 31))' due to data type mismatch: differing types in '(cast(substring(period_dt,1,10) as date) = ((2016 - 10) - 31))'
 (date and int)

Upvotes: 1

Views: 3209

Answers (1)

Alper t. Turker
Alper t. Turker

Reputation: 35249

Add quotes:

"select * from  tbl t1 where cast (substring(t1.dt,1,10) as date) = '{0}'"

otherwise date is converted to 2016-10-31 string and interpreted as arithmetic expression:

2016 - 10 - 31

Upvotes: 2

Related Questions