CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
JDoe
JDoe

Reputation: 493

TypeError: sql() missing 1 required positional argument: 'sqlQuery' in pyspark

I'm trying to estimate proportion of maths marks in pyspark 3.0.1 on data bricks. Some of the cases my total_marks are 0.So I wrote following code

 df_data.registerTempTable('myTable')
 df_oversees2=SQLContext.sql("select A.*,case when total_marks=0 then 0 else (maths_marks/total_marks) end as prop_maths from myTable A ")

But I'm getting error message

TypeError: sql() missing 1 required positional argument: 'sqlQuery'

Can you please help me to resolve the issue? I also verified that my total_marks & maths_marks are present in my table using below code

df1 = sqlContext.sql("select maths_marks,total_marks from myTable")
df1.show()

I'm seeing the output for both the fields

Upvotes: 1

Views: 5488

Answers (1)

mck
mck

Reputation: 42332

SQLContext is different from sqlContext - the latter is an SQLContext attached to a SparkContext, while the former is a class not attached to any SparkContext. Therefore, you need to do:

sqlContext = SQLContext(sc)
df_oversees2 = sqlContext.sql("your query")

In any case, it is deprecated and the better way would be to call

spark.sql("your query")

where spark is a sparkSession object.

Upvotes: 5

Related Questions