CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
harinath reddy
harinath reddy

Reputation: 1

Apache Spark SQL Query Returns No Results for a Column in Azure Synapse Notebook

I'm running an Apache Spark SQL query in an Azure Synapse Notebook to retrieve data from an Azure Synapse table.

%%pyspark
df = spark.sql(""" 
    SELECT scheduledend 
    FROM `database`.`email` 
    WHERE scheduledend IS NOT NULL
""")

df.show(truncate = False)

The Pyspark query is returning zero rows.

Datatype of the column in database is datetime2, I tried casting column to different datatypes with in pyspark code.

However, the query works fine when queried directly in the database (in SSMS by connecting to synapse) and returns the expected results (4569 rows).

SELECT scheduledend FROM database.email WHERE scheduledend IS NOT NULL

Upvotes: 0

Views: 22

Answers (0)

Related Questions