CODEWITHSUNDEEP
sqlazurepysparkazure-synapse
Amir
Amir

Reputation: 441

Cannot access to the table with pyspark in azure

I created a Databse with a table inside it in azure synaps which was an external table of sample azure data.

enter image description here

I run the code below to get access to the data in this table through pyspark

df = spark.sql("select * FROM greentaxidb.dbo.taxitable")

when I run the code above I get:

pyspark.sql.utils.AnalysisException: The namespace in session catalog must have exactly one name part: greentaxidb.dbo.taxitable

Upvotes: 1

Views: 2403

Answers (1)

NiharikaMoola
NiharikaMoola

Reputation: 5074

Load the data into dataframe with saveAsTable to run it in pyspark notebook.

%%pyspark
df = spark.read.load('abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<filename>', format='parquet')
df.write.mode("overwrite").saveAsTable("testdb.test1")

--

%%pyspark
df = spark.sql("select * from testdb.test1")
display(df)

enter image description here

You can go through the links from here and here for more details.

Upvotes: 1

Related Questions