CODEWITHSUNDEEP
pythonsqlapache-sparkdatabricks
Semihcan Doken
Semihcan Doken

Reputation: 828

How to convert sql table into a pyspark/python data structure and return back to sql in databricks notebook

I am running a sql notebook on databricks. I would like to analyze a table with half a billion records in it. I can run simple sql queries on the data. However, I need to change the date column type from str to date.

Unfortunately, update/alter statements do not seem to be supported by sparkSQL so it seems I cannot modify the data in the table.

What would be the one-line of code that would allow me to convert the SQL table to a python data structure (in pyspark) in the next cell? Then I could modify the file and return it to SQL.

Upvotes: 1

Views: 11978

Answers (2)

srikanth holur
srikanth holur

Reputation: 770

df=sqlContext.sql("select * from table")

To convert dataframe back to sql view,

df.createOrReplaceTempView("myview")

Upvotes: 1

David
David

Reputation: 11573

dataFrame = sqlContext.sql('select * from myTable')

Upvotes: 5

Related Questions