Reputation: 2978
I created Spark DataFrame in Scala using Databricks. After doing some preprocessing,I came up with a smaller data subset that fits into memory. Therefore I want to convert it to Pandas and then save as CSV file.
The problem is that the DataFrame df
on which I worked in Databricks notebook in Scala cells is not visible in a Python cell.
%python
df.toPandas().to_csv("dbfs:/FileStore/tables/test.csv", header=True, index=False)
How can I make df
visible in the Python cell?
Upvotes: 0
Views: 1232
Reputation: 272
Do this display(df)
. It usually displays some nested Structs as well.
Or I would do something like this
df.createOrReplaceTempView("dfViewName")
In the next cell
%sql
Select * from dfViewName
Upvotes: 2