CODEWITHSUNDEEP
pythonpandasapache-sparkapache-spark-sqldatabricks
Fluxy
Fluxy

Reputation: 2978

How to make a DataFrame visible in Python cell in Databricks notebook?

I created Spark DataFrame in Scala using Databricks. After doing some preprocessing,I came up with a smaller data subset that fits into memory. Therefore I want to convert it to Pandas and then save as CSV file.

The problem is that the DataFrame df on which I worked in Databricks notebook in Scala cells is not visible in a Python cell.

%python

df.toPandas().to_csv("dbfs:/FileStore/tables/test.csv", header=True, index=False)

How can I make df visible in the Python cell?

Upvotes: 0

Views: 1232

Answers (1)

Harsha TJ
Harsha TJ

Reputation: 272

Do this display(df) . It usually displays some nested Structs as well.

Or I would do something like this df.createOrReplaceTempView("dfViewName") In the next cell %sql

Select * from dfViewName

Upvotes: 2

Related Questions