CODEWITHSUNDEEP
pythonscalapysparkdatabricks
iKnowNothing
iKnowNothing

Reputation: 79

How to Use DataFrame Created in Scala in Databricks' PySpark

My Databricks notebook is on Python. Some codes in the notebook are written in Scala (using the %scala) and one of them is for creating dataframe.

If I use Python/PySpark (the default mode) again, how can I use / access this dataframe that was created when it was on scala mode?

Is it even possible?

Thanks

Upvotes: 2

Views: 1382

Answers (1)

Raphael K
Raphael K

Reputation: 2353

You can access DataFrames created in one language with another language through temp tables in SparkSQL.

For instance, say you have a DataFarame in scala called scalaDF. You can create a temporary view of that and make it accessible to a Python cell, for instance:

scalaDF.createOrReplaceTempView("my_table")

Then in a Python cell you can run

pythonDF = spark.sql("select * from my_table")

pythonDF.show()

The same works for passing dataframes between those languages and R. The common construct is a SparkSQL table.

Upvotes: 6

Related Questions