CODEWITHSUNDEEP
pythonapache-sparkpysparkapache-spark-sqldatabricks
Ashley O
Ashley O

Reputation: 1200

Pass variables from Scala to Python in Databricks

I'm using Databricks and trying to pass a dataframe from Scala to Python, within the same Scala notebook. I passed a dataframe from Python to Spark using: 

%python
python_df.registerTempTable("temp_table")


val scalaDF = table("temp_table")

How do I do that same thing in reverse? Thank you so much!!

Upvotes: 12

Views: 12686

Answers (2)

Daniel Mcauley
Daniel Mcauley

Reputation: 196

You can make use of the .createOrReplaceTempView() method or sql().

Here is an example to pass a dataframe through from scala, python, onto sql with a modification along the way ...and back to scala.

%scala 
var df = spark.range(0,10).selectExpr("*","'scala' language_origin")
df.createOrReplaceTempView("tableName")
display(df)

%python
df = sql("select * from tableName")
df2 = df.selectExpr("*","'python' language_added")
df2.createOrReplaceTempView("tableName")
display(df2)

%sql
create or replace temp view tableName as
select *, 'sql' language_added from tableName;
select * from tableName

%scala
df = sql("select * from tableName")
display(df)

Upvotes: 1

Alper t. Turker
Alper t. Turker

Reputation: 35249

The reverse will pretty much the same. In Scala:

scalaDF.registerTempTable("some_table")

In Python:

spark.table("some_table")

If you use recent Spark version you should use createOrReplaceTempView in place of registerTempTable.

Upvotes: 14

Related Questions