CODEWITHSUNDEEP
dataframescalaapache-sparkdatabricks
Void S
Void S

Reputation: 802

Databricks- Convert Python DataFrame to Scala DataFrame

I have a dataframe in python, df, that i want to pass to be able to use in % scala.

I have tried -

%python
pyDf.createOrReplaceTempView("testDF") // error message

Upvotes: 0

Views: 1506

Answers (2)

Venu A Positive
Venu A Positive

Reputation: 3062

it's not too difficult. I am sharing a sample code pls try. It's working in Pycharm or databricks.

from pyspark.sql import *

import pandas as pd

spark = SparkSession.builder.master("local").appName("testing").getOrCreate()

data = [['venu', 50], ['renu', 45], ['anu', 54],['bhanu',14]]

Create the pandas DataFrame

pdf= pd.DataFrame(data, columns = ['Name', 'Age'])

print(pdf)

Python Pands convert to Spark Dataframe.

sparkDF=spark.createDataFrame(pdf)

sparkDF.printSchema()

sparkDF.show()

enter image description here

Upvotes: 1

Alex Ott
Alex Ott

Reputation: 87154

Just query it with spark.sql:

val scalaDf = spark.sql("select * from testDF")

Upvotes: 1

Related Questions