CODEWITHSUNDEEP
pythonapache-sparkpysparkapache-spark-sqlpyodbc
Robby star
Robby star

Reputation: 279

converting python code to python spark code

Below code is in Python and i want to convert this code to pyspark, basically i'm not sure what will be the codefor the statement - pd.read_sql(query,connect_to_hive) to convert into pyspark

Need to extract from data from the EDL, so making the connection to the EDL using PYODBC and them extract the data using sql query.

pyodbc connection to the Enterprise Data Lake:

connect_to_hive = pyodbc.connect("DSN=Hive", autocommit=True)
transaction=pd.read_sql(query, connect_to_hive)
connect_to_hive.close()

#Query function: Below is just a basic sql query to replicate this problem.

query=f'''
with trans as (
    SELECT
        a.employee_name,
        a.employee_id
    
    FROM EMP
'''

Upvotes: 2

Views: 3272

Answers (1)

Yayati Sule
Yayati Sule

Reputation: 1631

The above code can be converted to SparkSQL code as follows:

spark = SparkSession.builder.enableHiveSupport().getOrCreate()

query=f'''
with trans as (
    SELECT
        a.employee_name,
        a.employee_id
    
    FROM EMP
'''

employeeDF = spark.sql(query)

employeeDF.show(truncate=False)

The query would be run as is on Hive and the result shall be available to you as a Spark DataFrame

Upvotes: 2

Related Questions