CODEWITHSUNDEEP
dataframepyspark
DrGenius
DrGenius

Reputation: 967

Pyspark dataframe into hive table

How can I parse a pyspark df in a hive table? Also, is there any way to create a csv with header from my df?

I do not use pandas, my dfs are created with spark.sql() .

Upvotes: 2

Views: 2319

Answers (1)

notNull
notNull

Reputation: 31490

You can read hive table data in pyspark with df then write the df with header using .option("header","true").

Example:

df=spark.sql("select * from <db>.<hive_table>")

df.write.mode("overwrite").option("header","true").csv("<file_path>")

UPDATE:

#choosee mode either overwrite/append
df.write.mode("overwrite").saveAsTable("<hive_db>.<hive_table>")

#or using spark sql
df.createOrReplaceTempView("tmp")
spark.sql("insert into <hive_db>.<hive_table> select * from tmp")

Upvotes: 1

Related Questions