CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
sah.stc
sah.stc

Reputation: 115

pyspark add data to a TempTable

I have created a temporary table from a DataFrame

df1 = sqlContext.createDataFrame([(146,000000)],['id','size'])
df1.registerTempTable("table_test")

Now I want to add a new dataframe to the existing tempTable.

df2 = sqlContext.createDataFrame([(147,000001)],['id','size'])

I tried to do the following

df2.write.mode("append").saveAsTable("table_test")

But then realized that one can do that only for persistent tables.

Is it possible to add new data to an existing tempTable. If so, how?

Upvotes: 2

Views: 23066

Answers (3)

kjs
kjs

Reputation: 51

 df3=df1.union(df2) 
 df3.registerTempTable("table_test")
 hc.sql("select * from table_test").show()

+---+----+
| id|size|
+---+----+
|146|   0|
|147|   1|
+---+----+

Upvotes: 3

Ged
Ged

Reputation: 18108

union for spark 2 or higher, unionAll for 1.x. With same sets of cols.

Upvotes: 0

Pushkr
Pushkr

Reputation: 3619

You can just union two dataframes and re-register temp table

df1.union(df2).registerTempTable("table_test")

Upvotes: 0

Related Questions