CODEWITHSUNDEEP
pysparkapache-spark-sqldatabricks
CloverCeline
CloverCeline

Reputation: 541

pyspark save dataframe to hive table using variable in the name

I am using databrick pyspark for coding wondering how could I pass the variable value to the name of the table which I want to save in the Azure

I am able to use this if that is the fix table name 

result.write.format("parquet").mode("overwrite").saveAsTable("result0911")

but I hope to do this

Having a variable: time = datetime.today() the value of that is "2019-09-11 12:10:48.969602"

I want to first format the "time" to "20190911121048" which is good for the name of a table (I need the time also, because people may save the records more than once per day)

then use that value "20190911121048" table name which I want to save the table name should be "result_20190911121048"

any suggestion? Thank you the following does not work

result.write.format("parquet").mode("overwrite").saveAsTable("result_%s time")

Upvotes: 1

Views: 1653

Answers (2)

CloverCeline
CloverCeline

Reputation: 541

Thank you for the input. I am able to use this:

time = datetime.today()
t=time.strftime("%Y%m%d%H%M")
result_name ="recommendation_%s" % (t)
final_recon.write.format("parquet").mode("overwrite").saveAsTable(result_name)

Upvotes: 1

ravi malhotra
ravi malhotra

Reputation: 723

Say you have 2 variables having values 

time = "20190911121048"
table_name = 'result'

Then you can make table name as :

final_table_name = '{}_{}'.format(table_name, time)

result.write.format("parquet").mode("overwrite").saveAsTable(final_table_name)

Upvotes: 2

Related Questions