CODEWITHSUNDEEP
pysparktimestamp
jgtrz
jgtrz

Reputation: 375

pySpark output file with timestamp name

I need to name my outputfile with the timestamp but getting an error. Not sure what I'm doing wrong

timestamp = spark.sql("select string(date_format(current_timestamp,'yyyy/MM/dd_HH:mm:ss'))").collect()[0][0]
print(timestamp)



Error: ADLException: Error getting info for file 
/06/05_13:14:01

no error if I use current date instead of timestamp. But I need timestamp

Upvotes: 0

Views: 183

Answers (1)

Steven
Steven

Reputation: 15258

some caracters are not allowed in file naming:

#L1234_ABC123_2020/06/05_13:14:01 is not vaide. Try something like #L1234_ABC123_20200605_131401 for example, or with underscore _. colons : are not allowed basically.

Upvotes: 2

Related Questions