CODEWITHSUNDEEP
pythonpysparktimestamp
Vlad Aleshin
Vlad Aleshin

Reputation: 411

Pyspark convert string to timestamp

Need to convert string column in format '12/1/2010 8:26' into timestamp. Try to use following code:

F.to_timestamp(dataset.InvoiceDate,'MM/dd/yyyy HH:mm')

but get an error

Py4JJavaError: An error occurred while calling o640.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 123.0 failed 1 times, most recent failure: Lost task 0.0 in stage 123.0 (TID 119) (13c59da6fb19 executor driver): org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '12/1/2010 8:26' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.

How can I convert string to timestamp in this case?

Upvotes: 0

Views: 200

Answers (1)

Luiz Viola
Luiz Viola

Reputation: 2416

try:

F.to_timestamp(dataset.InvoiceDate,'M/d/y H:m')

Upvotes: 1

Related Questions