CODEWITHSUNDEEP
pythonpython-3.xapache-sparkdatabricksazure-databricks
nam
nam

Reputation: 23809

ParseException when changing data type in Spark Dataframe

In my Databricks notebook, I am getting ParseException in the last line of the code below when converting string to Date data type. The column in csv file does correctly have hiring_date in a date format.

Question: What I may be doing wrong here and how can we fix the error?

Remark: I am using python and NOT scala. I do not know scala.

from pyspark.sql.functions import *

df = spark.read.csv(".../Test/MyFile.csv", header="true", inferSchema="true")
df2 = df.withColumn("hiring_date",df["hiring_date"].cast('DateType'))

Upvotes: 0

Views: 534

Answers (1)

Phuri Chalermkiatsakul
Phuri Chalermkiatsakul

Reputation: 581

If it is the last line of your code, with reference to this doc, the code should be modified as follows:

df2 = df.withColumn("hiring_date", df.hiring_date.cast(DateType()))

It seems you put a wrong value for cast function.

The following code would work as well:

df2 = df.withColumn("hiring_date", df["hiring_date"].cast('Date'))

Upvotes: 2

Related Questions