CODEWITHSUNDEEP
scalaapache-spark
Jader Martins
Jader Martins

Reputation: 789

Scala/Spark can't match function

I'm trying to run the following command:

df = df.withColumn("DATATmp", to_date($"DATA", "yyyyMMdd"))

And getting this error:

<console>:34: error: too many arguments for method to_date: (e: org.apache.spark.sql.Column)org.apache.spark.sql.Column

How could I specify the exactly function to import? Has another way to avoid this error?

EDIT: Spark version 2.1

Upvotes: 2

Views: 876

Answers (1)

Andrey Tyukin
Andrey Tyukin

Reputation: 44908

As can be seen in the detailed scaladoc, the to_date function with two parameters has been added in 2.2.0, whereas the one-argument version existed since 1.5.

If you are working with an older Spark version, either upgrade, or don't use this function.

Upvotes: 5

Related Questions