CODEWITHSUNDEEP
palantir-foundry
ivanhoe1982
ivanhoe1982

Reputation: 530

date_format failing in the workbook environment

In a Palantir Foundry Code Workbook Spark SQL node (or in the Spark console in SQL mode), this works:

SELECT date_format('2021-01-01',"yyyy-MM")
2021-01

But executing a pattern asking for a quarter doesn't:

SELECT date_format('2021-01-01',"yyyy-Q")
java.lang.IllegalArgumentException: Illegal pattern character 'Q'

This is a legal pattern in spark 3.2.0

https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html

Datetime patterns - Spark 3.2.0 Documentation

Is there some environment configuration that changes this behavior? There is a switch set spark.sql.legacy.timeParserPolicy=LEGACY that perhaps could have to do with it. If this is the culprit - how to change this in the workbook environment?

Upvotes: 2

Views: 278

Answers (1)

vanhooser
vanhooser

Reputation: 1747

You've found a bug!

In the stack trace when I run this, I get:

Caused by: java.lang.IllegalArgumentException: Illegal pattern character 'Q'
    at java.base/java.text.SimpleDateFormat.compile(SimpleDateFormat.java:845)
    at java.base/java.text.SimpleDateFormat.initialize(SimpleDateFormat.java:653)
    at java.base/java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:624)
    at org.apache.spark.sql.catalyst.util.LegacySimpleTimestampFormatter.sdf$lzycompute(TimestampFormatter.scala:229)
...

In the SimpleDateFormat.java spec, Quarter is not allowed.

Spark has told you it allows a format string that the underlying Java implementation does not. I'd recommend filing a bug on the Apache Spark Jira

Upvotes: 1

Related Questions