CODEWITHSUNDEEP
javaapache-spark
mithra
mithra

Reputation: 1141

SparkSql not supporting Date Format

I tried to use date file with sparkSql but its not working

Like for in example javaSparkSqlExample

I tried to add datecoloumn dob

In the Person class i added

setter and getter for dob as Date

when tried to execute 

SELECT dob,name,age,count(*) as totalCount FROM Person WHERE dob >= '1995-01-01' AND age <= '2014-02-01';

Also tried using between also in the query instead of <= & >= as well

/Volumes/Official/spark-1.0.2-bin-hadoop2$: bin/spark-submit --class "SimpleApp" --master local[4] try/simple-project/target/simple-project-1.0.jar 
Spark assembly has been built with Hive, including Datanucleus jars on classpath
2014-08-21 11:42:47.360 java[955:1903] Unable to load realm mapping info from SCDynamicStore
=== Data source: RDD ===
Exception in thread "main" scala.MatchError: class java.util.Date (of class java.lang.Class)

Upvotes: 3

Views: 4918

Answers (1)

emecas
emecas

Reputation: 1586

It's still in pending, instead Date you can use Timestamp in Person class SPARK-2552

Spark SQL currently supports Timestamp, but not Date.

We are going tohave to wait for a while until the 1.2.0 version.

Details:

  • Type: Improvement Improvement
  • Status: Open
  • Priority: Minor
  • Resolution: Unresolved
  • Affects Version/s: 1.0.1
  • Fix Version/s: None
  • Component/s: SQL
  • Target Version/s: 1.2.0

Upvotes: 3

Related Questions