CODEWITHSUNDEEP
apache-sparkcastingpysparkspark-streaming
shaikh
shaikh

Reputation: 592

Spark Streaming 2.3.1 Type Casting: String to Timestamp

I am using apsche spark streaming 2.3.1, where I am receiving a stream containing a timestamp values (13:09:05.761237147) of the format "HH:mm:ss.xxxxxxxxx" as string.

I am in need to cast this string to timestamp data type.

spark = SparkSession \
    .builder \
    .appName("abc") \
    .getOrCreate()

schema = StructType().add("timestamp", "string").add("object", "string").add("score", "double")

lines = spark \
    .readStream \
    .option("sep", ",") \
    .schema(schema) \
    .csv("/path/to/folder/")

Any suggestion how to convert "timestamp" to timestamp data type?

Upvotes: 2

Views: 740

Answers (1)

Arati Nagmal
Arati Nagmal

Reputation: 60

As per the description provided in source code of TimestampType and DateTimeUtils classes, they support timestamps till microseconds precision only.

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/TimestampType.scala

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

Upvotes: 1

Related Questions