CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Dev
Dev

Reputation: 13753

How to covert dataframe datatypes to String?

I have a hive Table having Date and Timestamp datatypes. I am creating DataFrame using below java code:

SparkConf conf = new SparkConf(true).setMaster("yarn-cluster").setAppName("SAMPLE_APP");
SparkContext sc = new SparkContext(conf);
HiveContext hc = new HiveContext(sc);
DataFrame df = hc.table("testdb.tbl1");

Dataframe schema:

 df.printSchema
root
 |-- c_date: date (nullable = true)
 |-- c_timestamp: timestamp (nullable = true)

I want to covert these columns to String. How can I achieve this?

I need this because of issue : Spark csv data validation failed for date and timestamp data types of Hive

Upvotes: 3

Views: 113

Answers (2)

karthik manchala
karthik manchala

Reputation: 13640

You can do the following:

df.withColumn("c_date", df.col("c_date").cast(StringType))

Upvotes: 2

ROOT
ROOT

Reputation: 1775

In scala, we generally cast datatypes like this:

df.select($"date".cast(StringType).as("new_date"))

Upvotes: 2

Related Questions