CODEWITHSUNDEEP
scaladataframeapache-sparkapache-spark-sql
Apoorv Shrivastava
Apoorv Shrivastava

Reputation: 73

How to reinfer datatype of a Spark Dataframe column after updating values in the column in Scala

This is a sample of my original dataframe df

+----+
| mix|
+----+
|   1|
|   2|
| cap|
|   3|
|  53|
|  56|
|  98|
|  90|
+----+

The current datatype of the column is StringType After replacing the value cap with 0 , there could be two cases

How to infer datatype again so i can know if the column is purely numerical after replacement or not.And if Numerical , what is the exact datatype viz Integer, Float ,Double

df.withColumn("mix",when(col("mix") === "cap",0).otherwise(col("mix")))

Upvotes: 0

Views: 231

Answers (1)

z_1_p
z_1_p

Reputation: 409

val mix = Seq("1","2","cap","4").toDF("mix")    
mix.printSchema()
root
|-- mix: string (nullable = true)

val after = mix.withColumn("mix",when(col("mix") === "cap",0).otherwise(col("mix")).cast(IntegerType))
after.printSchema()
root
|-- mix: integer (nullable = true)

Let me know, if this helps you.

Upvotes: 1

Related Questions