CODEWITHSUNDEEP
javaapache-spark
Bhagesh Arora
Bhagesh Arora

Reputation: 547

How to change column types in Spark SQL's(In java) DataFrame?

I am trying to convert a column data type from long to int in spark sql using java, I have seen some of example in scala and trying out but, Its not wotking

df.withColumn("is_open",  toInt("is_open"));

where do I need to change. thanks in advance

Upvotes: 2

Views: 4849

Answers (2)

Volodymyr Zubariev
Volodymyr Zubariev

Reputation: 443

df = df.withColumn("is_open", df.col("is_open").cast("int"));

Please be aware, this cast is applying after the data is computed on previous step. If the previous step is select or something similar, it will compute into original type first, and then convert to new type on the next step. This will not solve an issue with selecting into original type.

Upvotes: 2

Constantine
Constantine

Reputation: 1406

You can make use of the cast function.

scala> val df = spark.range(10)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> df.withColumn("new_col", col("id").cast("string"))
res4: org.apache.spark.sql.DataFrame = [id: bigint, new_col: string]

Upvotes: 3

Related Questions