CODEWITHSUNDEEP
scalaapache-sparktypes
mommomonthewind
mommomonthewind

Reputation: 4640

Scala Databricks: casting all bigint columns to double

I refer to this question: Cast multiples columns in a DataFrame

I have a dataframe with many columns. Some beginning columns (say 5) should not be touched because they are ID, name, etc.

From column 6 onward, I would like to cast a column if its datatype is bigint to double DataTypes.

Currently, I am using:

val df2 = df.withColumn("col_name", df.col("col_name").cast(DataTypes.DoubleType))

for each column and it is really time-consuming.

Upvotes: 0

Views: 868

Answers (1)

Charlie Flowers
Charlie Flowers

Reputation: 1380

1 - Exclude first 5 columns, find all ensuing columns with BigInt/Long type

2 - Fold over list of BigInt columns, changing them to Double

val df2 = df.schema.drop(5).collect{case c if c.dataType == DataTypes.LongType => c.name}.foldLeft(df){(acc, nxt) => acc.withColumn(nxt, acc.col(nxt).cast(DataTypes.DoubleType))}

Upvotes: 1

Related Questions