CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
user9272398
user9272398

Reputation:

how to make lower case and delete the original column in pyspark?

I have a very big data test. contain text. I wanted to make it all lower case I did this:

df1=df.select("*", lower(col('name')))

but it makes a new column called lower(name). I don't want to keep the previous column. So I deleted by this:

df2=df1.drop(*'title_split')

But it takes a lot to delete it. how I can make it faster? can I make it lower case and don't keep the previous one?

Upvotes: 0

Views: 176

Answers (1)

mck
mck

Reputation: 42352

You can use withColumn to replace the old column:

df1 = df.withColumn('name2', lower(col('name'))).drop('name')

Upvotes: 1

Related Questions