CODEWITHSUNDEEP
pysparkapache-spark-sql
rupak das
rupak das

Reputation: 141

Padding in a Pyspark Dataframe

I have a Pyspark dataframe(Original Dataframe) having below data(all columns have string datatype):

  id           Value
   1             103
   2             1504
   3              1

I need to create a new modified dataframe with padding in value column, so that length of this column should be 4 characters. If length is less than 4 characters, then add 0's in data as shown below:

  id             Value
   1             0103
   2             1504
   3             0001

Can someone help me out? How can i achieve it using Pyspark dataframe? Any help will be appreciated.

Upvotes: 12

Views: 33241

Answers (2)

ucsky
ucsky

Reputation: 442

Using PySpark lpad function in conjunction with withColumn:

import pyspark.sql.functions as F
dfNew = dfOrigin.withColumn('Value', F.lpad(dfOrigin['Value'], 4, '0'))

Upvotes: 9

Suresh
Suresh

Reputation: 5880

You can use lpad from functions module,

from pyspark.sql.functions import lpad
>>> df.select('id',lpad(df['value'],4,'0').alias('value')).show()
+---+-----+
| id|value|
+---+-----+
|  1| 0103|
|  2| 1504|
|  3| 0001|
+---+-----+

Upvotes: 26

Related Questions