CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
Ana
Ana

Reputation: 103

Pyspark: merge conditions in a when clause

I would appreciate some help to merge lines that get TYPE2 and type3.

I would need just one line to get TYPE2 and one line to get TYPE3:

df2 = df1.withColumn("DEVICETYPE",
                        when(length(col("deviceid")) ==21,lit("type1"))
                        .when(col("DEVICESERIALNO").like("1200125%"),lit("type2"))
                        .when(length(col("deviceid")) ==17,lit("type2"))
                        .when(col("DEVICESERIALNO").isNull(),lit("type3"))
                        .when(length(col("deviceid")) ==17,lit("type3"))
                        .when(length(col("deviceid")) ==7,lit("type4"))
                        .when(length(col("deviceid")) ==5,lit("type4"))
                    )

Thanks much in advance!!

Upvotes: 0

Views: 138

Answers (2)

Marioanzas
Marioanzas

Reputation: 1945

I can see that you have multiple conditions for type2, type3 and type4. Hence, I would suggest the following modified piece of code:

df2 = df1.withColumn("DEVICETYPE",
                        when(length(col("deviceid")) ==21,lit("type1"))
                        .when((col("DEVICESERIALNO").like("1200125%")) & (length(col("deviceid")) ==17),lit("type2"))
                        .when((col("DEVICESERIALNO").isNull()) & (length(col("deviceid")) ==17),lit("type3"))
                        .when((length(col("deviceid")) ==7) & (length(col("deviceid")) ==5),lit("type4"))
                    )

Upvotes: 2

mck
mck

Reputation: 42422

You can use &:

df2 = df1.withColumn("DEVICETYPE",
                        when(length(col("deviceid")) ==21,lit("type1"))
                        .when((col("DEVICESERIALNO").like("1200125%")) & (length(col("deviceid")) ==17),lit("type2"))
                        .when((col("DEVICESERIALNO").isNull()) & (length(col("deviceid")) ==17),lit("type3"))
                        .when(length(col("deviceid")) ==7,lit("type4"))
                        .when(length(col("deviceid")) ==5,lit("type4"))
                    )

Upvotes: 1

Related Questions