CODEWITHSUNDEEP
sqlscalaapache-sparkwindow-functions
M.S.Visser
M.S.Visser

Reputation: 81

spark scala ignore nulls in windowing clause

In spark SQl, you can write

SELECT title, rn, 
       lead(rn, 1) IGNORE NULLS over(order by rn) as next_rn
FROM   my_table
;

How would you add the IGNORE NULLS part in the equivalent Scala code?

val my_df = my_table
  .withColumn("next_rn", lead($"rn", 1).over(Window.orderBy("rn"))

Upvotes: 0

Views: 56

Answers (1)

M.S.Visser
M.S.Visser

Reputation: 81

A colleague of mine pointed me to Scala documentation over here. The working Scala code then becomes

.withColumn("next_tbl_rn", lead($"tbl_rn", 1, null, true).over(Window.orderBy($"rn")))

in which the fourth argument does the trick.

Upvotes: 2

Related Questions