CODEWITHSUNDEEP
sqlapache-sparkpysparkapache-spark-sqlwindow-functions
M80
M80

Reputation: 994

How to define WINDOWING function in Spark SQL query to avoid repetitive code

I have a query which has many lead and lag, due to which the partition by code is repeated.

If I use Scala code I can define the window spec and reuse it , so is there a way I can reuse the partition code in Spark SQL.

Objective is to avoid the repetition of "over ( partition by sessionId, deviceId order by entry_datetime ) "

SELECT * ,
lag( channel,1,null ) over ( partition by sessionId, deviceId order by entry_datetime ) as prev_chnl,
lead( channel,1,null ) over ( partition by sessionId, deviceId order by entry_datetime ) as next_chnl,
lag( channel-source,1,null ) over ( partition by sessionId, deviceId order by entry_datetime ) as prev_chnl_source,
lead( channel-source,1,null ) over ( partition by sessionId, deviceId order by entry_datetime ) as next_chnl_source,
FROM RAW_VIEW

RAW_VIEW

+------------+-----------+---------------------+---------+-----------------+
|sessionId   |deviceId   |entry_datetime       |channel  |channel-source   |
+------------+-----------+---------------------+---------+-----------------+
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 15:00:00.0|001      |Internet         |
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 16:00:00.0|002      |Cable            |
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 17:00:00.0|003      |Satellite        |
+------------+-----------+---------------------+---------+-----------------+

FINAL VIEW

+------------+-----------+---------------------+---------+-----------------+---------+---------+-----------------+-----------------+
|sessionId   |deviceId   |entry_datetime       |channel  |channel-source   |prev_chnl|next_chnl|prev_chnl_source |next_chnl_source |
+------------+-----------+---------------------+---------+-----------------+---------+---------+-----------------+-----------------+
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 15:00:00.0|001      |Internet         |null     |002      |null             |Cable            |
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 15:01:00.0|002      |Cable            |001      |003      |Internet         |Satellite        |
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 15:02:00.0|003      |Satellite        |002      |null     |Cable            |null             |
+------------+-----------+---------------------+---------+-----------------+---------+---------+-----------------+-----------------+

Upvotes: 4

Views: 731

Answers (2)

Alper t. Turker
Alper t. Turker

Reputation: 35249

You should be able to define named window and reference it in the query:

SELECT * ,
  lag(channel, 1) OVER w AS prev_chnl,
  lead(channel, 1) OVER w AS next_chnl,
  lag(channel-source, 1) OVER w AS prev_chnl_source,
  lead(channel-source, 1) OVER w AS next_chnl_source,
FROM raw_view
WINDOW w AS (PARTITION BY sessionId, deviceId ORDER BY entry_datetime)

but it looks like this functionality is currently broken.

Upvotes: 6

pault
pault

Reputation: 43544

If you want to do this in spark-sql, one way is to to add row_number() to your table over your ordered partitions. Then create a lag and lead version of this table by subtracting / adding 1 to the row_number. Finally do a LEFT JOIN of the current table with the previous and next versions and select the appropriate columns.

For example, try the following:

   SELECT curr.*,
          prev.channel AS prev_chnl,
          next.channel AS next_chnl,
          prev.channel_source AS prev_chnl_source,
          next.channel_source AS next_chnl_source
     FROM (SELECT *,
                 ROW_NUMBER() OVER (partition by sessionId, 
                                                  deviceId 
                                        order by entry_datetime) AS row_num 
           FROM RAW_VIEW
     ) curr
LEFT JOIN (SELECT *,
                  ROW_NUMBER() OVER (partition by sessionId,
                                                  deviceId
                                         order by entry_datetime) + 1 AS row_num
           FROM RAW_VIEW
     ) prev ON (curr.row_num = prev.row_num)
LEFT JOIN (SELECT *,
                  ROW_NUMBER() OVER (partition by sessionId,
                                                  deviceId
                                         order by entry_datetime) - 1 AS row_num
           FROM RAW_VIEW
     ) next ON (next.row_num = curr.row_num)
 ORDER BY entry_datetime

Which results in:

+------------+-----------+---------------------+-------+--------------+-------+---------+---------+----------------+----------------+
|sessionId   |deviceId   |entry_datetime       |channel|channel_source|row_num|prev_chnl|next_chnl|prev_chnl_source|next_chnl_source|
+------------+-----------+---------------------+-------+--------------+-------+---------+---------+----------------+----------------+
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 15:00:00.0|001    |Internet      |1      |null     |002      |null            |Cable           |
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 16:00:00.0|002    |Cable         |2      |001      |003      |Internet        |Satellite       |
|SESSION-ID-1|DEVICE-ID-1|2018-04-09 17:00:00.0|003    |Satellite     |3      |002      |null     |Cable           |null            |
+------------+-----------+---------------------+-------+--------------+-------+---------+---------+----------------+----------------+

Upvotes: 0

Related Questions