CODEWITHSUNDEEP
google-bigquery
DarkLeafyGreen
DarkLeafyGreen

Reputation: 70466

SUM aggregation with a threshold over two tables in bigquery

Following table shows hourly energy usage of devices: 

+--------------+-----------+-----------------------+
| energy_usage | device_id |  timestamp            |
+--------------+-----------+-----------------------+
| 10           | 1         |  2019-02-12T01:00:00  |
| 16           | 2         |  2019-02-12T01:00:00  |
| 26           | 1         |  2019-03-12T02:00:00  |
| 24           | 2         |  2019-03-12T02:00:00  |
+--------------+-----------+-----------------------+

I aggregate this data, such that I get the energy usage during day and night time by day and device:

+--------------+------------------+--------------------+-----------+------------+
| energy_usage | energy_usage_day | energy_usage_night | device_id |    date    |
+--------------+------------------+--------------------+-----------+------------+
| 80           | 30               | 50                 | 1         | 2019-06-02 |
| 130          | 60               | 70                 | 2         | 2019-06-03 |
+--------------+------------------+--------------------+-----------+------------+

I am only interested in energy usage above a certain threshold. Following query works for me:

WITH temp AS (
  SELECT *, SUM(usage) OVER(win) > 50 qualified,
    SUM(usage) OVER(win) - 50 rolling_sum,
    EXTRACT(HOUR FROM timestamp) BETWEEN 8 AND 19 day_hour,
    EXTRACT(MONTH FROM timestamp) month,
    FORMAT_TIMESTAMP("%Y-%m-%d", timestamp) date
  FROM `project.dataset.table`
  WINDOW win AS (PARTITION BY device_id, TIMESTAMP_TRUNC(timestamp, MONTH) ORDER BY timestamp)
), temp_with_adjustments AS (
  SELECT *, 
    IF(
      ROW_NUMBER() OVER(PARTITION BY device_id, MONTH ORDER BY timestamp) = 1, 
      rolling_sum, 
      usage
    ) AS adjusted_energy_usage
  FROM temp 
  WHERE qualified
)
SELECT ROUND(SUM(adjusted_energy_usage), 4) energy_usage,
  ROUND(SUM(IF(day_hour, adjusted_energy_usage, 0)), 4) energy_usage_day,
  ROUND(SUM(IF(NOT day_hour, adjusted_energy_usage, 0)), 4) energy_usage_night,
  device_id,
  date
FROM temp_with_adjustments
GROUP BY device_id, date

While the first table shows the energy usage, I have another table that shows the corresponding billing for the usage:

+--------------+-----------+-----------------------+
| usage_charge | device_id |  timestamp            |
+--------------+-----------+-----------------------+
| 0.2          | 1         |  2019-02-12T01:00:00  |
| 0.6          | 2         |  2019-02-12T01:00:00  |
| 0.1          | 1         |  2019-03-12T02:00:00  |
| 1.2          | 2         |  2019-03-12T02:00:00  |
+--------------+-----------+-----------------------+

I want to get insights into usage charges at day and night by device and date for devices that have an energy usage > 50. Results may look as follows:

+--------------+------------------+--------------------+--------------+------------------+--------------------+-----------+------------+
| energy_usage | energy_usage_day | energy_usage_night | usage_charge | usage_charge_day | usage_charge_night | device_id |    date    |
+--------------+------------------+--------------------+--------------+------------------+--------------------+-----------+------------+
| 80           | 30               | 50                 | 1.2          | 0.4              | 0.8                | 1         | 2019-06-02 |
| 130          | 60               | 70                 | 2.5          | 1                | 1.5                | 2         | 2019-06-03 |
+--------------+------------------+--------------------+--------------+------------------+--------------------+-----------+------------+

So my first thought was to use the exact same query for usage charges as I did for energy usage. However while a threshold of 50 works for energy usage, I cannot name a fixed threshold for usage charge, since charge calculation varies by device. So I have to get energy usage > 50 first and use the timestamps to aggregate the usage charges. Any ideas how I could accomplish this in bigquery? Is it even possible?

Upvotes: 0

Views: 98

Answers (1)

Mikhail Berlyant
Mikhail Berlyant

Reputation: 173171

Below is for BigQuery Standard SQL and is just based on applying pattern that I see in initial query - so it is quite hard for me to be 100% sure it is exactly what you need. But in any case, it for sure good start here 

#standardSQL
WITH temp AS (
  SELECT *, SUM(IF(qualified, usage_charge, 0)) OVER(win) rolling_charge
  FROM (
    SELECT *, SUM(usage) OVER(win) > 50 qualified,
      SUM(usage) OVER(win) - 50 rolling_sum,
      EXTRACT(HOUR FROM timestamp) BETWEEN 8 AND 19 day_hour,
      EXTRACT(MONTH FROM timestamp) month,
      FORMAT_TIMESTAMP("%Y-%m-%d", timestamp) date
    FROM `project.dataset.usage`
    JOIN `project.dataset.charges` USING(device_id, timestamp)
    WINDOW win AS (PARTITION BY device_id, TIMESTAMP_TRUNC(timestamp, MONTH) ORDER BY timestamp)
  )
  WINDOW win AS (PARTITION BY device_id, TIMESTAMP_TRUNC(timestamp, MONTH) ORDER BY timestamp)
), temp_with_adjustments AS (
  SELECT *, 
    IF(
      ROW_NUMBER() OVER(PARTITION BY device_id, MONTH ORDER BY timestamp) = 1, 
      rolling_sum, 
      usage
    ) AS adjusted_energy_usage
  FROM temp 
  WHERE qualified
)
SELECT ROUND(SUM(adjusted_energy_usage), 4) energy_usage,
  ROUND(SUM(IF(day_hour, adjusted_energy_usage, 0)), 4) energy_usage_day,
  ROUND(SUM(IF(NOT day_hour, adjusted_energy_usage, 0)), 4) energy_usage_night,
  ROUND(SUM(rolling_charge), 4) usage_charge,
  ROUND(SUM(IF(day_hour, rolling_charge, 0)), 4) usage_charge_day,
  ROUND(SUM(IF(NOT day_hour, rolling_charge, 0)), 4) usage_charge_night,
  device_id,
  date
FROM temp_with_adjustments
GROUP BY device_id, date

Upvotes: 1

Related Questions