CODEWITHSUNDEEP
rdatedplyrtidyverse
eliasmaxil
eliasmaxil

Reputation: 558

Get the max date based on multiple columns of an R dplyr / tidyverse dataframe

From a csv file that looks like this:

Date Timestamp Units Name Condition Obj Param Attrib1 Atrrib2 Result
2019-07-31 2019-08-01 01:16:09 m3 n01 a1 o1 Nap TP IN 34937
2019-07-31 2019-08-01 01:16:10 m3 n01 a2 o2 Nap TP OUT 36673.09
2019-11-06 2019-11-18 20:21:06 mg/l n01 a3 o3 NO3 TP OUT 1
2019-11-06 2019-11-18 20:21:06 mg/l n01 z5 o4 BOD IO IN 220
2019-11-06 2019-11-18 20:21:06 mg/l n01 z5 o4 BOD TP IN 220
2019-11-06 2019-11-18 20:21:06 mg/l n01 z6 o1 NO2 TP OUT 0.31
2019-11-06 2019-11-18 20:21:13 mg/l n01 a11 o4 Ntot IO IN 47
2019-11-06 2019-11-18 20:21:13 mg/l n01 a11 o4 Ntot TP IN 47
2021-01-06 2021-01-07 02:15:06 m3 n01 a1 o1 Nap TP IN 17909
2021-01-06 2021-01-07 02:15:07 m3 n01 a2 o2 Nap TP OUT 19216.19

I want to remove the rows with the last (or max) Timestamp per value in column Date and column Condition.
The resulting table should not have the duplicated timestamps "2019-11-18 20:21:06" and "2019-11-18 20:21:13" (Which Condition and Result values were [z5, a11] and [220, 47] respectively).

Date Timestamp Units Name Condition Obj Param Attrib1 Atrrib2 Result
2019-07-31 2019-08-01 01:16:09 m3 n01 a1 o1 Nap TP IN 34937
2019-07-31 2019-08-01 01:16:10 m3 n01 a2 o2 Nap TP OUT 36673.09
2019-11-06 2019-11-18 20:21:06 mg/l n01 a3 o3 NO3 TP OUT 1
2019-11-06 2019-11-18 20:21:06 mg/l n01 z5 o4 BOD IO IN 220
2019-11-06 2019-11-18 20:21:06 mg/l n01 z6 o1 NO2 TP OUT 0.31
2019-11-06 2019-11-18 20:21:13 mg/l n01 a11 o4 Ntot IO IN 47
2021-01-06 2021-01-07 02:15:06 m3 n01 a1 o1 Nap TP IN 17909
2021-01-06 2021-01-07 02:15:07 m3 n01 a2 o2 Nap TP OUT 19216.19

I found two links (1 and 2) to come up with the following R script

library(tidyverse)
# Group per Date and Condition and filter max Timestamp
df <- read.csv("./Example.csv") %>%
    mutate(Date = as.POSIXct(Date, format = "%Y-%m-%d")) %>%
    mutate(Timestamp = as.POSIXct(Timestamp, format = "%Y-%m-%d %H:%M:%S")) %>%
    group_by(Date, Condition) %>%
    filter(Timestamp == max(Timestamp)) %>%
    distinct()
write_csv(df, file = "./ExampleResult.csv")

But I cannot get the wished results.
What is wrong with the approach? Is there any other easier way?
Thank you!

Upvotes: 1

Views: 1510

Answers (2)

Dan Adams
Dan Adams

Reputation: 5204

You have multiple values at max(Timestamp). To resolve this I'd suggest to use dplyr::slice_max and setting with_ties = FALSE.

Here's some code to get what you're after.

df %>% 
  mutate(Date = as.POSIXct(Date, format = "%Y-%m-%d")) %>%
  mutate(Timestamp = as.POSIXct(Timestamp, format = "%Y-%m-%d %H:%M:%S")) %>%
  group_by(Date, Condition) %>%
  slice_max(order_by = Timestamp, n = 1, with_ties = FALSE)

But depending on your application, you may want to be explicit about how to resolve those ties by supplying additional variables to the order_by argument.

Upvotes: 3

Ronak Shah
Ronak Shah

Reputation: 388962

Try using the following :

library(dplyr)

read.csv("./Example.csv") %>%
#df %>%
  mutate(Date = as.Date(Date), 
        Timestamp = as.POSIXct(Timestamp, format = "%Y-%m-%d %H:%M:%S")) %>%
  distinct(Date, Condition, Result, .keep_all = TRUE) -> result

result

#        Date           Timestamp Units Name Condition Obj Param Attrib1 Atrrib2   Result
#1 2019-07-31 2019-08-01 01:16:09    m3  n01        a1  o1   Nap      TP      IN 34937.00
#2 2019-07-31 2019-08-01 01:16:10    m3  n01        a2  o2   Nap      TP     OUT 36673.09
#3 2019-11-06 2019-11-18 20:21:06  mg/l  n01        a3  o3   NO3      TP     OUT     1.00
#4 2019-11-06 2019-11-18 20:21:06  mg/l  n01        z5  o4   BOD      IO      IN   220.00
#5 2019-11-06 2019-11-18 20:21:06  mg/l  n01        z6  o1   NO2      TP     OUT     0.31
#6 2019-11-06 2019-11-18 20:21:13  mg/l  n01       a11  o4  Ntot      IO      IN    47.00
#7 2021-01-06 2021-01-07 02:15:06    m3  n01        a1  o1   Nap      TP      IN 17909.00
#8 2021-01-06 2021-01-07 02:15:07    m3  n01        a2  o2   Nap      TP     OUT 19216.19

data

df <- structure(list(Date = c("2019-07-31", "2019-07-31", "2019-11-06", 
"2019-11-06", "2019-11-06", "2019-11-06", "2019-11-06", "2019-11-06", 
"2021-01-06", "2021-01-06"), Timestamp = c("2019-08-01 01:16:09", 
"2019-08-01 01:16:10", "2019-11-18 20:21:06", "2019-11-18 20:21:06", 
"2019-11-18 20:21:06", "2019-11-18 20:21:06", "2019-11-18 20:21:13", 
"2019-11-18 20:21:13", "2021-01-07 02:15:06", "2021-01-07 02:15:07"
), Units = c("m3", "m3", "mg/l", "mg/l", "mg/l", "mg/l", "mg/l", 
"mg/l", "m3", "m3"), Name = c("n01", "n01", "n01", "n01", "n01", 
"n01", "n01", "n01", "n01", "n01"), Condition = c("a1", "a2", 
"a3", "z5", "z5", "z6", "a11", "a11", "a1", "a2"), Obj = c("o1", 
"o2", "o3", "o4", "o4", "o1", "o4", "o4", "o1", "o2"), Param = c("Nap", 
"Nap", "NO3", "BOD", "BOD", "NO2", "Ntot", "Ntot", "Nap", "Nap"
), Attrib1 = c("TP", "TP", "TP", "IO", "TP", "TP", "IO", "TP", 
"TP", "TP"), Atrrib2 = c("IN", "OUT", "OUT", "IN", "IN", "OUT", 
"IN", "IN", "IN", "OUT"), Result = c(34937, 36673.09, 1, 220, 
220, 0.31, 47, 47, 17909, 19216.19)),class = "data.frame",row.names = c(NA,-10L))

Upvotes: 2

Related Questions