CODEWITHSUNDEEP
rdplyrnamissing-data
discipulus
discipulus

Reputation: 2725

Replace the values of NA with a sum of previous value and a current value in different column

I have a dataset where I have to fill NA values using the previous value and a sum of current value in another column. Basically, my data looks like 

library(lubridate)
library(tidyverse)
library(zoo)
df <- tibble(
  Id = c(1, 1, 1, 1, 2, 2, 2, 2),
  Time = ymd(c("2012-09-01", "2012-09-02", "2012-09-03", "2012-09-04", "2012-09-01", "2012-09-02", "2012-09-03", "2012-09-04")),
  av = c(18, NA, NA, NA, 21, NA, NA, NA),
  Value = c(121, NA,NA, NA, 146, NA, NA, NA)
)

# A tibble: 8 x 4
Id      Time       av   Value
<dbl>  <date>     <dbl> <dbl>
1     2012-09-01    18   121
1     2012-09-02    NA    NA
1     2012-09-03    NA    NA
1     2012-09-04    NA    NA
2     2012-09-01    21   146
2     2012-09-02    NA    NA
2     2012-09-03    NA    NA
2     2012-09-04    NA    NA

What I want to do is: where the Value is NA, I want to replace it by sum of previous Value and current value of av. If av is NA, it can be replaced with previous value. I use na.locf function from zoo package as 

df1 <- df %>% arrange(Id, Time) %>% group_by(Id) %>% 
     mutate(av = zoo::na.locf(av))

However, filling in for Value seems to be difficult. I can do it using for loop as 

# Back up the Value column for testing
df1$Value_backup <- df1$Value

for(i in 2:nrow(df1))
{
  df1$Value[i] <- ifelse(is.na(df1$Value[i]), df1$av[i] + df1$Value[i-1], df1$Value[i])

}

This produces the result I want but for a large dataset, I believe there are better ways to do it in R. I tried complete function from dplyr but it adds two additional rows as: 

df1 <- df %>% arrange(Id, Time) %>% group_by(Id) %>% mutate(av = zoo::na.locf(av)) %>% 
  mutate(num_rows = n()) %>%
  complete(nesting(Id), Value = seq(min(Value, na.rm = TRUE), 
                                    (min(Value, na.rm = TRUE) + max(num_rows) * min(na.omit(av))), min(na.omit(av))))

The output has two extra rows; 10 instead of 8

# A tibble: 10 x 5
# Groups:   Id [2]
Id    Value Time         av    num_rows
<dbl> <dbl> <date>     < dbl>    <int>
1     121   2012-09-01    18        4
1     139   NA            NA       NA
1     157   NA            NA       NA
1     175   NA            NA       NA
1     193   NA            NA       NA
2     146   2012-09-01    21        4
2     167   NA            NA       NA
2     188   NA            NA       NA
2     209   NA            NA       NA
2     230   NA            NA       NA

Any help to do it faster without loops would be greatly appreciated.

Upvotes: 2

Views: 418

Answers (1)

G. Grothendieck
G. Grothendieck

Reputation: 270248

In the question av starts with a non-NA in each group and is followed by NAs so if this is the general pattern then this will work. Note that it is good form to close any group_by with ungroup; however, we did not do that below so that we could compare df2 with df1.

df2 <- df %>% 
  group_by(Id) %>% 
  mutate(Value_backup = Value,
         av = first(av), 
         Value = first(Value) + cumsum(av) - av)

identical(df1, df2)
## [1] TRUE

Note

For reproducibility first run this (taken from question except we only load needed packages):

library(dplyr)
library(tibble)
library(lubridate)

df <- tibble(
  Id = c(1, 1, 1, 1, 2, 2, 2, 2),
  Time = ymd(c("2012-09-01", "2012-09-02", "2012-09-03", "2012-09-04", "
    2012-09-01", "2012-09-02", "2012-09-03", "2012-09-04")),
  av = c(18, NA, NA, NA, 21, NA, NA, NA),
  Value = c(121, NA,NA, NA, 146, NA, NA, NA)
)

df1 <- df %>% arrange(Id, Time) %>% group_by(Id) %>% 
     mutate(av = zoo::na.locf(av))  
df1$Value_backup <- df1$Value
for(i in 2:nrow(df1))
{
  df1$Value[i] <- ifelse(is.na(df1$Value[i]), df1$av[i] + df1$Value[i-1], df1$Value[i])

}

Upvotes: 2

Related Questions