CODEWITHSUNDEEP
rfiltersignalstime-seriesna
Yu Deng
Yu Deng

Reputation: 1061

R filter() dealing with NAs

I am trying to implement Chebyshev filter to smooth a time series but, unfortunately, there are NAs in the data series.

For example,

t <- seq(0, 1, len = 100)                     
x <- c(sin(2*pi*t*2.3) + 0.25*rnorm(length(t)),NA, cos(2*pi*t*2.3) + 0.25*rnorm(length(t)))

I am using Chebyshev filter: cf1 = cheby1(5, 3, 1/44, type = "low")

I am trying to filter the time series exclude NAs, but not mess up the orders/position. So, I have already tried na.rm=T, but it seems there's no such argument. Then 

z <- filter(cf1, x)   # apply filter

Thank you guys.

Upvotes: 7

Views: 8936

Answers (4)

H. Radtke
H. Radtke

Reputation: 1

Here is a function you can use to filter a signal with NAs in it. The NAs are ignored rather than replaced by zero.

You can then specify a maximum percentage of weight which the NAs may take at any point of the filtered signal. If there are too many NAs (and too few actual data) at a specific point, the filtered signal itself will be set to NA.

# This function applies a filter to a time series with potentially missing data 

filter_with_NA <- function(x,
                           window_length=12,                            # will be applied centrally
                           myfilter=rep(1/window_length,window_length), # a boxcar filter by default
                           max_percentage_NA=25)                        # which percentage of weight created by NA should not be exceeded
{
  # make the signal longer at both sides
  signal <- c(rep(NA,window_length),x,rep(NA,window_length))
  # see where data are present and not NA
  present <- is.finite(signal)

  # replace the NA values by zero
  signal[!is.finite(signal)] <- 0
  # apply the filter
  filtered_signal <- as.numeric(filter(signal,myfilter, sides=2))

  # find out which percentage of the filtered signal was created by non-NA values
  # this is easy because the filter is linear
  original_weight <- as.numeric(filter(present,myfilter, sides=2))
  # where this is lower than one, the signal is now artificially smaller 
  # because we added zeros - compensate that
  filtered_signal <- filtered_signal / original_weight
  # but where there are too few values present, discard the signal
  filtered_signal[100*(1-original_weight) > max_percentage_NA] <- NA

  # cut away the padding to left and right which we previously inserted
  filtered_signal <- filtered_signal[((window_length+1):(window_length+length(x)))]
  return(filtered_signal)
}

Upvotes: 0

chandler
chandler

Reputation: 846

You can remove the NAs beforehand using the compelete.cases function. You also might consider imputing the missing data. Check out the mtsdi or Amelia II packages.

EDIT:

Here's a solution with Rcpp. This might be helpful is speed is important:

require(inline)
require(Rcpp)
t <- seq(0, 1, len = 100)
set.seed(7337)
x <- c(sin(2*pi*t*2.3) + 0.25*rnorm(length(t)),NA, cos(2*pi*t*2.3) + 0.25*rnorm(length(t)))
NAs <- x
x2 <- x[!is.na(x)]
#do something to x2
src <- '
Rcpp::NumericVector vecX(vx);
Rcpp::NumericVector vecNA(vNA);
int j = 0;   //counter for vx
for (int i=0;i<vecNA.size();i++) {
  if (!(R_IsNA(vecNA[i]))) {
    //replace and update j
    vecNA[i] = vecX[j];
    j++;
  }
 }
return Rcpp::wrap(vecNA);
'
fun <- cxxfunction(signature(vx="numeric",
                             vNA="numeric"),
                   src,plugin="Rcpp")
if (identical(x,fun(x2,NAs)))
    print("worked")
# [1] "worked"

Upvotes: 1

lockedoff
lockedoff

Reputation: 513

I don't know if ts objects can have missing values, but if you just want to re-insert the NA values, you can use ?insert from R.utils. There might be a better way to do this.

install.packages(c('R.utils', 'signal'))
require(R.utils)
require(signal)
t <- seq(0, 1, len = 100)                     
set.seed(7337)
x <- c(sin(2*pi*t*2.3) + 0.25*rnorm(length(t)), NA, NA, cos(2*pi*t*2.3) + 0.25*rnorm(length(t)), NA)
cf1 = cheby1(5, 3, 1/44, type = "low")
xex <- na.omit(x)
z <- filter(cf1, xex)   # apply
z <- as.numeric(z)
for (m in attributes(xex)$na.action) {
  z <- insert(z, ats = m, values = NA)
}
all.equal(is.na(z), is.na(x))
?insert

Upvotes: 1

Eli Sander
Eli Sander

Reputation: 1248

Try using x <- x[!is.na(x)] to remove the NAs, then run the filter.

Upvotes: 2

Related Questions