CODEWITHSUNDEEP
rxts
Tatha
Tatha

Reputation: 137

Modify usual hpfilter function to ignore na's

I'm a new R user, trying to learn quickly, but I couldn't crack this myself. I work mostly with economic time series – hence, try to maintain my dataset in xts multi-column format, e.g.:

> head(USDATAq)
         tq   ngdp    rgdp  profit
1947 Q1   0  237.2  1770.7    20.7
1947 Q2   1  240.4  1768.0    23.9
1947 Q3   2  244.5  1766.5    23.8
1947 Q4   3  254.3  1793.3    25.5
1948 Q1   4  260.3  1821.8    29.4
1948 Q2   5  267.3  1855.3    31.2

I apply the hpfilter function for filtering. Elsewhere on this site, I found this implementation which uses the coredata function to apply hpfilter to xts objects:

hpfilter <- function(x, lambda=2){
  eye <- diag(length(x))
  dcrossprod <- crossprod(diff(eye, lag=1, d=2))
  coredata(x) <- solve(eye + lambda * dcrossprod, coredata(x))
  return(x)
}

My question is:

How can I modify the function so that it will work with variables having NA observations (at present, it calculates NA for the entire date range if there is any NA)?

I can pass on the dataset as na.omit(USDATAq), which works, but this curtails all variables in dataset to the minimum observations. But, different variables are available until different dates, followed by NA's. I would like to eventually apply the function to every column of the dataset in a loop or mapply, so that the function returns each filtered series using all available observations of that series.

Upvotes: 6

Views: 1095

Answers (3)

user2750268
user2750268

Reputation: 23

With zoo objects, it is marginally cleaner using attributes() rather than coredata() then you can merge straight back into the zoo object. (I haven't tried this for xts objects):

hpfilter <- function(x,lambda=1600){
    y<-na.omit(x)
    eye <- diag(length(y))
    result <- solve(eye+lambda*crossprod(diff(eye,lag=1,d=2)),y)
    attributes(result) <- attributes(y)
    return(result)
}

Upvotes: 0

Tatha
Tatha

Reputation: 137

Thanks @ran2. I worked on your suggestion and managed to solve the problem – but, in a rather inelegant way. First of all, I could not get any of the 'apply family' functions to work correctly on an xts object, maintaining its structure. Plain apply with the apply(x, MARGIN=2,..) for column-wise application showed promise, but stalled at the 'coredata' statement. lapply etc yielded mangled lists.

I then went to the for loop. But because the x<-na.omit(x) changes the length of the variable, it cannot replace the original within loop. 

> for(i in 1:ncol(USDATAq)) {
+ USDATAq[,i]<-hpfilter(USDATAq[,i])
+ }

Error in NextMethod(.Generic) : number of items to replace is not a multiple of replacement length

So, I had to add unseemly code to hpfilter to ‘merge’ the result back to the original (with NA) and then return the variable. This merging matches the 2 variables by date (hence, length) filling NA’s into the result. Then, this result can replace the original in a loop. In conclusion, I had to modify hpfilter to:

hpfilter <- function(x,lambda=2){
y<-na.omit(x)
eye <- diag(length(y))
coredata(y) <- solve(eye + lambda * crossprod(diff(eye, lag=1, d=2)), coredata(y))
xy<-merge(x,y) 
return(xy[,2])
}

and then use the loop above, to finally get error free results. My knowledge of R is so rudimentary, though, that there probably are easier ways to do this. But, at least, I can proceed now. Thanks to all for pointing me in the right direction. I'd still welcome further corrections to my code above.

Upvotes: 3

Matt Bannert
Matt Bannert

Reputation: 28264

I think you are on the right track. Why not just add na.omit inside this function? Just before creating the eye matrix? x<-na.omit(x). Then you all you have to do is pass univariate series to it instead of whole data.frames. In other words: Leave the function as it is, add na.omit and combine it with lapply (or whatever form of the apply family (sapply,tapply,lapply) suits you best.

Upvotes: 2

Related Questions