CODEWITHSUNDEEP
rforecasting
FBE
FBE

Reputation: 661

Forecast accuracy: no MASE with two vectors as arguments

I'm using the accuracy function from the forecast package, to calculate accuracy measures. I'm using it to calculate measures for fitted time series models, such as ARIMA or exponential smoothing. As I'm testing different model types on different dimensions and aggregation levels, I'm using the MASE, mean absolute scaled error, introduced by Hyndman et al (2006, "Another look at measures of forecast accuracy"), to compare different models on different levels.

Now I'm also comparing models with forecast history. As I only have the forecast values and not the models, I tried to use the accuracy function. In the function description is mentioned that it is also allowed provide two vector arguments, one with forecast values and one with actuals, to calculate the measures (instead of a fitted model):

f: An object of class "forecast", or a numerical vector containing forecasts. It will also work with Arima, ets and lm objects if x is omitted – in which case in-sample accuracy measures are returned.

x: An optional numerical vector containing actual values of the same length as object.

But I was suprised by the fact that all measures are returned, expect the MASE. So I was wondering if somebody knows what the reason is for that? Why is the MASE not returned, while using two vectors as arguments in the accuracy function?

Upvotes: 8

Views: 6524

Answers (3)

Shubham Saini
Shubham Saini

Reputation: 738

The paper on MASE clearly explains how to find it (even for non time-series data)

computeMASE <- function(forecast,train,test,period){

  # forecast - forecasted values
  # train - data used for forecasting .. used to find scaling factor
  # test - actual data used for finding MASE.. same length as forecast
  # period - in case of seasonal data.. if not, use 1

  forecast <- as.vector(forecast)
  train <- as.vector(train)
  test <- as.vector(test)

  n <- length(train)
  scalingFactor <- sum(abs(train[(period+1):n] - train[1:(n-period)])) / (n-period)

  et <- abs(test-forecast)
  qt <- et/scalingFactor
  meanMASE <- mean(qt)
  return(meanMASE)
}

Upvotes: 13

Rob Hyndman
Rob Hyndman

Reputation: 31810

The MASE requires the historical data to compute the scaling factor. It is not computed from the future data as in the answer by @FBE. So if you don't pass the historical data to accuracy(), the MASE cannot be computed. For example,

> library(forecast)
> fcast <- snaive(window(USAccDeaths,end=1977.99))
> accuracy(fcast$mean,USAccDeaths)
         ME        RMSE         MAE         MPE        MAPE        ACF1 
225.1666667 341.1639391 259.5000000   2.4692164   2.8505546   0.3086626 
  Theil's U 
  0.4474491

But if you pass the whole fcast object (which includes the historical data), you get

> accuracy(fcast,USAccDeaths)
         ME        RMSE         MAE         MPE        MAPE        MASE 
225.1666667 341.1639391 259.5000000   2.4692164   2.8505546   0.5387310 
       ACF1   Theil's U 
  0.3086626   0.4474491

Upvotes: 15

FBE
FBE

Reputation: 661

To help myself a little bit, I created a function to calculate the MASE, as described by Hyndman et al in "Another look at measures of forecast accuracy" (2006).

calculateMASE <- function(f,y) { # f = vector with forecasts, y = vector with actuals
    if(length(f)!=length(y)){ stop("Vector length is not equal") }
    n <- length(f)
    return(mean(abs((y - f) / ((1/(n-1)) * sum(abs(y[2:n]-y[1:n-1]))))))
}

For reference, see:

Upvotes: 0

Related Questions