In this article, we discuss 3 ways to calculate the Relative Absolute Error (RAE) in R.
The Relative Absolute Error is a method to measure the performance of a predictive model, such as regression. It’s a simple metric that gives a first indication of how well a model works.
But, how do you calculate the Relative Absolute Error?
In R, you can calculate the Relative Absolute Error by either writing your own code or by using the RAE() function from the Metrics packages. In both cases, you only need to provide the realized and the predicted values and R returns the RAE. Additionally, you can create a user-defined function to make your code more robust.
In this article, we discuss 3 methods to find the RAE. Therefore, we use examples and provide R code that you can use in your own project. But first, we take a look at the definition of the Relative Absolute Error and its interpretation.
The Relative Absolute Error
Definition & Formula
The Relative Absolute Error is a relative measure that compares the performance of a predictive model with the performance of a simple model.
The performance of the predictive model is defined as the total absolute difference between the realized and predicted values (i.e., the error). Whereas the performance of the simple model is the total absolute difference between the realized value and the average of all realized values.
In other words, the Relative Absolute Error checks if a model performs better than just predicting the average (i.e., the simple model).
The formula of the Relative Absolute Error:
, where:
- n: represents the number of observations
- yi: represents the realized value
- ŷi: represents the predicted value
- ȳ: represents the average of the realized values
The interpretation of the Relative Absolute Error is simple: if the RAE is smaller than one, then the model performs better than the simple model. In the case of a perfect model, the Relative Absolute Error is 0. Typically, you want an RAE as close a possible to zero.
Comparison between the RAE, RSE, and RRSE
Besides the RAE, there also exist the RSE (Relative Squared Error) and RRSE (Root Relative Squared Error).
Although the definition of each metric is slightly different, you can use all metrics to compare models whose errors are measured in different units. The main difference between the 3 metrics is how they consider the gravity of each error. For example, the RAE considers each error equally important whereas the RSE assumes that bigger errors are more severe.
We have written separate articles about the Relative Squared Error and the Root Relative Squared Error.
3 Ways to Calculate the Relative Absolute Error in R
Before we show 3 methods to calculate the Relative Absolute Error in R, we first create 2 variables with random numbers. These variables represent the realized values (y) and predicted values (y_hat).
We use the SAMPLE.INT() and RNORM() functions to create the variables.
set.seed(123)
y <- sample.int(100, 100, replace = TRUE)
y
set.seed(123)
y_hat <- y + rnorm(100, 0, 1)
y_hat1. Calculate the RAE with Basic R Code
Firstly, you can calculate the Relative Absolute Error in R is by writing your own code. You only need the SUM(), ABS(), and MEAN() functions to find the RAE.
These are the steps:
- Use the ABS() and SUM() functions to calculate the sum of absolute errors of the predictive model.
- Use the ABS(), MEAN(), and SUM() functions to calculate the sum of the absolute errors of the simple model.
- Divide the result of Step 1 by the result of Step 2.
Syntax:
sum(abs(realized - predicted)) / sum(abs(realized- mean(realized)))
The advantage of writing your own code is that people not familiar with the RAE can grasp this metric rapidly.
Example:
sum(abs(y - y_hat)) / sum(abs(y - mean(y)))
2. Calculate the RAE with a Function from a Package
Secondly, you can find the RAE by using the RAE() function from the Metrics package. This method does exactly the same as the first method, but with less code.
The RAE() function requires two parameters, namely the realized and predicted values ad, as a result, it returns the Relative Absolute Error.
Syntax:
rae(actual, predicted)
Example:
library(Metrics)
rae(actual = y, predicted = y_hat)
As you can see, the result of the first and the second method is the same.
3. Calculate the RAE with a User-Defined Function
Lastly, you can create a user-defined function to calculate the Relative Absolute Error.
Although the first and the second method work fine in normal circumstances, they are not robust. For example, both can’t handle situations where the realized and/or the predicted values contain NA’s (i.e., missing values).
y1 <- y
y1[1] <- NA
y_hat1 <- y_hat
y_hat1[1] <- NA
sum(abs(y1 - y_hat1)) / sum(abs(y1 - mean(y1)))
rae(actual = y1, predicted = y_hat1)
Therefore, we recommend creating a user-defined function that calculates the Relative Absolute Error in case of missing values. Below we show an example of such a function.
Example:
RelativeAbsError <- function(realized, predicted, na.rm = TRUE){
abs_error <- abs(realized - predicted)
mean_realized <- mean(realized, na.rm = na.rm)
sum(abs_error, na.rm = na.rm) / sum(abs(realized - mean_realized), na.rm = na.rm)
}
RelativeAbsError(realized = y1, predicted = y_hat1)
