In this article, we discuss how to calculate the Relative Squared Error (RSE) in R.
The Relative Squared Error is a simple metric to measure the performance of predictive models, such as regression. Although this metric is basic, it is a good start to assess model quality.
Moreover, you can use the RSE to simply calculate the popular R-squared metric.
But, how do you calculate the Relative Squared Error?
In R, you calculate the Relative Squared Error (RSE) with one simple line of code or by using the RSE() function from the Metrics package. Either way, you only need to provide the realized values and predicted values to calculate the RSE.
In this article, we discuss 3 methods to calculate the Relative Squared Error. We provide examples and R code that you can use in your own project. But first, we take a look at the definition of the RSE and its interpretation.
The Relative Squared Error
Definition & Formula
The Relative Squared Error (RSE) is a relative metric that divides the squared error of a predictive model by the squared error of a simple model. In general, one uses the average of the realized values as a simple model.
The image below shows the mathematical formula of the RSE.
, where:
- n: represents the number of observations
- yi: represents the realized value
- ŷi: represents the predicted value
- ȳ: represents the average of the realized values
The interpretation of the Relative Squared Error is simple: if the RSE is lower than 1, then the predictive model performs better than the simple model. In other words, the error of the predicted model is smaller than the error one obtains by simply using the mean as the predictor.
The lower the RSE, the better the model. A perfect model would have an RSE of zero.
Although not frequently mentioned, there is a simple relationship between the Relative Squared Error and the well-known R-squared. Namely, the R-squared of a model is defined as one minus the RSE.
Comparison RSE, RAE, & RRSE
Besides the RSE, there also exist the Relative Absolute Error (RAE) and the Root Relative Squared Error (RRSE).
All metrics (RSE, RAE, and RRSE) compare the performance of an (advanced) model with the performance of a simple model. Likewise, the lower the metric, the better the model.
The difference between the RSE, RAE, and RRSE is that both the RSE and RRSE penalize severe errors more than the RAE.
We have written separate articles that discuss the Relative Absolute Error and Root Relative Squared Error in more detail.
3 Ways to Calculate the Relative Squared Error in R
Before we show 3 methods to calculate the Relative Squared Error in R, we first create two vectors. These vectors represent the realized values (y) and predicted values (y_hat).
We use the SAMPLE.INT() and RNORM() functions to create the vectors.
set.seed(123)
y <- sample.int(100, 100, replace = TRUE)
y
set.seed(123)
y_hat <- y + rnorm(100, 0, 1)
y_hat1. Calculate the RSE with Basic R Code
An easy way to calculate the Relative Squared Error (RSE) is by using basic R code. It just requires one line of simple code and 3 functions, namely SUM(), ABS(), and MEAN().
This is our preferred method if you want to keep your code readable. Moreover, it shows directly the definition of the RSE which is useful for those not familiar with this metric.
These are the steps to calculate the RSE:
- Square the prediction error (i.e., realized value minus predicted value).
- Sum the squared errors.
- Square the difference between the realized value and the average of all realized values.
- Sum the differences.
- Divide the result of Step 2 by the result of Step 4
In R, you can use either the double asterisks (**) or the caret (^) to raise a number to the desired power.
Syntax
sum((realized - predicted)**2) / sum((realized - mean(realized))**2)
Example
The R code below shows the RSE of the vectors previously created.
sum((y - y_hat)**2) / sum((y - mean(y))**2)
2. Calculate the RSE with a Function from a Package
Alternatively, you can calculate the Relative Squared Error (RSE) with the RSE() function from the Metrics package.
This function is convenient for those who are familiar with the definition of the RSE and want tidy R code. You only need to provide the realized values and the predicted values, and the RSE() function returns the Relative Squared Error.
Besides the RSE() function, the Metrics package has many other functions to assess the performance of predictive models.
Syntax
rse(actual, predicted)
Example
library(Metrics)
rse(actual = y, predicted = y_hat)
3. Calculate the RSE with a User-Defined Function
Lastly, you can find the Relative Squared Error (RSE) by writing your own user-defined function.
Creating a user-defined function is especially useful when the realized values and/or the predicted values might have missing values (NA’s). As the example below shows, the previously discussed methods are not robust. That is to say, they return a NA for input data containing missing values.
y1 <- y
y1[1] <- NA
y_hat1 <- y_hat
y_hat1[1] <- NA
sum((y1 - y_hat1)**2) / sum((y1 - mean(y1))**2)
rse(actual = y1, predicted = y_hat1)
By writing a user-defined function, you can handle possible missing values. For example, the R code below shows a robust function to calculate the Relative Squared Error.
RelativeSquaredError <- function(realized, predicted, na.rm = TRUE){
error <- realized - predicted
mean_realized <- mean(realized, na.rm = na.rm)
sum(error**2, na.rm = na.rm) / sum((realized - mean_realized)**2, na.rm = na.rm)
}
RelativeSquaredError(realized = y1, predicted = y_hat1)
