Lesson 4: Forecast Diagnostics and Prediction Intervals

Ágoston Kolos Csaba

Department of Operations Research and Actuarial Sciences

Institute of Operations and Decision Sciences
Corvinus University of Hungary

2026.

Course materials were developed with the assistance of ChatGPT.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 1 / 31

Lecture Overview

Today we discuss how to evaluate forecasting models.

Topics:
Forecast errors (short recap)
Additional accuracy measures
Residual diagnostics
Prediction intervals
Forecast uncertainty

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 2 / 31

Forecast Error (Recap)

Definition
et = yt − ŷt

yt = observed value
ŷt = forecast

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 3 / 31

Forecast Error (Recap)

Definition
et = yt − ŷt

yt = observed value
ŷt = forecast
Interpretation:
Positive error → under-forecast
Negative error → over-forecast

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 3 / 31

Accuracy Measures (Recap)

Previously we introduced:

Mean Error (ME)

1X
ME = et
n
Mean Absolute Error (MAE)

1X
MAE = |et |
n
Mean Squared Error (MSE)

1X 2
MSE = et
n

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 4 / 31

Root Mean Squared Error

Definition
r
1X 2
RMSE = et
n

Square root of MSE

Same unit as the original data
Large errors receive stronger penalty

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 5 / 31

Mean Absolute Percentage Error

Definition
100 X et
MAPE =
n yt

Error expressed as percentage

Easy to interpret

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 6 / 31

Mean Absolute Percentage Error

Definition
100 X et
MAPE =
n yt

Error expressed as percentage

Easy to interpret

Limitation
Problematic when observed values are close to zero.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 6 / 31

Comparing Forecast Models

How can we decide which forecasting method is better?

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 7 / 31

Comparing Forecast Models

How can we decide which forecasting method is better?

Calculate accuracy measures
Compare errors across models

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 7 / 31

Comparing Forecast Models

How can we decide which forecasting method is better?

Calculate accuracy measures
Compare errors across models

General Rule
Lower forecast error indicates better model performance.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 7 / 31

What Makes a Good Forecasting Model?

A good model should satisfy two conditions:

1 Forecast errors should be small

2 Residuals should behave like random noise

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 8 / 31

Residual Diagnostics

Residual
Residual = forecast error produced by the model.

We analyze residuals to check whether the model captured the structure of

the data.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals 9 / 31

What Should Residuals Look Like?

Residuals should behave like white noise.

Meaning:
Mean close to zero
No systematic pattern
No autocorrelation
Constant variance

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals10 / 31

Residual Plot

One simple diagnostic tool:

Plot residuals over time.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals11 / 31

Residual Plot

One simple diagnostic tool:

Plot residuals over time.

Warning signs:
Trend
Seasonal pattern
Long runs of positive or negative errors
Increasing variance

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals11 / 31

Error plot

Value Fitted trend

Time

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals12 / 31

Error plot

Error
Residuals

Time

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals13 / 31

Heteroscedasticity

Value

Fitted trend

Time

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals14 / 31

Autocorrelation of Residuals

If residuals are correlated, the model did not capture all patterns in the
data.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals15 / 31

Autocorrelation of Residuals

If residuals are correlated, the model did not capture all patterns in the
data.
Common diagnostic tool:
Residual autocorrelation function (ACF)

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals15 / 31

Autocorrelation of Residuals

If residuals are correlated, the model did not capture all patterns in the
data.
Common diagnostic tool:
Residual autocorrelation function (ACF)
Significant spikes indicate remaining structure.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals15 / 31

Ljung–Box Test

Statistical test for residual autocorrelation.

Null hypothesis Residuals are not autocorrelated.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals16 / 31

Ljung–Box Test

Statistical test for residual autocorrelation.

Null hypothesis Residuals are not autocorrelated.

Interpretation:
Large p-value → residuals behave like noise
Small p-value → model may be inadequate

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals16 / 31

In-Sample vs Out-of-Sample Errors

When evaluating forecasts we must distinguish between two types of

errors. In-sample error

Error calculated on the data used to estimate the model

Out-of-sample error
Error calculated on new data not used for model estimation

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals17 / 31

In-Sample Error

In-sample errors measure how well the model fits the historical data.

Example:
Data from 2018–2024
Model estimated on this dataset
Errors calculated on the same observations

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals18 / 31

In-Sample Error

In-sample errors measure how well the model fits the historical data.

Example:
Data from 2018–2024
Model estimated on this dataset
Errors calculated on the same observations

Interpretation
Measures goodness of fit, not necessarily forecasting ability.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals18 / 31

Out-of-Sample Error

Out-of-sample errors evaluate true forecasting performance.

Example:
Estimate model using data from 2018–2023
Produce forecasts for 2024
Compare forecasts with the actual observations

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals19 / 31

Out-of-Sample Error

Out-of-sample errors evaluate true forecasting performance.

Example:
Estimate model using data from 2018–2023
Produce forecasts for 2024
Compare forecasts with the actual observations

Important
The model has never seen these observations before.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals19 / 31

Why Out-of-Sample Evaluation Matters

A model may fit historical data extremely well but still produce poor
forecasts.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals20 / 31

Why Out-of-Sample Evaluation Matters

A model may fit historical data extremely well but still produce poor
forecasts.
This often happens because of:
Overfitting
Too many parameters
Capturing noise instead of real patterns

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals20 / 31

Why Out-of-Sample Evaluation Matters

A model may fit historical data extremely well but still produce poor
forecasts.
This often happens because of:
Overfitting
Too many parameters
Capturing noise instead of real patterns

Key Idea
The real test of a forecasting model is how well it predicts unseen data.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals20 / 31

Typical Evaluation Procedure

A common approach is the training–test split.

Training sample
▶ used for model estimation
Test sample
▶ used for evaluating out-of-sample errors

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals21 / 31

Typical Evaluation Procedure

A common approach is the training–test split.

Training sample
▶ used for model estimation
Test sample
▶ used for evaluating out-of-sample errors
Forecast accuracy is usually reported on the test sample.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals21 / 31

Point Forecasts vs Forecast Uncertainty

So far we focused on point forecasts.

ŷt+h

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals22 / 31

Point Forecasts vs Forecast Uncertainty

So far we focused on point forecasts.

ŷt+h

But forecasts are uncertain.

We need a way to measure this uncertainty.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals22 / 31

Prediction Intervals

Prediction Interval
A range that future observations are expected to fall into with a given
probability.

General form:
ŷt+h ± zα/2 σh

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals23 / 31

Prediction vs Confidence Intervals

Confidence interval
Uncertainty of the estimated mean

Prediction interval
Uncertainty of future observations

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals24 / 31

Prediction vs Confidence Intervals

Confidence interval
Uncertainty of the estimated mean

Prediction interval
Uncertainty of future observations
Prediction intervals are always wider.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals24 / 31

Forecast Horizon and Uncertainty

Prediction intervals widen as the forecast horizon increases.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals25 / 31

Forecast Horizon and Uncertainty

Prediction intervals widen as the forecast horizon increases. Reason:

Forecast errors accumulate
Future becomes more uncertain

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals25 / 31

Visualizing Forecast Uncertainty

Forecasts are often presented with intervals:

point forecast
lower prediction bound
upper prediction bound

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals26 / 31

Visualizing Forecast Uncertainty

Forecasts are often presented with intervals:

point forecast
lower prediction bound
upper prediction bound
The result often looks like a fan-shaped forecast.

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals26 / 31

Prediction interval (regression)

110
Observed
100 Trend
Forecast
90

80
Value

70

60

50

40

30
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
Period

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals27 / 31

Prediction interval (Holt method)

110
Observed
100 Trend
Forecast
90

80
Value

70

60

50

40

30
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
Period

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals28 / 31

Changing trend

Value Forecast (regression)

Forecast (Holt)

Time

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals29 / 31

Key Takeaways

Forecast errors measure model accuracy

Several error measures exist
Residual diagnostics help detect model problems
Forecasts are uncertain
Prediction intervals quantify this uncertainty

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals30 / 31

Thank you for your attention!

Ágoston K. Cs. Lesson 4: Forecast Diagnostics and Prediction Intervals31 / 31