Backtesting VaR
Backtesting
Backtesting is the process of comparing losses predicted
by the VaR model to those actually experienced over the
testing period.
If a model were completely accurate, we would expect VaR
loss limits to be exceeded (this is called an exception) with
the same frequency predicted by the confidence level used
in the VaR model.
© Kaplan, Inc. 1
1
Internal Models Approach for Market Risk Capital
Basel Accord
Basel I (1988) – Focused on Credit Risk
Basel I Amendment (1995) – CR, Introduced Netting
Basel I Amendment (1996) – MR, Standardized & Internal Models
Internal Models Approach (VaR, 99% CL, 10 days)
Previous day VaR
Average of last 60 days VaR × Multiplier (minimum multiplier = 3)
© Kaplan, Inc. 2
Backtesting Exceptions
An unbiased measure of the number of exceptions as a
proportion of the number of samples is called the failure rate.
The probability of exception equals one minus the confidence
level, or (p = 1 – c).
If N is the number of exceptions and T is the number of samples,
then N/T is the failure rate.
A sample cannot be used to determine with absolute certainty
whether or not the model is accurate.
© Kaplan, Inc. 3
2
Conditional Coverage in Backtesting
A problem with backtesting is that timing of the exceptions
is not considered.
Also, exceptions occur fairly equally over time—no
bunching, exceptions, or independent.
If exceptions bunch, a likelihood ratio test that determines
serial dependency should be performed.
© Kaplan, Inc. 4
Basel Committee Rules
Backtesting is required at the 99% confidence level over
the past year (250 business days), so we would expect 2.5
exceptions.
We must balance the probabilities of two types of errors: a
Type I error is rejecting an accurate model, and a Type II
error is accepting an inaccurate model.
© Kaplan, Inc. 5
3
Basel Committee Rules (cont.)
The Basel rules establish zones of the number of
exceptions and corresponding penalties or increases in the
capital requirement multiplier from 3 to 4 (i.e., safety
factor).
These zones (green, yellow, and red) will be discussed in
greater detail in the Operational Risk and Resiliency topic
area (Book 3).
© Kaplan, Inc. 6
GARP 2020 Market Risk Q58 (also 2019, 2018 & 2017 Q58)
In the Basel framework, a penalty is given to banks that have more than four exceptions to their
1-day 99% VaR over the course of the last 250 trading days. Which of the following causes of
exceptions is most likely to lead to a penalty?
A. A large move in interest rates was combined with a small move in correlations.
B. The bank’s model calculates interest rate risk based on the median duration of the
bonds in the portfolio.
C. A sudden market crisis in an emerging market, which leads to losses in the equity
positions in that country.
D. A sudden devastating earthquake that caused major losses in the bank’s key area of
operation.
© Kaplan, Inc. 7
4
GARP 2020 Q58—Answer
Correct answer: B
In the case of a bank that changed positions more frequently during the day, a penalty
should be considered, but it is not necessarily given. In the case of bad luck, no penalty is
given, as would be the case for a bank affected by unpredictable movements in rates or
markets. However, when risk models are not precise enough, a penalty is typically given since
model accuracy could have easily been improved.
© Kaplan, Inc. 8
Hypothesis Test – VaR Model Calibrated Correctly?
Null Hypothesis: VaR model is calibrated correctly
Alt Hypothesis: VaR model is not calibrated correctly
Significance level of test: 5%, (95% CL)
Compare: Test Statistic vs Critical Value (1.96)
If: Test Statistic > Critical Value = reject null
x T
Teststatistic
1 T
© Kaplan, Inc. 9
5
GARP 2017 Market Risk Q12
A risk manager is backtesting a company’s 1-day 99.5% VaR model over a 1-year
horizon at a 95% confidence level. Assuming 250 days in a year, what is the maximum
number of daily losses exceeding the 1-day 99.5% VaR that is acceptable to conclude
that the model is calibrated correctly?
A. 3.
B. 5.
C. 15.
D. 19.
© Kaplan, Inc. 10
GARP 2017 Q12—Answer
Null hypothesis: VaR model is calibrated correctly
Alternative hypothesis: VaR model is not calibrated correctly
Test statistic vs. Critical value (based on test significance level)
x T
Test statistic vs. 1.96 (for a 5% significance level)
1 T
3 0.005 250 5 0.005 250
Teststatistic 1.57 Teststatistic 3.36
0.00510.005 250 0.0051 0.005 250
Conclusion
The maximum number of exceptions is 3
Instructor Tip: We input the VaR significance level in the formula (not the test SL)
© Kaplan, Inc. 11
6
GARP 2020 Market Risk Q12 (same as 2019 & 2018 Q12)
A risk manager is backtesting a company’s 1-day 99.5% VaR model over a 10-year horizon at a
95% confidence level. Assuming 250 days in a year and the daily returns are independently and
identically distributed, what is the maximum number of daily losses exceeding the 1-day 99.5%
VaR in 10 years that is acceptable to conclude that the model is calibrated correctly?
A. 19.
B. 25.
C. 35.
D. 39.
© Kaplan, Inc. 12
GARP 2020 Q12—Answer
Null hypothesis: VaR model is calibrated correctly
Alternative hypothesis: VaR model is not calibrated correctly
Test statistic vs. Critical value (based on test significance level)
x T
Test statistic
1 T vs. 1.96 (for a 5% significance level)
19 0.005 250 10 25 0.005 250 10
Teststatistic 1.84 Test statistic 3.54
0.00510.005 250 10 0.005 1 0.005 250 10
Conclusion
The maximum number of exceptions is 19. Correct answer: A
Instructor Tip: The time horizon is now 10 years!
© Kaplan, Inc. 13
7
Backtesting Type I & II Errors
Null Hypothesis: model is calibrated correctly
Alternative Hypothesis: model is not calibrated correctly
Type I error: reject the null in error, we reject a good model
Type II error: fail to reject null in error, we keep a bad model
Question
Which is the bigger problem?
© Kaplan, Inc. 14
Changing from a Higher to a Lower VaR CL
Example change from 99% to 95% VaR Confidence Level:
We have a higher number of exceptions (5% v 1%)
With more data, the conclusions are more significant and reliable
What happens to Type I and II Errors?
Type II errors: reduce, less chance of retaining a bad model
Type I errors: increase, more chance of rejecting a good model
© Kaplan, Inc. 15
8
GARP 2020 Market Risk Q53 (also 2019, 2018 & 2017 Q53)
A newly hired risk analyst is backtesting a firm’s VaR model. Previously, the firm calculated a
1-day VaR at the 95% confidence level. Following the Basel framework, the risk analyst is
recommending that the firm switch to a 99% confidence level. Which of the following statements
concerning this switch is correct?
A. The decision to accept or reject a VaR model based on backtesting results is less
reliable with a 99% confidence level VaR model than with a 95% confidence level model.
B. The 95% VaR model is less likely to be rejected using backtesting than the 99% VaR
model.
C. When validating with backtesting at the 90% confidence level, there is a smaller
probability of incorrectly rejecting a 95% VaR model than a 99% VaR model.
D. When backtesting using a 90% confidence level, there is a smaller probability of
committing a type I error when backtesting a 95% VaR model than with a 99% VaR model.
© Kaplan, Inc. 16
GARP 2020 Q53—Answer
Correct answer: A
The concept tested here is the understanding of the VaR confidence level to Type I and Type II
errors, and the impact on the statistical significance of the backtest.
Type I Error—rejecting the null (in error)…rejecting an accurate model
Type II Error—failing to reject the null (in error)…failing to reject an accurate model
Moving from a VaR 97.5% CL to 95% CL (numbers from the Schweser Mapping
Topic)
Type I error increases from 10.8% to 12.5%
Type II error reduces from 12.8% to 7.4%
Key learning: Reducing the CL of the VaR model results in more exceptions, resulting
in a higher statistical significance. So the lower the CL the more reliable the backtest
(Type II errors fall).
© Kaplan, Inc. 17