Chapter 4: Backtesting VaR hypothetical return R*t represents a frozen
portfolio, obtained from fixed positions applied
Model validation is the general process of
to the actual returns on all securities, measured
checking whether a model is adequate. This can
from close to close.
be done with a set of tools, including
backtesting, stress testing, and independent Sometimes an approximation is obtained by
review and oversight. This chapter turns to using a cleaned return, which is the actual
backtesting techniques for verifying the return minus all non-mark-to-market items, such
accuracy of VaR models. as fees, commissions, and net interest income.
Backtesting is a formal statistical framework if the model does not pass backtesting with
that consists of verifying that actual losses are in hypothetical returns, then the modeling
line with projected losses. This involves methodology should be reexamined.
systematically comparing the history of VaR
4.2 MODEL BACKTESTING WITH EXCEPTIONS
forecasts with their associated portfolio returns.
Model backtesting involves systematically
4.1 SETUP FOR BACKTESTING
comparing historical VaR measures with the
When the model is perfectly calibrated, the subsequent returns.
number of observations falling outside VaR
Model Verification Based on Failure Rates
should be in line with the confidence level. The
number of exceedences is also known as the The simplest method to verify the accuracy of
number of exceptions. With too many the model is to record the failure rate, which
exceptions, the model underestimates risk. This gives the proportion of times VaR is exceeded in
is a major problem because too little capital may a given sample.
be allocated to risk-taking units; penalties also
may be imposed by the regulator. Too few Define N as the number of exceptions and N/T
exceptions are also a problem because they lead as the failure rate. Ideally, the failure rate should
to excess, or inefficient, allocation of capital give an unbiased measure of p, that is, should
across units.
VaR measures assume that the current
portfolio is "frozen" over the horizon. In
practice, the trading portfolio evolves converge to p as the sample size increases.
dynamically during the day. Thus the actual
portfolio is "contaminated" by changes in its
composition. The actual return corresponds to We want to know, at a given confidence level,
the actual P&L, taking into account intraday whether N is too small or too large under the
trades and other profit items such as fees, null hypothesis that p = 0.01 in a sample of size
commissions, spreads, and net interest income. T. Note that this test makes no assumption
This contamination will be minimized if the about the return distribution. The distribution
horizon is relatively short. could be normal, or skewed, or with heavy tails,
or time-varying. We simply count the number of
For verification to be meaningful, the risk exceptions. As a result, this approach is fully
manager should track both the actual portfolio nonparametric.
return Rt and the hypothetical return R*t; that
most closely matches the VaR forecast. The
Unconditional coverage refers to the fact that
we are not concerned about the independence
of exception observations or the timing of when
the exceptions occurs.
so the model is not accurate.
The Basel Rules
The setup for this test is the classic testing The current verification procedure consists of
framework for a sequence of success and recording daily exceptions of the 99 percent VaR
failures, also called Bernoulli trials. over the last year. One would expect, on
average, 1 percent of 250, or 2.5 instances of
For backtesting purposes, users of VaR models
exceptions over the last year. The Basel
need to balance type 1 errors against type 2
Committee has decided that up to four
errors. Ideally, one would want to set a low type
exceptions are acceptable, which defines a
"green light" zone for the bank. If the number of
exceptions is five or more, the bank falls into a
"yellow" or "red" zone and incurs a progressive
penalty whereby the multiplicative factor k is
increased from 3 to 4, as described in Table 4.3.
1 error rate and then have a test that creates a An incursion into the "red" zone gener ates an
very low type 2 error rate, in which case the test automatic penalty.
is said to be powerful.
unconditional coverage
Kupiec determined a measure to accept or
reject models using the tail points of the log-
likelihood ratio (LR).
Conditional Coverage Models