Model Insight

Barra China Equity Model (CNE5) Empirical Notes

July 2012

5. Forecasting Accuracy
5.1. Overview of Testing Methodology
In this section, we describe our methodology for evaluating and comparing the accuracy of risk model
forecasts. We aim for a systematic and quantitative approach, yet one that is also visually intuitive.
The foundation of our approach rests on the bias statistic, described in Appendix C. Conceptually, the
bias statistic is an out-of-sample measure that represents the ratio of realized risk to predicted risk. The
ideal bias statistic for perfect risk forecasts should be close to 1. However, even for perfect risk
forecasts, the bias statistic will never be exactly 1 due to sampling error. Nevertheless, we may define a
confidence interval that is expected to contain 95 percent of the observations under the hypothesis of
perfect risk forecasts. If the bias statistic falls outside of the confidence interval, we infer that the risk
forecast was not accurate.
When determining the size of the confidence interval, standard practice is to assume that returns are
normally distributed. In reality, however, stock returns tend to have fat tails (i.e., positive excess
kurtosis). As shown in Appendix C, fewer than 95 percent of the observations are expected to fall within
the standard confidence interval when kurtosis is taken into account.
We are interested in testing a full sample period that lasts more than 15 years. One potential
shortcoming of the bias statistic is that over long windows, we may have sub-periods of overforecasting
and underforecasting, yet obtain a bias statistic close to 1 over the entire window. In other words,
forecasting errors may cancel out over the long term, even though the risk forecasts may be poor over
shorter periods. For a portfolio manager who may be devastated by a single year of poor performance,
it is small consolation knowing that a risk forecast is good on average.
For this reason, we focus on 12-period rolling windows. In the case of the standard models, CNE5S and
CNE5L, we use 12-month rolling windows. For the daily model, CNE5D, we use 12-day rolling windows.
By plotting the mean rolling 12-period bias statistic across time for a collection of portfolios, we quickly
visualize the magnitude of the average biases and can judge whether they were persistent or regime-
dependent.
It is not enough, however, knowing the average bias statistic. We must also understand the extremes.
We also compute, therefore, the 5-percentile (P5) and 95-percentile (P95) bias statistics across time.
Assuming normally distributed returns and perfect risk forecasts, on average 5 percent of the rolling 12-
period bias statistics will fall below 0.66 by pure chance. Therefore, if the P5 bias statistic falls
significantly below this level, we infer that we are likely overpredicting the risk of at least some of the
portfolios with bias statistics below 0.66. Similarly, if the P95 bias statistic lies well above 1.34, we infer
that we are underpredicting the risk of some portfolios with bias statistics above 1.34. It is worth
pointing out, however, that if we relax the normality assumption and allow for fat-tailed distributions,
then for perfect risk forecasts the P5 bias statistic tends to fall below 0.66, and the P95 value generally
lies above 1.34.
Another measure that provides insight into the accuracy of risk forecasts is the mean rolling absolute
deviation, or MRAD. As described in Appendix C, this is computed by averaging the absolute deviation of
the bias statistics from 1 for a collection of portfolios. Conceptually, MRAD penalizes any deviation from
the ideal bias statistic of 1, whether due to overforecasting or underforecasting.

MSCI Portfolio Management Analytics [Link]

© 2012 MSCI Inc. All rights reserved.
Please refer to the disclaimer at the end of this document 29 of 59
 Model Insight
Barra China Equity Model (CNE5) Empirical Notes
July 2012

Assuming normally distributed returns and perfect risk forecasts, the expected value of MRAD is 0.17.
Real financial returns, of course, tend to have fat tails. As shown in Appendix C, kurtosis levels within the
range of 3.5 to 4.0 lead to MRAD values of approximately 0.19 for perfect risk forecasts. When
comparing MRAD values across two models, it is crucial to keep in mind the lower bound of MRAD. For
instance, assuming a 0.19 lower bound, reducing MRAD from 0.23 to 0.21 constitutes a 50 percent
reduction in excess MRAD.
It is also important to recognize that MRAD is a statistical measure. As such, by pure chance the MRAD
may dip below the level of 0.17. Indeed, consider a portfolio that has been overforecast for many
periods, leading to a bias statistic less than 1. Eventually, the risk model may begin underforecasting the
risk of that same portfolio. When the transition from overforecasting to underforecasting occurs, the
bias statistic must necessarily cross through 1, thereby producing an MRAD value close to zero. For a
large collection of portfolios, however, it is highly improbable that the bias statistics of all portfolios will
cross through 1 simultaneously. Consequently, for a sufficiently diverse set of portfolios, the MRAD is
unlikely to dip significantly below 0.17 for any sustained period of time.
Our testing approach therefore relies principally on these four measures: the mean bias statistic, the P5
and P95 bias statistics, and the MRAD. All are computed and plotted on a rolling 12-period basis. These
plots allow us to quickly evaluate the accuracy of risk forecasts in a visually intuitive manner.
In order to develop a better understanding for how these measures behave in the ideal case of
stationary returns and perfect risk forecasts, we perform two separate simulations for 100 sets of
returns over 191 months (representing July 1995 through May 2011). In the first simulation, the returns
were drawn from a standard normal distribution. In the second simulation, the returns were drawn from
a t-distribution with standard deviation of 1 and kurtosis of 4. In all simulations, the predicted volatilities
were equal to 1 (i.e., perfect risk forecasts).
In Figure 5.1 we plot MRAD and bias statistics for the mean, P5 and P95 levels. The dashed horizontal
lines represent the ideal positions of the curves for the case of perfect forecasts and normal
distributions. On the left panel (normal distribution), we see that the realized curves indeed lie close to
their ideal positions. In particular, the MRAD is closely centered at the 0.17 level. Note that the degree
of “noise” in the lines depends on the number of portfolios in the sample. That is, the more portfolios
that we use, the smaller the observed variability.
On the right panel of Figure 5.1 we plot MRAD and bias statistics for perfect risk forecasts and a kurtosis
of 4. The effect of higher kurtosis is to increase the frequency of observations with bias statistics above
1.34 or below 0.66. In this case, the mean of the P5 line is shifted down to 0.61, whereas the P95 line
moves upward to a mean of 1.40. This has the effect of increasing MRAD to approximately 0.19.

MSCI Portfolio Management Analytics [Link]

© 2012 MSCI Inc. All rights reserved.
Please refer to the disclaimer at the end of this document 30 of 59
 Model Insight
Barra China Equity Model (CNE5) Empirical Notes
July 2012

Figure 5.1: Simulated results for the rolling 12-month mean bias statistic, the 5-percentile and 95-percentile
bias statistics, and MRAD.

Perfect Forecasts (Normal) Perfect Forecasts (K=4)

2.5 2.5

2 2

1.5 1.5
P95

1 Mean 1

P5
0.5 0.5

MRAD
0 0
1996 1999 2002 2005 2008 2011 1996 1999 2002 2005 2008 2011
Year Year

It is worth reiterating that Figure 5.1 represents the idealized case of perfect risk forecasts and
stationary returns. In reality, risk forecasts are never perfect and returns are not stationary.
Nevertheless, Figure 5.1 serves as a useful baseline for understanding the empirical backtesting results
that follow.

MSCI Portfolio Management Analytics [Link]

© 2012 MSCI Inc. All rights reserved.
Please refer to the disclaimer at the end of this document 31 of 59