Unit4

Lecture1

Introduction to Time Series Regression and Forecasting

Time series data is collected for a single entity over time. This is fundamentally different
from cross-section data which is data on multiple entities at the same point in time. Time
series data allows estimation of the effect on Y of a change in X over time. This is what
econometricians call a dynamic causal effect. Let us go back to the application of cigarette
consumption of where we were interested in estimating the effect on cigarette demand of a
price increase caused by a raise of the general sales tax. One might use time series data to
assess the causal effect of a tax increase on smoking both, initially and in subsequent periods.
Another application of time series data is forecasting. For example, weather services use time
series data to predict tomorrow’s temperature by, inter alia, using today’s temperature and
temperatures of the past. To motivate an economic example, central banks are interested in
forecasting next month’s unemployment rates.

The remaining chapters in the book deal with econometric techniques for analyzing time
series data, as well as their applications in forecasting and estimating dynamic causal
effects. This section covers the basic concepts presented here, explains how to visualize time
series data and demonstrates how to estimate simple autoregressive models, where the
regressors are past values of the dependent variable or other variables. In this context we also
discuss the concept of stationarity, an important property which has far-reaching
consequences.

Most empirical applications in this chapter are concerned with forecasting and use data on
U.S. macroeconomic indicators or financial time series like Gross Domestic Product (GDP),
the unemployment rate or excess stock returns.

Regression models

Introduction

For regression analysis to be performed, data has to be stationary. Or the equation has to be
rewritten in such a form that indicates a relationship among stationary variables.

What is stationary? Stationarity refers to a situation where the underlying stochastic process
that generates the data is invariant with respect to time. On the other hand, if the
characteristics over the time changes we call it a non-stationary process.
Now the obvious question is what are the characteristics that has to be invariant? We can list
it in the following manner: mean variance covariance etc

If a series is stationary then we can model it via an equation with fixed coefficents estimated
from the past data.

Conditions for Stationarity

E(yt)is independent of t
What it means is that a stationary series must have a constant mean. That is it should revert
back to the mean. It many wonder around here and there but it should have an tendency to
come back to the mean. That means the series should show certain amount of randomness
around its mean with respect to time.

This also implies that expected value will be the same in each time period. Is that true? Think
about a stationary time series: change in the stock prices. Does this mean that expected
change in the stock price will be the same in each time period? Is this a realistic thing to
expect?

Let’s take a look into an example which might generate stable series:
 Lecture2

Spurious Regression

In a famous paper, Granger and Newbold(1974) introduced the notion of Spurious

Regression. They are the one who first warned that macroeconomic regression has highly
autocorrelated residual. That means regression are violating assumptions of no
autocorrelations.

In this situation, statistical inference goes haywire. We usually find statistically significant
relationship among variables. For example, we might find inflation is affected by
temperature and that is statistically significant which is totally absurd.

In general,regression involving two integrated time series of the same order yields significant
relationship which is called spurious regression. This is characterized by high R2�2,
significant t� but this has no meaning. But this does not apply to cointegration which is a
special case. We will elaborate on this later.
A spurious regression is like this:

yt=α+β1xt+ϵt

Here yt and xt are both integrated time series.

Statistical properties of regression

It has been already established that statiscal properties of OLS hold only when the underlying
stochastic process is stationary. Data is stationary or the stochastic process is stationary? Are
they the same thing? Data is realization of one particular instance of the underlying
stochastic process. Then how they are the same?

Trends in time series can be classified as stochastic or deterministic. We may consider a

trend to be stochastic when it shows inexplicable changes in direction We attribute apparent
transient trends to high serial correlation with random error The meaning of above statement
is that the trend we see in the data is actually not deterministic, it is actually stochastic
because it was generated from high serial correlation in random error component

Now it says that data must have the trend removed before they can be used. There are two
common approaches to this:

-detrending using a time trend or some deterministic function of time - using differences

Question is which approach is the best?

It depends on the source of non-stationarity. If the non-stationarity stems from the trending
pattern, then obviously we should go for detrending with a time trend.
On the other hand, if non-stationarity stems from stochastic or random behaviour then
obviously the first approach will not work. In that case, the differncing has to be employed.

Deterministic Trends

We assume that the process is driven by a deterministic trend. Why the term deterministic?
Deterministic here probably means there is a specific pattern to it. It addition to deterministic
trend, there is also a random component to it. It it’s a linear trend, then it’s easy to take care.
Let’s start with a generic expression:

yt=f(t)+ϵt)Now if we consider a linear trend, then

yt=α+β∗t+ϵt
The least square residual from the above equation is a detrended stationary series which can
be used in the regression analysis.
But there is another type of non-stationarity which simply does not go away with detrending.
This is the type of non-stationarity that was mentioned by Granger and Newbold (1974). If
we still detrend it and use in the regression, it does not help actually. It is called stochastic
trending behaviour.

Integrated process that was mentioned in Granger and Newbold(1974) display stochastic
trend.

non-stationary.

Time series Regression

In this lesson, various regression models are studied that are suitable for a time series
analysis of data that contains deterministic trends and regular seasonal changes

We first look at linear models for trends

Then we introduce regression models that account for seasonal variation using indicator and
harmonic variables

Regression models can also include explanatory variables

The logarithmic transformation, which is often used to stabilise variance, is also considered

Difference from standard regression

Time series regression usually differs from a standard regression analysis because the
residuals themselves form a time series

Therefore residuals tend to be serially correlated

When this correlation is positive, the estimated standard errors will be biased, will tend to be
less than their true value

Low standard errors will lead to high value of t-statistics and as a result high statistical
significance will be attributed

Estimation process

Therefore, correct statistical evidence is important

For example, an environmental protection group can be accused of falsely claiming

statistically significant trends.

In this lesson, generalized least squares is used to obtain improved estimates of the standard
error to account for autocorrelation in the residual series.

Linear Models

Definition

A model for a time series xt:t=1,…n is linear if it can be expressed as

xt=α0+α1u1,t+α2u2,t+…+αmum,t+zt
Stationarity

Linear models for time series are non-stationary when they include functions of time

Differencing can often transform a non-stationary series with a deterministic trend to

stationary series

Suppose this is our modelxt=α0+α1t+zt

then first order differences are given by:

▽xt=xt−xt−1=zt−zt−1+α1▽
Assuming the error series zt is stationary
the series ▽xt▽ is stationary as it is not a function of t
Previously we have found that first-order differencing can transform a non-stationary series
with a stochastic trend to a stationary series

For example, random walk, the first difference makes it stationary series

Differencing can remove both stochastic and deterministic trends from time series

If the underlying trend is a polynomial or order m, then mth-order differencing is required to

remove the trend
Differencing the straight-line function plus white noise leads to a different stationary time
series compared to the one we can by subtracting the trend.

When we subtract the trend it gives white noise

When we difference, it gives rise to consecutive white noise terms which is an example of an
MA process
** Simulation **

In a time series, it is common for the error series zt in equation to be autocorrelated

In the code below, a time series with an increasing staright line trend 50+3t50+3� with
autocorrelated errors is simulated and plotted:
[Link](1)
z <- w <- rnorm(100, sd=20)
for (t in 2:100) z[t] <- 0.8*z[t-1]+w[t]
Time <- 1:100
x <- 50 + 3*Time +z
plot(x,xlab="time", type="l")

In the code above, the estimated parameters of the linear model are extracted using coef.
Note that estimates are close to underlying parameter values of 50 for the intercept and 3 for
the slope

The standard errors are extracted using the square root of the diagonal elements obtained
from vcov
These standard errors are to be underestimated because of autocorrelation in the residuals.
The function summary can also be used to obtain this information but tends to give additional
information
For example t-tests, which may be incorrect for a time series regression analysis due to
autocorrelation in the residuals

After fitting a regression model, we should consider various diagnostics plots

\
 Lecture3

Generalised least square (GLS)

We have seen that in time series regression it is common and expected that the residual series
will be autocorrelated

For a positive serial correlation in the resdiual series, this implies that the standard errors of
the estimated regression parameters are likely to be underestimated and should therefore be
corrected.

A fitting procedure known as generalized least square (GLS) can be used to provide better
estimates of the standard errors of the regression parameters to account for the
autocorrelation in the residual series

The procedure is essentially based on maximizing the likelihood given the autocorrelation in
the data and is implemented in R in the gls function within the nlme library

GLS fit to simulated series

Fitting simulated data

The following example illustrates how to fit a regression model to the simulated series

library(nlme)
[Link] <- gls(x ~ Time, cor=corAR1(0.8))
coef([Link])
sqrt(diag(vcov([Link])))
A lag 1 autocorrelation of 0.8 is used above because this value was used to simulate the data.
Since we have simulated data, we already knew the data generating process.

But in real life, we will have historial data. Hence we will have to first estimate the
coefficient from the residuals.

This value will be estimated from the correlogram of the residuals of a fitted linear model

The parameter estimates from GLS will generally be slightly different from those obtained
with OLS.

The slope is estimated as 3.06 using lm but 3.04 using gls

In principle GLS estimators are preferable because they have smaller standard errors

Temperature data

Let’s calculate an approximate 95% confidence interval for the trend in the global
temperature series (1970 - 2005)

GLS is used to estimate the standard error accounting for the autocorrelation in the residual
series
 In the gls funcdtion, the residual series is approximated as an AR(1) process with lag 1
autocorrelation of 0.7 read from figure
[Link] <- gls(temp ~ time(temp), cor = corAR1(0.7))
confint([Link])
The confidence interval above are now wider than they were previously

Zero is not contained in the intervals

This implies that the estimates are statistically signficant

That means the trend is significant

Thus there is statistical evidence of an increasing trend in global temperatures over the
period 1970-2005

If the current condition persists, temperatures may be expected to continue to rise in the
future

Linear models with seasonal variables

Introduction

As time series are observations mesured sequentially in time, sesonal effects are often
present in the data

Especially annual cycles caused directly or indirectly by the Earth’s movement around the
Sun

Seasonal effects have already been observed in several of the series we have looked at

 Airline series
 Temperature series
 Electricity production series

Additive seasonal indicator variables

Suppose a time series contains s seasons. For example, with time series measured over each
calendar month, s=12, for series measured over six-month intervals, corresponding to
summer and winter, s=2.
A seasonal indicator model for a time series xt:t=1,…,n
containing s seasons and a trend mt is given by
xt=mt+st+zt
where st=βi when t falls in the i-th season

Unit root

Since it is very important to learn whether a series is stationary or not, how do we know that?
Then comes the question of testing for stationarity and it has become known as .
To see why these are called Unit root tests we have to briefly overview what is root in a
polynomial [Link]’s think about a AR(1) process in the following form:
 Lecture4

Autoregressive Models

Autoregressive Models

A time series is a sequence of measurements of the same variable(s) made over time. Usually,
the measurements are made at evenly spaced times - for example, monthly or yearly. Let us first
consider the problem in which we have a y-variable measured as a time series. As an example,
we might have y a measure of global temperature, with measurements observed each year. To
emphasize that we have measured values over time, we use "t" as a subscript rather than the
usual "i," i.e., means measured in time period �.

An autoregressive model is when a value from a time series is regressed on previous values
from that same time series. for example,

In this regression model, the response variable in the previous time period has become the
predictor and the errors have our usual assumptions about errors in a simple linear regression
model. The order of an autoregression is the number of immediately preceding values in the
series that are used to predict the value at the present time. So, the preceding model is a first-
order autoregression, written as AR(1).

If we want to predict � this year using measurements of global temperature in the previous two
years), then the autoregressive model for doing so would be:

This model is a second-order autoregression, written as AR(2) since the value at time � is
predicted from the values at times x−1 and x−2. More generally, a �th-order autoregression,
written as AR(k), is a multiple linear regression in which the value of the series at any time t is a
(linear) function of the values at times

Autocorrelation and Partial Autocorrelation

The coefficient of correlation between two values in a time series is called the autocorrelation
function (ACF) For example the ACF for a time series �� is given by:

This value of k is the time gap being considered and is called the lag. A lag 1 autocorrelation
(i.e., k = 1 in the above) is the correlation between values that are one time period apart. More
generally, a lag k autocorrelation is a correlation between values that are k time periods apart.

The ACF is a way to measure the linear relationship between an observation at time t and the
observations at previous times. If we assume an AR(k) model, then we may wish to only measure
the association between and and filter out the linear influence of the random variables that lie in
between (i.e., which requires a transformation on the time series. Then by calculating the
correlation of the transformed time series we obtain the partial autocorrelation
function (PACF).

The PACF is most useful for identifying the order of an autoregressive model. Specifically,
sample partial autocorrelations that are significantly different from 0 indicate lagged terms
of � that are useful predictors of . To help differentiate between ACF and PACF, think of them
as analogs to �2 and partial �2 values as discussed previously.

Graphical approaches to assessing the lag of an autoregressive model include looking at the ACF
and PACF values versus the lag. In a plot of ACF versus the lag, if you see large ACF values and
a non-random pattern, then likely the values are serially correlated. In a plot of PACF versus the
lag, the pattern will usually appear random, but large PACF values at a given lag indicate this
value as a possible choice for the order of an autoregressive model. It is important that the choice
of order makes sense. For example, suppose you have had blood pressure readings for every day
over the past two years. You may find that an AR(1) or AR(2) model is appropriate for modeling
blood pressure. However, the PACF may indicate a large partial autocorrelation value at a lag of
17, but such a large order for an autoregressive model likely does not make much sense.

Example 14-1: Google Data

The Google Stock dataset consists of n = 105 values which are the closing stock price of a share
of Google stock from 2-7-2005 to 7-7-2005. We will analyze the dataset to identify the order of
an autoregressive model. A plot of the stock prices versus time is presented in the figure below
(Minitab: Stat > Time Series > Time Series Plot, select "price" for the Series, click the
Time/Scale button, click "Stamp" under "Time Scale" and select "date" to be a Stamp
column): [1]
Consecutive values appear to follow one another fairly closely, suggesting an autoregression
model could be appropriate. We next look at a plot of partial autocorrelations for the data:
To obtain this in Minitab select Stat > Time Series > Partial Autocorrelation. Here we notice that
there is a significant spike at a lag of 1 and much lower spikes for the subsequent lags. Thus, an
AR(1) model would likely be feasible for this data set.

Approximate bounds can also be constructed (as given by the red lines in the plot above) for this
plot to aid in determining large values. Approximate (1−�)×100% significance bounds are
given by ±�1−�/2/�. Values lying outside of either of these bounds are indicative of an
autoregressive process.

We can next create a lag-1 price variable and consider a scatterplot of price versus this lag-1
variable:

There appears to be a moderate linear pattern, suggesting that the first-order autoregression
model

could be useful.

Example 14-2: Quake Data

Let n = the annual number of worldwide earthquakes with a magnitude greater than 7 on the
Richter scale for n = 100 years (Earthquakes data obtained from the USGS Website). The plot
below gives a time series plot for this dataset. [2] [3]
The plot below gives a plot of the PACF (partial autocorrelation function), which can be
interpreted to mean that a third-order autoregression may be warranted since there are notable
partial autocorrelations for lags 1 and 3.
The next step is to do a multiple linear regression with the number of quakes as the response
variable and lag-1, lag-2, and lag-3 quakes as the predictor variables. (In Minitab, we used Stat
>> Time Series >> Lag to create the lag variables.) In the results below we see that the lag-3
predictor is significant at the 0.05 level (and the lag-1 predictor p-value is also relatively small).

Model Summary
S R-sq R-sq(adj) R-sq(pred)

3.85068 13.88% 11.10% 6.47%

Coefficients
Term Coef SE Coef T-Value P-Value VIF

Constant 6.45 1.79 3.61 0.000

lag1Quakes 0.164 0.101 1.63 0.106 1.07

lag2Quakes 0.071 0.101 0.70 0.484 1.12

lag3Quakes 0.2693 0.0978 2.75 0.007 1.09

Regression Equation

Quakes = 6.45 + 0.164 lag1Quakes + 0.071 lag2Quakes + 0.2693 lag3Quakes

Legend

[1]
Link

↥ Has Tooltip/Popover

Toggleable Visibility

Lecture4

Simple Linear Regression I – Least Squares Estimation

Previously, we have worked with a random variable x that comes from a population that is
normally distributed with mean µ and variance σ2. We have seen that we can write x in terms of
µ and a random error component ε, that is, x = µ + ε. For the time being, we are going to change
our notation for our random variable from x to y. So, we now write y = µ+ε. We will now find it
useful to call the random variable y a dependent or response variable. Many times, the response
variable of interest may be related to the value(s) of one or more known or controllable
independent or predictor variables. Consider the following situations: LR1 A college recruiter
would like to be able to predict a potential incoming student’s first–year GPA (y) based on
known information concerning high school GPA (x1) and college entrance examination score
(x2). She feels that the student’s first–year GPA will be related to the values of these two known
variables. LR2 A marketer is interested in the effect of changing shelf height (x1) and shelf
width (x2) on the weekly sales (y) of her brand of laundry detergent in a grocery store. LR3 A
psychologist is interested in testing whether the amount of time to become proficient in a foreign
language (y) is related to the child’s age (x). In each case we have at least one variable that is
known (in some cases it is controllable), and a response variable that is a random variable. We
would like to fit a model that relates the response to the known or controllable variable(s). The
main reasons that scientists and social researchers use linear regression are the following: 1.
Prediction – To predict a future response based on known values of the predictor variables and
past data related to the process. 2. Description – To measure the effect of changing a controllable
variable on the mean value of the response variable. 3. Control – To confirm that a process is
providing responses (results) that we ‘expect’ under the present operating conditions (measured
by the level(s) of the predictor variable(s)). 1.1 A Linear Deterministic Model Suppose you are a
vendor who sells a product that is in high demand (e.g. cold beer on the beach, cable television in
Gainesville, or life jackets on the Titanic, to name a few). If you begin your day with 100 items,
have a profit of $10 per item, and an overhead of $30 per day, you know exactly how much
profit you will make that day, namely 100(10)-30=$970. Similarly, if you begin the day with 50
items, you can also state your profits with certainty. In fact for any number of items you begin
the day with (x), you can state what the day’s profits (y) will be. That is, y = 10 · x − 30. This is
called a deterministic model. In general, we can write the equation for a straight line as y = β0 +
β1x,

In statistics, linear regression is a statistical model which estimates the linear relationship
between a scalar response and one or more explanatory variables (also known as dependent and
independent variables). The case of one explanatory variable is called simple linear regression;
for more than one, the process is called multiple linear regression.[1] This term is distinct
from multivariate linear regression, where multiple correlated dependent variables are predicted,
rather than a single scalar variable.[2] If the explanatory variables are measured with error
then errors-in-variables models are required, also known as measurement error models.
In linear regression, the relationships are modeled using linear predictor functions whose
unknown model parameters are estimated from the data. Such models are called linear models.
[3]
Most commonly, the conditional mean of the response given the values of the explanatory
variables (or predictors) is assumed to be an affine function of those values; less commonly, the
conditional median or some other quantile is used. Like all forms of regression analysis, linear
regression focuses on the conditional probability distribution of the response given the values of
the predictors, rather than on the joint probability distribution of all of these variables, which is
the domain of multivariate analysis.
Linear regression was the first type of regression analysis to be studied rigorously, and to be used
extensively in practical applications.[4] This is because models which depend linearly on their
unknown parameters are easier to fit than models which are non-linearly related to their
parameters and because the statistical properties of the resulting estimators are easier to
determine.
Linear regression has many practical uses. Most applications fall into one of the following two
broad categories:

 If the goal is error i.e variance reduction in prediction or forecasting, linear regression can
be used to fit a predictive model to an observed data set of values of the response and
explanatory variables. After developing such a model, if additional values of the explanatory
variables are collected without an accompanying response value, the fitted model can be
used to make a prediction of the response.
 If the goal is to explain variation in the response variable that can be attributed to
variation in the explanatory variables, linear regression analysis can be applied to quantify
the strength of the relationship between the response and the explanatory variables, and in
particular to determine whether some explanatory variables may have no linear relationship
with the response at all, or to identify which subsets of explanatory variables may contain
redundant information about the response.

Linear regression models are often fitted using the least squares approach, but they may also be
fitted in other ways, such as by minimizing the "lack of fit" in some other norm (as with least
absolute deviations regression), or by minimizing a penalized version of the least squares cost
function as in ridge regression (L2-norm penalty) and lasso (L1-norm penalty). Use of the Mean
Squared Error(MSE) as the cost on a dataset that has many large outliers, can result in a model
that fits the outliers more than the true data due to the higher importance assigned by MSE to
large errors. So, cost functions that are robust to outliers should be used if the dataset has many
large outliers. Conversely, the least squares approach can be used to fit models that are not linear
models. Thus, although the terms "least squares" and "linear model" are closely linked, they are
not synonymous.
Formulation[edit]

In linear regression, the observations (red) are assumed to be

the result of random deviations (green) from an underlying relationship (blue) between a
dependent variable (y) and an independent variable (x).

Given a data set of n statistical units, a linear regression model assumes that the
relationship between the dependent variable y and the vector of regressors x is linear. This
relationship is modeled through a disturbance term or error variable ε — an unobserved random
variable that adds "noise" to the linear relationship between the dependent variable and
regressors. Thus the model takes the form

where T denotes the transpose, so that xiTβ is the inner product between vectors xi and β.

Often these n equations are stacked together and written in matrix notation as

where

Least Square Method

 Lesson5

The least square method is the process of finding the best-fitting curve or line of best fit for a
set of data points by reducing the sum of the squares of the offsets (residual part) of the points
from the curve. During the process of finding the relation between two variables, the trend of
outcomes are estimated quantitatively. This process is termed as regression analysis. The
method of curve fitting is an approach to regression analysis. This method of fitting equations
which approximates the curves to given raw data is the least squares.

It is quite obvious that the fitting of curves for a particular data set are not always unique. Thus,
it is required to find a curve having a minimal deviation from all the measured data points. This
is known as the best-fitting curve and is found by using the least-squares method.

Least Square Method Definition

The least-squares method is a crucial statistical method that is practised to find a regression line
or a best-fit line for the given pattern. This method is described by an equation with specific
parameters. The method of least squares is generously used in evaluation and regression. In
regression analysis, this method is said to be a standard approach for the approximation of sets of
equations having more equations than the number of unknowns.

The method of least squares actually defines the solution for the minimization of the sum of
squares of deviations or the errors in the result of each equation. Find the formula for sum of
squares of errors, which help to find the variation in observed data.

The least-squares method is often applied in data fitting. The best fit result is assumed to reduce
the sum of squared errors or residuals which are stated to be the differences between the
observed or experimental value and corresponding fitted value given in the model.

There are two basic categories of least-squares problems:

 Ordinary or linear least squares

 Nonlinear least squares

These depend upon linearity or nonlinearity of the residuals. The linear problems are often seen
in regression analysis in statistics. On the other hand, the non-linear problems are generally used
in the iterative method of refinement in which the model is approximated to the linear one with
each iteration.

Least Square Method Graph

In linear regression, the line of best fit is a straight line as shown in the following diagram:

The given data points are to be minimized by the method of reducing residuals or offsets of each
point from the line. The vertical offsets are generally used in surface, polynomial and hyperplane
problems, while perpendicular offsets are utilized in common practice.

Least Square Method Formula

The least-square method states that the curve that best fits a given set of observations, is said to
be a curve having a minimum sum of the squared residuals (or deviations or errors) from the
given data points. Let us assume that the given points of data are (x 1, y1), (x2, y2), (x3, y3), …, (xn,
yn) in which all x’s are independent variables, while all y’s are dependent ones. Also, suppose
that f(x) is the fitting curve and d represents error or deviation from each given point.

Now, we can write:

d1 = y1 − f(x1)

d2 = y2 − f(x2)

d3 = y3 − f(x3)

…..

dn = yn – f(xn)

The least-squares explain that the curve that best fits is represented by the property that the sum
of squares of all the deviations from given values must be minimum, i.e:

Sum = Minimum Quantity

Suppose when we have to determine the equation of line of best fit for the given data, then we
first use the following formula.

The equation of least square line is given by Y = a + bX

Normal equation for ‘a’:

∑Y = na + b∑X

Normal equation for ‘b’:

∑XY = a∑X + b∑X2

Solving these two normal equations we can get the required trend line equation.

Thus, we can get the line of best fit with formula y = ax + b

Solved Example

The Least Squares Model for a set of data (x1, y1), (x2, y2), (x3, y3), …, (xn, yn) passes through the
point (xa, ya) where xa is the average of the x i‘s and ya is the average of the y i‘s. The below
example explains how to find the equation of a straight line or a least square line using the least
square method.

Question:

Consider the time series data given below:

xi 8 3 2 10 11 3 6 5 6 8

yi 4 12 1 12 9 4 9 6 1 14

Use the least square method to determine the equation of line of best fit for the data. Then plot
the line.

Solution:

Mean of xi values = (8 + 3 + 2 + 10 + 11 + 3 + 6 + 5 + 6 + 8)/10 = 62/10 = 6.2

Mean of yi values = (4 + 12 + 1 + 12 + 9 + 4 + 9 + 6 + 1 + 14)/10 = 72/10 = 7.2

Straight line equation is y = a + bx.

The normal equations are

∑y = an + b∑x

∑xy = a∑x + b∑x2

x y x2 xy

8 4 64 32

3 12 9 36

2 1 4 2
10 12 100 120

11 9 121 99

3 4 9 12

6 9 36 54

5 6 25 30

6 1 36 6

Second-Order Exponential Smoothing Forecast

Intended learning outcomes: Disclose the determination of trend lines in second-order

exponential smoothing. Explain the formulas for calculation of the trend line and forecast

error in second-order exponential smoothing. Present an example of determination of forecast

value using second-order exponential smoothing.

Second-order exponential smoothing forecast technique extends first-order exponential

smoothing to create a technique capable of capturing linear trend.

Second-order exponential smoothing starts out from:

 The mean, calculated using first-order smoothing

 The mean of this first-order means, calculated according to the same recursion formula

Figure shows the formulas necessary for calculating the trend line; this gives us the second-order
forecast value for subsequent periods as well as the corresponding forecast error. See also the
definitions in Figure [Link].

The following numbered explanations correspond to those presented in Figure:

1. The previous formula to determine first-order mean.
2. The new formula to determine the second-order mean, as the mean of the first-order
means. The second-order mean lies at the same distance from the first-order mean as does
the latter from the current period.

3. Slope of the trend line to time t, when two means are given.

4. Starting value Tt for the forecast at time t.

5. Forecast for subsequent periods.

6. Forecast error for the next period t + 1. Because a linear trend entails that the forecast
error is dependent on k, the same formula does not automatically hold for period t + k,
although it is often used.

7. The determination of the starting value that can be calculated, for example, by means of
regression analysis.
 Lesson6

Models with Second Order Exponential Smoothing

Definition

Second-order exponential smoothing is used in forecast strategies 22 and 23. It is based on a

linear trend and consists of two equations. The first equation corresponds to that of first-order
exponential smoothing except for the bracketed indices. In the second equation, the values
calculated in the first equation are used as initial values and are smoothed again.

Formulas for Second-Order Exponential Smoothing

For more information on strategy 23, see Automatic Adaptation of the Alpha Factor.
Use

If, over several periods, a time series shows a change in the average value such that a trend
pattern is revealed, first-order exponential smoothing produces forecast values that lag behind
the actual values by one or several periods. You can achieve a more efficient adjustment of the
forecast to the actual values pattern by using second-order exponential smoothing.

Exponential Smoothing

Exponential smoothing was first suggested in the statistical literature without reference to
previous work by Robert Goodell Brown in 1956 and then expanded by Charles C. Holt in 1957.
Exponential smoothing is a broadly accurate principle for smoothing time series data using the
exponential window function. The controlling input of the exponential smoothing calculation is
defined as the smoothing factor or the smoothing constant.

As we know that, in the simple moving average, the past observations are weighted
equally, exponential functions are used to assign exponentially decreasing weights over time. It
is an easily learned and easily applied method for making some determination based on prior
assumptions by the user, such as seasonality. Exponential smoothing is generally used for the
analysis of time-series data.

Exponential Smoothing Formula

The simplest form of an exponential smoothing formula is given by:

st = αxt+(1 – α)st-1= st-1+ α(xt – st-1)

Here,

st = smoothed statistic, it is the simple weighted average of current observation xt

st-1 = previous smoothed statistic

α = smoothing factor of data; 0 < α < 1

t = time period

If the value of the smoothing factor is larger, then the level of smoothing will reduce. Value of α
close to 1 has less of a smoothing effect and give greater weight to recent changes in the data,
while the value of α closer to zero has a greater smoothing effect and are less responsive to
recent changes.

There is no official accurate procedure for choosing α. The statistician’s judgment is used to
choose an appropriate factor sometimes. Otherwise, a statistical technique may be used to
optimize the value of α. For example, the method of least squares can be used to determine the
value of α for which the sum of the quantities is minimized.

Exponential Smoothing Forecasting

Exponential smoothing is generally used to make short term forecasts, but longer-term forecasts
using this technique can be quite unreliable. More recent observations given larger weights by
exponential smoothing methods, and the weights decrease exponentially as the observations
become more distant. When the parameters describing the time series are changing slowly over
time, then these methods are most effective.

Exponential Smoothing Methods

There are three main methods to estimate exponential smoothing. They are:

 Simple or single exponential smoothing

 Double exponential smoothing
 Triple exponential smoothing
Simple or single exponential smoothing

If the data has no trend and no seasonal pattern, then this method of forecasting the time series is
essentially used. This method uses weighted moving averages with exponentially decreasing
weights.

The single exponential smoothing formula is given by:

st = αxt+(1 – α)st-1 = st-1 + α(xt – st-1)

Double exponential smoothing

This method is also called as Holt’s trend corrected or second-order exponential smoothing. This
method is used for forecasting the time series when the data has a linear trend and no seasonal
pattern. The primary idea behind double exponential smoothing is to introduce a term to take into
account the possibility of a series showing some form of trend. This slope component is itself
updated through exponential smoothing.

The double exponential smoothing formulas are given by:

S1 = x1

B1 = x1-x0

For t>1,

st = αxt + (1 – α)(st-1 + bt-1)

βt = β(st – st-1) + (1 – β)bt-1

Here,

st = smoothed statistic, it is the simple weighted average of current observation xt

st-1 = previous smoothed statistic

α = smoothing factor of data; 0 < α < 1

t = time period

bt = best estimate of trend at time t

β = trend smoothing factor; 0 < β <1

Triple exponential smoothing

In this method, exponential smoothing applied three times. This method is used for forecasting
the time series when the data has both linear trend and seasonal pattern. This method is also
called Holt-Winters exponential smoothing.

The triple exponential smoothing formulas are given by:

Here,

st = smoothed statistic, it is the simple weighted average of current observation xt

st-1 = previous smoothed statistic

α = smoothing factor of data; 0 < α < 1

t = time period

bt = best estimate of a trend at time t

β = trend smoothing factor; 0 < β <1

ct = sequence of seasonal correction factor at time t

γ = seasonal change smoothing factor; 0 < γ < 1

Example problem

The sales of a magazine in a stall for the previous 10 months are given below.

Month Sales

January 30

February 25

March 35

April 25

May 20

June 30

July 35

August 40

September 30

October 45

Calculate the simple exponential smoothing taking α =0.3 for the above data.

Solution:

Month Sales Exponential smooth

 α =0.3
January 30 30.00

February 25 30.00

March 35 28.50

April 25 30.45

May 20 14.1

June 30 15.87

July 35 20.109

August 40 24.5763

September 30 29.20341

October 45 29.442387

November – 34.1096709

Read more:

*
An Introduction to Exponential Smoothing for Time Series Forecasting

Exponential smoothing is a widely preferred statistical technique to forecast a time series. We

use this simple yet powerful forecasting method for smoothing univariate time series data by
using the exponential window function.
Lecture7

What Is Exponential Smoothing?

Exponential smoothing is a time series method for forecasting univariate time series data. Time
series methods work on the principle that a prediction is a weighted linear sum of past
observations or lags. The Exponential Smoothing time series method works by assigning
exponentially decreasing weights for past observations. It is called so because the weight
assigned to each demand observation is exponentially decreased.

The model assumes that the future will be somewhat the same as the recent past. The only
pattern that Exponential Smoothing learns from demand history is its level - the average value
around which the demand varies over time.

Exponential smoothing is generally used to make forecasts of time-series data based on prior
assumptions by the user, such as seasonality or systematic trends.

Exponential Smoothing Forecasting

Exponential smoothing is a broadly accurate forecasting method for short-term forecasts. The
technique assigns larger weights to more recent observations while assigning exponentially
decreasing weights as the observations get increasingly distant. This method produces slightly
unreliable long-term forecasts.

Exponential smoothing can be most effective when the time series parameters vary slowly over
time.
Types of Exponential Smoothing

The main types of Exponential Smoothing forecasting methods are:

1. Simple or Single Exponential Smoothing

Simple or single exponential smoothing (SES) is the method of time series forecasting used with
univariate data with no trend and no seasonal pattern. It needs a single parameter called alpha (a),
also known as the smoothing factor. Alpha controls the rate at which the influence of past
observations decreases exponentially. The parameter is often set to a value between 0 and 1.

The simple exponential smoothing formula is given by:

st = αxt+(1 – α)st-1= st-1+ α(xt – st-1)

here,

st = smoothed statistic (simple weighted average of current observation xt)

st-1 = previous smoothed statistic

α = smoothing factor of data; 0 < α < 1

t = time period

2. Double Exponential Smoothing

This method is known as Holt's trend model or second-order exponential smoothing. Double
exponential smoothing is used in time-series forecasting when the data has a linear trend but no
seasonal pattern. The basic idea here is to introduce a term that can consider the possibility of the
series exhibiting some trend.

In addition to the alpha parameter, Double exponential smoothing needs another smoothing
factor called beta (b), which controls the decay of the influence of change in trend. The method
supports trends that change in additive ways (smoothing with linear trend) and trends that change
in multiplicative ways (smoothing with exponential trend).

The Double exponential smoothing formulas are:

S1 = x1

B1 = x1-x0

For t>1,

st = αxt + (1 – α)(st-1 + bt-1)

βt = β(st – st-1) + (1 – β)bt-1

here,

bt = best estimate of the trend at time t

β = trend smoothing factor; 0 < β <1

3. Triple Exponential Smoothing

This method is the variation of exponential smoothing that's most advanced and is used for time
series forecasting when the data has linear trends and seasonal patterns. The technique applies
exponential smoothing three times – level smoothing, trend smoothing, and seasonal smoothing.
A new smoothing parameter called gamma (g) is added to control the influence of the seasonal
component.

The triple exponential smoothing method is called Holt-Winters Exponential Smoothing, named
after its contributors, Charles Holt and Peter Winters.

Holt-Winters Exponential Smoothing has two categories depending on the nature of the seasonal
component:

 Holt-Winter's Additive Method − for seasonality that is addictive.

 Holt-Winter's Multiplicative Method – for seasonality that is multiplicative.
 How to Configure Exponential Smoothing

To configure Exponential Smoothing, analysts need to specify all the model hyperparameters
explicitly. However, this can be challenging for both beginners and experts.

Instead, numerical optimization is commonly used to search for and fund the smoothing factors
(alpha, beta, gamma, phi) for the model resulting in the most negligible error.

An exponential smoothing method can obtain values for unknown parameters by estimating them
from the observed data. The initial values and unknown parameters can be estimated by
minimizing the sum of the squared errors (SSE).

The parameters that indicate the kind of change in trend or seasonality (for example, whether
they are additive or multiplicative or whether they should be dampened) need to be specified
explicitly.

Exponential Smoothing in Python

Let us look at how to implement exponential smoothing in Python.

The Statsmodels Python library provides the implementations of Exponential Smoothing in

Python.

Single Exponential Smoothing

The SimpleExpSmoothing Statsmodels class enables implementation of Single Exponential

Smoothing or simple smoothing in Python.

First, an instance of SimpleExpSmoothing is instantiated and passed the training data. Next, the
fit() function is called, giving the fit configuration, especially the alpha value. The fit() function
returns an instance of the HoltWintersResults class containing the learned coefficients. The
forecast() or the predict() function is then called on the result object to make a forecast.
Double and Triple Exponential Smoothing

The SimpleExpSmoothing Statsmodels class also enables the implementation of Double and
Triple Exponential Smoothing in Python.

First, an instance of SimpleExpSmoothing is instantiated, specifying training data and model

configuration. We must define the configuration parameters for trend, damped, seasonal, and
seasonal_periods. The fit() function is then called to fit the model on the training data.

The fit() function returns an instance of the HoltWintersResults class containing the learned
coefficients. The forecast() or the predict() function is then called on the result object to make a
forecast.
Simple exponential smoothing

The simplest of the exponentially smoothing methods is naturally called simple exponential
smoothing (SES)14. This method is suitable for forecasting data with no clear trend or
seasonal pattern. For example, the data in Figure 7.1 do not display any clear trending
behaviour or any seasonality. (There is a rise in the last few years, which might suggest a
trend. We will consider whether a trended method would be better for this series later in this
chapter.) We have already considered the naïve and the average as possible methods for
forecasting such data (Section 3.1).

oildata <- window(oil, start=1996)

autoplot(oildata) +
ylab("Oil (millions of tonnes)") + xlab("Year")

Figure : Oil production in Saudi Arabia from 1996 to 2013.