Time Series Components Trend
•Trend: the long-term direction.
•Seasonality: the periodic behavior.
•Residuals: the irregular fluctuations.
Image source:[Link]
stationarity-with-time-series-data-abd59fd8d5a0
Types of Decomposition
Additive Decomposition Multiplicative Decomposition
Image source:[Link]
Stationarity
A time series is stationary when there are no changes in the underlying system that generates it.
In the following, we define:
Image source:[Link]
Types of Stationarity
Image source: [Link]
Testing of Stationarity
Augmented Dickey-Fuller test (ADF):
Null Hypotehsis (H0): If accepted, it suggests the time series has a unit root, meaning it
is non-stationary. It has some time dependent structure.
Alternate Hypothesis (H1): The null hypothesis is rejected; it suggests the time series
does not have a unit root, meaning it is stationary.
p-value > 0.05: Accept H0, the data has a unit root and is non-stationary
p-value ≤ 0.05: Reject H0. the data does not have a unit root and is stationary
White Noise and Random Noise
• White noise is a purely • A random walk is a time series where
random sequence of values with no each value depends on the
pattern, trend, or dependence on previous one plus some white
past values. noise.
• Appears in residual analysis of • The mean and
time series models (e.g., ARIMA). variance increase over time.
Image credit: [Link]
AR Model
• An Autoregressive (AR) model is a type of time series model that uses
observations from previous time steps as input to a regression equation to predict
the value at the next time step.
• The AR model is dependent solely on its own past values.
• The general form of an AR model of order is:
AR(1) Model
• The first-order autoregressive model, AR(1), is
Higher order AR models
• Higher order AR models (AR(2), AR(3), etc.) depend on more than one past
value.
• For example, the AR(2) model is:
• These models are useful in scenarios where the effect of more than one previous
time step is significant.
MA Model
• Another approach to modeling univariate time series is the
moving average (MA) model.
• The MA model is a linear regression of the current value of the
series against the white noise of one or more of the previous
values of the series.
• The noise at each point is assumed to come from a normal
distribution with mean 0 and constant variance.
• The MA model is defined as:
MA(2) Model
Time(t) Actual value() Forecastedy() Error term(=-)
1 10 --
2 12 --
3 13 0+(0.6*0)+(0.3*0)=0 13-0=13
4 15 0+(0.6*13)+(0.3*0)=7.8 15-7.8=7.2
5 14 0+(0.6*7.2)+(0.3*13)=8.82 14-8.82=5.18
6 16 0+(0.6*5.18)+(0.3*7.2)=5.86 16-5.86=10.13
7 18 0+(0.6*10.13)+(0.3*7.2)=8.46 18-8.46=9.54
ARMA(p,q) Model
The ARMA model (also known as the Box-Jenkins approach)
combines two models:
• An autoregressive (AR) model of order p.
• A moving average (MA) model of order q.
• ARMA models assume the time series is stationary.
ARIMA(p,d,q) Model
The ARIMA model contains:
• An autoregressive (AR) model of order p.
• A moving average (MA) model of order q.
• Adds d-order differencing to ARMA.
SARIMA Model
• To apply ARMA and ARIMA, we must remove the seasonal
component.
• After computing the predictions we had to put the
seasonal component back.
• It would be convenient to directly work on data with
seasonality.
• SARIMA is an extension of ARIMA that includes seasonal
terms
• The model is specified as SARIMA(P,D,Q,s)
•p - the order of the autoregressive trend
•d - the order of the trend differencing
•q - the order of the moving average trend
What do (P,D,Q)s mean?
•P - the order of the autoregressive seasonality
•D - the order of the seasonal differncing
•Q - the order of the moving average seasonality
•s - the number of periods in your season
Evolution of Time series Models
Models Significance Limitation Example
AR • AR models assume that the current Cannot handle random Stock prices where today’s
value ytdepends on its past values shocks in data effectively. price depends on previous
• Suitable for stationary time series with days.
strong autocorrelation.
MA • Instead of past values, it Cannot capture long-term Weather forecasting, where
models dependence on past errors (ϵt) trends or dependencies in today’s temperature is
• Helps smooth out fluctuations and model past observations. influenced by random past
random variations. variations.
ARMA • ARMA combines both AR and MA Only works Economic indicators (like
components, capturing both trends and for stationary series inflation), which depend on
random shocks past values and external
shocks.
ARIMA • Introduces differencing (d) to make data Cannot model seasonal Sales forecasting, where
stationary. effects sales increase over time.
SARIMA • Adds seasonal differencing (D) More complex to tune Retail sales, electricity
and seasonal AR/MA terms (choosing p,d,q,P,D,Q,s requi demand, and temperature
• Ideal for data with seasonal patterns res expertise). forecasting.
A local learning approach for sequential hybrid systems in time series
forecasting
Training stage
• The first step is the generation of linear forecasts for the time series 𝑍𝑡 using an ARIMA model,
represented by
• The second step receives the residual of the linear model and performs a nonlinear prediction
using an SVR
• These are stored during the training stage into a matrix called M as arrays in the form of [feature
1, feature 2, target]
Image source : [Link]
Prediction stage
The steps required to perform a prediction of each incoming data pattern are four:
1. linear time series prediction
2. Nonlinear prediction of the residual
3. region of competence creation
4. Local model selection using (3) to combine (1) and (2).
Image source : [Link]
Image source : [Link]
Experiments Results
Models MSE
LSTM 210.221
Transformer 229.035
LSTM+Attention 260.121
TCN 6788.29
TCN+LSTM 6535.42
Patching 6919.83
GRU 6379.37
BiLSTM 6346.16