A Comparison of ARIMA and LSTM in Forecasting Time Series

Sima Siami-Namini Neda Tavakoli Akbar Siami Namin

Department of Applied Economics Department of Computer Science Department of Computer Science

Texas Tech University Georgia Institute of Technology Texas Tech University
Email: [Link]-namini@[Link] Email: [Link]@[Link] Email: [Link]@[Link]

Abstract—Forecasting time series data is an important subject (VAR) models are the other most popular forecasting models,
in economics, business, and finance. Traditionally, there are which in turn, generalize the univariate ARIMA models and
several techniques to effectively forecast the next lag of time univariate autoregressive (AR) model by allowing for more
series data such as univariate Autoregressive (AR), univariate
Moving Average (MA), Simple Exponential Smoothing (SES), than one evolving variable.
and more notably Autoregressive Integrated Moving Average Machine learning techniques and more importantly deep
(ARIMA) with its many variations. In particular, ARIMA model learning algorithms have introduced new approaches to pre-
has demonstrated its outperformance in precision and accuracy diction problems where the relationships between variables are
of predicting the next lags of time series. With the recent advance-
ment in computational power of computers and more importantly modeled in a deep and layered hierarchy. Machine learning-
development of more advanced machine learning algorithms and based techniques such as Support Vector Machines (SVM) and
approaches such as deep learning, new algorithms are developed Random Forests (RF) and deep learning-based algorithms such
to analyze and forecast time series data. The research question as Recurrent Neural Network (RNN), and Long Short-Term
investigated in this article is that whether and how the newly Memory (LSTM) have gained lots of attentions in recent years
developed deep learning-based algorithms for forecasting time
series data, such as “Long Short-Term Memory (LSTM)”, are with their applications in many disciplines including finance.
superior to the traditional algorithms. The empirical studies Deep learning methods are capable of identifying structure
conducted and reported in this article show that deep learning- and pattern of data such as non-linearity and complexity in
based algorithms such as LSTM outperform traditional-based time series forecasting. In particular, LSTM has been used in
algorithms such as ARIMA model. More specifically, the average time-series prediction [4], [8], [9], [18] and in economics and
reduction in error rates obtained by LSTM was between 84 - 87
percent when compared to ARIMA indicating the superiority of finance data such as predicting the volatility of the S&P 500
LSTM to ARIMA. Furthermore, it was noticed that the number [10].Many other Computer Science-based problems can also
of training times, known as “epoch” in deep learning, had no be formulated and analyzed using time series, such scheduling
effect on the performance of the trained forecast model and it I/O in a client-server architecture [23].
exhibited a truly random behavior.
An interesting and important research question is then the
I. I NTRODUCTION accuracy and precision of traditional forecasting techniques
when compared to deep learning-based forecasting algorithms.
Prediction of time series data is a challenging task mainly To the best of our knowledge, there is no specific empirical
due to the unprecedented changes in economic trends and con- evidence for using LSTM method in forecasting economic
ditions in one hand and incomplete information on the other and financial timer series data to assess its performance and
hand. Market volatility in recent years has introduced serious compare it with traditional econometric forecasting methods
concerns for economic and financial time series forecasting. such as ARIMA.
Therefore, assessing the accuracy of forecasts is necessary
This paper compares ARIMA and LSTM models with
when employing various forms of forecasting methods, and
respect to their performance in reducing error rates. As a
more specifically forecasting using regression analysis as they
representative of traditional forecast modeling, ARIMA is
have several limitations in applications.
chosen due to the non-stationary property of the data collected
The main objective of this article is to investigate which
and modeled. In an analogous way and as a representative of
forecasting methods offer best predictions with respect to
deep learning-based algorithms, LSTM method is used due to
lower forecast errors and higher accuracy of forecasts. In
its use in preserving and training the features of given data
this regard, there are varieties of stochastic models in time
for a longer period of time. The paper provides an in-depth
series forecasting. The most well known method is univariate
guidance on data processing and training of LSTM models
“Auto-Regressive Moving Average (ARMA)” for a single time
for a set of economic and financial time series data. The key
series data in which Auto-Regressive (AR) and Moving Aver-
contributions of this paper are:
age (MA) models are combined. Univariate “Auto-Regressive
Integrated Moving Average (ARIMA)” is a special type of – Conduct an empirical study and analysis with the goal of
ARIMA where differencing is taken into account in the model. investigating the performance of traditional forecasting
Multivariate ARIMA models and Vector Auto-Regression techniques and deep learning-based algorithms.
 – Compare the performance of LSTM and ARIMA with deep learning-based approaches have gained popularity among
respect to minimization achieved in the error rates in researchers. For instance, Krauss et al. [15] use various forms
prediction. The study shows that LSTM outperforms of forecasting models such as deep learning, gradient-boosted
ARIMA. The average reduction in error rates obtained trees, and random forests to model S&P 500 constitutes.
by LSTM is between 84 - 87 percent when compared to Surprisingly, they reported that deep learning-based modeling
ARIMA indicating the superiority of LSTM. under-performed gradient-boosted trees and random forests.
– Investigate the influence of the number of training times. Additionally, Krauss et al. report that training neural networks
The study shows that the number of training performed and consequently deep learning-based algorithms is very dif-
on data, known as “epoch” in deep learning, has no effect ficult. Lee and Yoo [16] introduced an RNN-based approach
on the performance of the trained forecast model and it to predict stock returns. The idea was to build portfolios by
exhibits a truly random behavior. adjusting the threshold levels of return by internal layers of
The article is organized as follows. Section II outlines the the RNN built. Similar work is performed by Fischera et al.
state of the art of time series forecasting. Section III discusses [7] for financial market prediction. In this article, we compare
the mathematical background of ARIMA and LSTM. Section the performance of an ARIMA model with the LSTM model
IV describes an experimental study for ARIMA versus LSTM in the prediction of economics and financial time series to
model. The ARIMA and LSTM algorithms developed and determine the optimal qualities of involved variables in a
compared are presented in Section V. The results of data typical prediction model.
analysis and empirical results are presented in Section VI.
III. BACKGROUND
Section VII discusses the impact of the number of iterations
on fitting model. Finally, Section VIII concludes the paper. This section reviews the mathematical background of the
compared time series techniques used and studied in this arti-
II. T IME S ERIES F ORECASTING : S TATE - OF - THE -A RT cle. More specifically, the background knowledge of Autore-
Time series analysis and dynamic modeling is an interesting gressive Integrated Moving Average (ARIMA) and the deep
research area with a great number of applications in business, learning-based technique, Long Short-Term Memory (LSTM)
economics, finance and computer science. The aim of time is presented.
series analysis is to study the path observations of time
series and build a model to describe the structure of data A. ARIMA
and then predict the future values of time series. Due to the ARIMA combines Autoregressive (AR) process and Mov-
importance of time series forecasting in many branches of ing Average (M A) processes and builds a composite model
applied sciences, it is essential to build an effective model of the time series. As acronym indicates, ARIM A(p, d, q)
with the aim of improving the forecasting accuracy. A variety captures the key elements of the model:
of the time series forecasting models have been evolved in the – AR: Autoregression. A regression model that uses the
literature. dependencies between an observation and a number of
Time series forecasting is traditionally performed in econo- lagged observations (p).
metric using ARIMA models, which is generalized by Box – I: Integrated. To make the time series stationary by
and Jenkins [5]. ARIMA has been a standard method for measuring the differences of observations at different
time series forecasting for a long time. Even though ARIMA time (d).
models are very prevalent in modeling economical and fi- – M A: Moving Average. An approach that takes into
nancial time series [1], [2], [14], they have some major accounts the dependency between observations and the
limitations [6]. For instance, in a simple ARIMA model, it is residual error terms when a moving average model is
hard to model the nonlinear relationships between variables. used to the lagged observations (q).
Furthermore, it is assumed that there is a constant standard A simple form of an AR model of order p, i.e., AR(p), can
deviation in errors in ARIMA model, which in practice it may be written as a linear process given by:
not be satisfied. When an ARIMA model is integrated with X p

a Generalized Auto-regressive Conditional Heteroskedasticity xt = c + φi xt−i + t (1)

(GARCH) model, this assumption can be relaxed. On the other i=1
hand, the optimization of an GARCH model and its parameters Where xt is the stationary variable, c is constant, the terms
might be challenging and problematic [13]. There are several in φi are autocorrelation coefficients at lags 1, 2, , p and t ,
other applications of ARIMA for modeling short and long run the residuals, are the Gaussian white noise series with mean
Effects of economics parameters [19]–[22]. zero and variance σ2 . An MA model of order q, i.e., M A(q),
Recently, new techniques in deep learning have been de- can be written in the form:
q
veloped to address the challenges related to the forecasting X
models. LSTM (Long Short-Term Memory) is a special case xt = µ + θi t−i (2)
of Recurrent Neural Network (RNN) method that was initially i=0

introduced by Hochreiter and Schmidhuber [9]. Even though Where µ is the expectation of xt (usually assumed equal
it is a relatively new approach to address prediction problems, to zero), the θi terms are the weights applied to the current
and prior values of a stochastic term in the time series, and of a decision maker to decide which signal, or input, may
θ0 = 1. We assume that t is a Gaussian white noise series pass through and which may not. The weights also show
with mean zero and variance σ2 . We can combine these two the strength or extent to the hidden layer. A neural network
models by adding them together and form an ARIMA model basically learns by adjusting the weight for each synopsis.
of order (p, q): In the hidden layers, the nodes apply an activation function
p q
X X (e.g., sigmoid or tangent hyperbolic (tanh)) on the weighted
xt = c + φi xt−i + t + θi t−i (3) sum of inputs to transform the inputs to the outputs, or
i=1 i=0
predicted values. The output layer generates a vector of
Where φi 6= 0, θi 6= 0, and σ2> 0. The parameters p and q probabilities for the various outputs and selects the one with
are called the AR and M A orders, respectively. ARIMA fore- minimum error rate or cost, i.e., minimizing the differences
casting, also known as Box and Jenkins forecasting, is capable between expected and predicted values, also known as the cost,
of dealing with non-stationary time series data because of its using a function called SoftMax.
“integrate” step. In fact, the “integrate” component involves The assignments to the weights vector and thus the errors
differencing the time series to convert a non-stationary time obtained through the network training for the first time might
series into a stationary. The general form of a ARIMA model not be the best. To find the most optimal values for errors, the
is denoted as ARIM A(p, d, q). errors are “back propagated” into the network from the output
With seasonal time series data, it is likely that short run non- layer towards the hidden layers and as a result the weights are
seasonal components contribute to the model. Therefore, we adjusted. The procedure is repeated, i.e., epochs, several times
need to estimate seasonal ARIMA model, which incorporates with the same observations and the weights are re-adjusted
both non-seasonal and seasonal factors in a multiplicative until there is an improvement in the predicted values and
model. The general form of a seasonal ARIMA model is subsequently in the cost. When the cost function is minimized,
denoted as ARIM A(p, d, q) × (P, D, Q)S, where p is the the model is trained.
non-seasonal AR order, d is the non-seasonal differencing, q 2) Recurrent Neural Network (RNN): A recurrent neural
is the non-seasonal M A order, P is the seasonal AR order, D network (RNN) is a special case of neural network where
is the seasonal differencing, Q is the seasonal M A order, and the objective is to predict the next step in the sequence of
S is the time span of repeating seasonal pattern, respectively. observations with respect to the previous steps observed in
The most important step in estimating seasonal ARIMA the sequence. In fact, the idea behind RNNs is to make use
model is to identify the values of (p, d, q) and (P, D, Q). of sequential observations and learn from the earlier stages to
Based on the time plot of the data, if for instance, the variance forecast future trends. As a result, the earlier stages data need
grows with time, we should use variance-stabilizing transfor- to be remembered when guessing the next steps. In RNNs, the
mations and differencing. Then, using autocorrelation function hidden layers act as internal storage for storing the information
(ACF) to measure the amount of linear dependence between captured in earlier stages of reading sequential data. RNNs
observations in a time series that are separated by a lag p, are called “recurrent” because they perform the same task
and the partial autocorrelation function (PACF) to determine for every element of the sequence, with the characteristic of
how many autoregressive terms q are necessary and inverse utilizing information captured earlier to predict future unseen
autocorrelation function (IACF) for detecting over differenc- sequential data. The major challenge with a typical generic
ing, we can identify the preliminary values of autoregressive RNN is that these networks remember only a few earlier steps
order p, the order of differencing d, the moving average order in the sequence and thus are not suitable to remembering
q and their corresponding seasonal parameters P , D and Q. longer sequences of data. This challenging problem is solved
The parameter d is the order of difference frequency changing using the “memory line” introduced in the Long Short-Term
from non-stationary time series to stationary time series. Memory (LSTM) recurrent network.
3) Long Short-Term Memory (LSTM): LSTM is a special
B. LSTM kind of RNNs with additional features to memorize the se-
Long Short-Term Memory (LSTM) [17] is a kind of Recur- quence of data. The memorization of the earlier trend of the
rent Neural Network (RNN) with the capability of remember- data is possible through some gates along with a memory line
ing the values from earlier stages for the purpose of future use. incorporated in a typical LSTM.
Before delving into LSTM, it is necessary to have a glimpse LSTM is a special kind of RNNs with additional features
of what a neural network looks like. to memorize the sequence of data. Each LSTM is a set of
1) Artificial Neural Network (ANN): A neural network cells, or system modules, where the data streams are captured
consists of at least three layers namely: 1) an input layer, 2) and stored. The cells resemble a transport line (the upper line
hidden layers, and 3) an output layer. The number of features in each cell) that connects out of one module to another one
of the data set determines the dimensionality or the number of conveying data from past and gathering them for the present
nodes in the input layer. These nodes are connected through one. Due to the use of some gates in each cell, data in each cell
links called “synapses” to the nodes created in the hidden can be disposed, filtered, or added for the next cells. Hence,
layer(s). The synapses links carry some weights for every the gates, which are based on sigmoidal neural network layer,
node in the input layer. The weights basically play the role enable the cells to optionally let data pass through or disposed.
 Each sigmoid layer yields numbers in the range of zero and TABLE I
one, depicting the amount of every segment of data ought to T HE NUMBER OF TIME SERIES OBSERVATIONS .
be let through in each cell. More precisely, an estimation of Stock Observations Total
zero value implies that “let nothing pass through”; whereas; an Train 70% Test 30%
estimation of one indicates that “let everything pass through.” N225 283 120 403
IXIC 391 167 558
Three types of gates are involved in each LSTM with the goal HSI 258 110 368
of controlling the state of each cell: GSPC 568 243 811
DJI-Monthly 274 117 391
– Forget Gate outputs a number between 0 and 1, where DJI-Weekly 1,189 509 1,698
1 shows “completely keep this”; whereas, 0 implies MC 593 254 847
“completely ignore this.” HO 425 181 606
ER 375 160 535
– Memory Gate chooses which new data need to be stored FB 425 181 606
in the cell. First, a sigmoid layer, called the “input door MS 492 210 702
layer” chooses which values will be modified. Next, a TR 593 254 847
tanh layer makes a vector of new candidate values that B. Data Preparation
could be added to the state.
– Output Gate decides what will be yield out of each cell. Each financial time series data set features a number of vari-
The yielded value will be based on the cell state along ables: Open, High, Low, Close, Adjusted Close and Volume.
with the filtered and newly added data. The authors chose the “Adjusted Close” variable as the only
feature of financial time series to be fed into the ARIMA and
IV. ARIMA VS . LSTM: A N E XPERIMENTAL S TUDY LSTM models. Each economic and financial time series data
With the goal of comparing the performance of ARIMA set was split into two subsets: training and test datasets where
and LSTM, the authors conducted a series of experiments on 70% of each dataset was used for training and the remaining
some selected economic and financial time series data. The 30% of each dataset was used for testing the accuracy of
main research questions investigated through this work are as models. Table I lists the number of time series observations
follows: for each dataset.
1) RQ1. Which algorithm, ARIMA or LSTM, performs
more accurate prediction of time series data? C. Assessment Metric
2) RQ2. Does the number of training times in deep The Root-Mean-Square Error (RMSE) is a measure fre-
learning-based algorithms influence the accuracy of the quently used for assessing the accuracy of prediction obtained
trained model? by a model. It measures the differences or residuals between
actual and predicated values. The metric compares prediction
A. Data Set
errors of different models for a particular data and not between
The authors extracted historical monthly financial time se- datasets. The formula for computing RMSE is as follows:
ries from Jan 1985 to Aug 2018 from the Yahoo finance Web-
v
u
u1 X N
site1 . The monthly data included Nikkei 225 index (N225), RM SE = t (xi − x̂i )2 (4)
NASDAQ composite index (IXIC), Hang Seng Index (HIS), N i=1
S&P 500 commodity price index (GSPC), and Dow Jones
industrial average index (DJ). Moreover, the authors also Where N is the total number of observations, xi is the actual
collected monthly economics time series for different time value; whereas, x̂i is the predicated value. The main benefit
periods from the Federal Reserve Bank of St. Louis2 , and of using RMSE is that it penalizes large errors. It also scales
the International Monetary Fund (IMF) Website3 . The data the scores in the same units as the forecast values (i.e., per
included Medical care commodities for all urban consumers, month for this study).
Index 1982-1984=100 for the period of Jan 1967 to July 2017
(MC), Housing for all urban consumers, Index 1982-1984=100 V. T HE A LGORITHMS
for the period of Jan 1967 to July 2017 (HO), the trade- The ARIMA and LSTM algorithms that were developed for
weighted U.S. dollar index in terms of major currencies, Index forecasting the time series are based on “Rolling Forecasting
Mar 1973=100 for the period of Aug 1967 to July 2017 (EX), Origin” [11]. The rolling forecasting origin focuses on a single
Food and Beverages for all urban consumers, Index 1982- forecast, i.e., the next data point to predict, for each data set.
1984=100 for the period of Jan 1967 to July 2017 (FB), M1 This approach uses training sets, each one containing one more
Money Stock, billions of dollars for the period of Jan 1959 to observation than the previous one, one-month look-ahead view
July 2017 (MS), and Transportation for all urban consumers, of the data. There are several variations of rolling forecast [12]:
Index 1982-1984=100 for the period of Jan 1947 to July 2017 – One-step forecasts without re-estimation. The model es-
(TR). timates a single set of training data and then one-step
1 [Link] forecasts are computed on the remaining data sets.
2 [Link] – Multi-step forecasts without re-estimation. Similar to one-
3 [Link] step forecasts when performed for the next multiple steps.
 – Multi-step forecasts with re-estimation. An alternative Listing 1. The developed rolling ARIMA algorithm.
approach where the model is re-fitted at each iteration # Rolling ARIMA
before each forecast is performed. Inputs: series
Outputs: RMSE of the forecasted data
With respect to our data sets, in which there is a dependency # Split data into:
in prior time steps, a rolling forecast is required. An intuitive # 70% training and 30% testing data
and basic way to perform the rolling forecast is to re-build 1. size ← length(series) * 0.70
the model when each time a new observation is added. A 2. train ← series[0...size]
3. test ← series[size...length(size)]
rolling forecast is sometimes called “walk-forward model # Data structure preparation
validation.” Python was used for implementing the algorithms 4. history ← train
along with Keras, an open source neural network library, and 5. predictions ← empty
Theano, a numerical computation library, both for Python. The # Forecast
experiments were executed on a cluster of high performance 6. for each t in range(length(test)) do
7. model ← ARIMA(history, order=(5, 1, 0))
computing facilities. 8. model fit ← [Link]()
9. hat ← model_fit.forecast()
A. The ARIMA Algorithm 10. [Link](hat)
11. observed ← test[t]
ARIMA is a class of models that captures temporal struc-
12. [Link](observed)
tures in time series data. ARIMA is a linear regression-based 13. end for
forecasting approach. Therefore it is best for forecasting one- 14. MSE = mean_squared_error(test,predictions)
step out-of-sample forecast. Here, the algorithm developed 15. RMSE = sqrt(MSE)
performs multi-step out-of-sample forecast with re-estimation, 16. Return RMSE
i.e., each time the model is re-fitted to build the best estimation
model [3]. The algorithm, listed in Listing 1, takes as input B. The LSTM Algorithm
“time series” data set, builds a forecast model and reports
Unlike modeling using regressions, in time series datasets
the root mean-square error of the prediction. The algorithm
there is a sequence of dependence among the input variables.
first splits the given data set into train and test sets, 70%
Recurrent Neural Networks are very powerful in handling the
and 30%, respectively (Lines 1-3). It then builds two data
dependency among the input variables. LSTM is a type of
structures to hold the accumulatively added training data set at
Recurrent Neural Network (RNN) that can hold and learn from
each iteration, “history,” and the continuously predicted values
long sequence of observations. The algorithm developed is a
for the test data sets, “prediction.”
multi-step univariate forecast algorithm [4].
As mentioned earlier, a well-known notation typically used
To implement the algorithm, Keras library along with
in building an ARIMA model is ARIM A(p, d, q), where:
Theano were installed on a cluster of high performance
– p is the number of lag observations utilized in training computing center. The LSTM algorithm developed is listed
the model (i.e., lag order). in Listing 2.
– d is the number of times differencing is applied (i.e., To be consistent with the ARIMA algorithm and in order to
degree of differencing). have a fair comparison, the algorithm starts with splitting the
– q is known as the size of the moving average window dataset into 70% training and 30% testing, respectively (Lines
(i.e., order of moving average). 1-3). To ensure the reproduction of the results and replications,
First the algorithm fits an ARIM A(5, 1, 0) model to the it is advised to fix the random number seed. In Line 4, the
test data (Lines 7 – 8). A value of 0 indicates that the element seed number is fixed to 7.
is not used when fitting the model. More specifically, an The algorithm defines a function called “fit lstm” that trains
ARIM A(5, 1, 0) indicates that the lag value is set to 5 for and builds the LSTM model. The function takes the training
autoregression. It uses a difference order of 1 to make the time dataset, the number of epochs, i.e., the number of time a given
series stationary, and finally does not consider any moving dataset is fitted to the model, and the number of neurons,
average window (i.e., a window with zero size). i.e., the number of memory units or blocks. Line 8 creates
An ARIM A(5, 1, 0) forecast model is used as the baseline an LSTM hidden later. As soon as the network is built, it
to model the forecast. This may not be the optimal model, must be compiled and parsed to comply with the mathematical
but it is generally a good baseline to build a model, as our notations and conventions used in Theano. When compiling a
explanatory experiments indicated. model, a loss function along with an optimization algorithm
The algorithm then forecasts the expected value (hat) (Line must be specified (Line 9). The “mean squared error” and
9), adds the hat to the prediction data structure (Line 10), and “ADAM” are used as the loss function and the optimization
then adds the actual value to the test set for refining and re- algorithm, respectively.
fitting the model (Line 12). Finally, having built the prediction After compilation, it is time to fit the model to the training
and history data structures, the algorithm calculates the RMSE dataset. Since the network model is stateful, the resetting stage
values, the performance metric to assess the accuracy of the of the network must be controlled specially when there is more
prediction and evaluate the forecasts (Lines 14-15). than one epoch (Lines 10 - 13). Also, since the objective is to
train an optimized model using earlier stages, it is necessary TABLE II
to set the shuffling parameter to false in order to improve T HE RMSE S OF ARIMA AND LSTM MODELS .
the learning mechanism. In Line 12, the algorithm resets the Stock RMSE % Reduction
internal state of the training and makes it ready for the next ARIMA LSTM in RMSE
iteration, i.e., epoch. N225 766.45 105.315 -86.259
IXIC 135.607 22.211 -83.621
HSI 1,306.954 141.686 -89.159
Listing 2. The developed rolling LSTM algorithm. GSPC 55.3 7.814 -85.869
# Rolling LSTM DJI-Monthly 516.979 77.643 84.981
Inputs: Time series DJI-Weekly 287.6 30.61 -89.356
Outputs: RMSE of the forecasted data Average 511.481 64.213 -87.445
# Split data into: MC 0.81 0.801 -1.111
# 70\% training and 30\% testing data HO 0.522 0.43 -17.624
1. size ← length(series) * 0.70 ER 1.286 0.251 -80.482
2. train ← series[0...size] FB 0.478 0.397 -16.945
3. test ← series[size...length(size)] MS 30.231 3.17 -89.514
TR 2.672 0.569 -78.705
# Set the random seed to a fixed value
Average 5.999 0.936 -84.394
4. set [Link](7)
of epoch and neurons. Furthermore, in Line 19 the forecast is
# Fit an LSTM model to training data
taking place for the training data. Lines 20 - 27 use the built
Procedure fit_lstm(train, epoch, neurons)
5. X ← train LSTM model to forecast the test dataset, and Lines 28 - 29
6. y ← train - X report the obtained RMSE values. It is important to note that,
7. model = Sequential() for reducing the complexity of the algorithm, some parts of the
8. [Link](LSTM(neurons), stateful=True)) algorithms are not shown in Box 2 such as dense, batch size,
9. [Link](loss=’mean_squared_error’,
transformation, etc. However, these parts are integral parts of
optimizer=’adam’)
[Link] each i in range(epoch) do the developed algorithm.
11. [Link](X, y, epochs=1, shuffle=False)
12. model.reset_states() VI. R ESULTS
[Link] for The results are reported in Table II. The data related
return model to the financial time series or stock market show that the
# Make a one-step forecast average Rooted Mean Squared Error (RMSE) using Rolling
Procedure forecast_lstm(model, X) ARIMA and Rolling LSTM models are 511.481 and 64.213,
14. yhat ← [Link](X) respectively, yielding an average of 87.445 reductions in error
return yhat rates achieved by LSTM. On the other hand, the economic
related data show a reduction of 84.394 in RMSE where the
15. epoch ← 1
16. neurons ← 4 average RMSE values for Rolling ARIMA and Rolling LSTM
17. predictions ← empty are computed as 5.999 and 0.936, respectively. The RMSE
# Fit the lstm model values clearly indicate that LSTM-based models outperform
18. lstm_model = fit_lstm(train,epoch,neurons) ARIMA-based models with a high margin (i.e., between 84%
# Forecast the training dataset - 87% reduction in error rates).
19. lstm_model.predict(train)
VII. D ISCUSSION
# Walk-forward validation on the test data
20. for each i in range(length(test)) do The good performance observed through deep learning-
21. # make one-step forecast based approaches to the prediction problem is due to the
22. X ← test[i] “iterative” optimization algorithm used in these approaches
23. yhat ← forecast_lstm(lstm_model, X)
24. # record forecast with the goal of finding the best results. By iterative we
25. [Link](yhat) mean obtain the results several times and then select the most
26. expected ← test[i] optimal one, i.e., the iteration that minimizes the errors. As
27. end for a result, the iterations help in an under-fitted model to be
transformed to a model optimally fitted to the data.
28. MSE ← mean_squared_error(expected,
predictions) The iterative optimization algorithms in deep learning of-
29. Return (RMSE ← sqrt(MSE)) ten works around tuning model parameters with respect to
the “gradient descent” where i) gradient means the rate of
A small function is created in Line 14 to call the LSTM inclination of a slope, and ii) descent means the instance
model and predict the next step (one single look-ahead esti- of descending. The gradient descent is associated with a
mation) in the dataset. The number of epochs and the number parameter called “learning rate” that represents how fast or
of neurons are set in Lines 15-16 to 1 and 4, respectively. The slow the components of the neural network are trained. There
operational part of the algorithm starts from Line 18 where are several reasons that might explain why training a network
an LSTM model is built with given training dataset, number would need more iterations than training other networks. The
more compelling reason is the case when the data size is too sensitivity analysis and experiments are depicted in Figures 1
big and thus it is practically infeasible to pass all the data to and 2 for economic and financial time series, respectively.
the training model at once. An intuitive solution to overcome As demonstrated in Figures 1 and 2, there is no evidence
this problem is to divide the data into smaller sizes and then that training the network with same dataset more than once
train the model with smaller chunks one by one (i.e., batch would improve the accuracy of the prediction. In some cases,
size and iteration). Accordingly, the weights of the parameters the performance even gets worsen indicating that the trained
of the neural networks are updated at the end of each step to models are being over-fitted. However, as a take away lesson, it
account for the variance of the newly added training data. It is apparent that setting epoch = 1 does generate a reasonable
is also possible that we train a network with the same training prediction model and thus there is no need for further training
dataset more than once in order to optimize the parameters on the same data. Another explanation might be due to rolling
(i.e., epoch). aspect of the trained model. Since in a rolling setting, a model
is refined in each round and thus a whole new LSTM model
A. Batch Size and Iterations is being trained and thus the weights are updated each time
If data is too big to be given to a neural network for training for a new model.
purposes at once, we may need to divide it into several smaller
batches and train the network in multiple stages. The batch size VIII. C ONCLUSION
refers to the total number of training data used. In other words, With the recent advancement on developing sophisticated
when a big dataset is divided into smaller number of chunks, machine learning-based techniques and in particular deep
each chunk is called a batch. On the other hand, iteration is the learning algorithms, these techniques are gaining popular-
number of batches needed to complete training a model using ity among researchers across divers disciplines. The major
the entire dataset. In fact, the number of batches is equal to question is then how accurate and powerful these newly
number of iterations for one round of training using the entire introduced approaches are when compared with traditional
set of data. For instance, assume that we have 1000 training methods. This paper compares the accuracy of ARIMA and
examples. We can divide 1000 examples into batches of 250, LSTM, as representative techniques when forecasting time
which means four iterations would be needed to use the entire series data. These two techniques were implemented and
dataset for one round of training. applied on a set of financial data and the results indicated
that LSTM was superior to ARIMA. More specifically, the
B. Epoch
LSTM-based algorithm improved the prediction by 85% on
Epoch represents the total number of times a given dataset average compared to ARIMA. Furthermore, the paper reports
is used for training purposes. More specifically, one epoch no improvement when the number of epochs is changed.
indicates that an entire dataset is given to a model only once, The work described in this paper advocates the benefits
i.e., the dataset is passed forward and then backward through of applying deep learning-based algorithms and techniques
the network only once. Since deep learning algorithms use to the economics and financial data. There are several other
gradient descent to optimize their models, it makes sense to prediction problems in finance and economics that can be
pass the entire dataset through a single network multiple times formulated using deep learning. The authors plan to investigate
with the goal of updating the weights and thus obtaining a the improvement achieved through deep learning by applying
better and more accurate prediction model. However, it is these techniques to some other problems and datasets with
not clear how many rounds, i.e., epoch, would be required various numbers of features.
to train a model with the same dataset in order to obtain the
optimal weights. Different datasets exhibit different behavior ACKNOWLEDGEMENT
and thus different epoch might be needed to optimally train This work is funded in part by grants (Awards: 1564293
their networks. and 1723765) from National Science Foundation.
C. The Impact of Epoch R EFERENCES
The study of the influence of the number of training rounds
[1] A. A. Adebiyi, A. O. Adewumi, C. K. Ayo, “Stock Price Prediction Us-
(epochs) on the same data is the focus of this section. The ing the ARIMA Model,” in UKSim-AMSS 16th International Conference
authors performed a series of experiments and sensitivity on Computer Modeling and Simulation., 2014.
analysis in which they controlled the values of epoch and [2] A. M. Alonso, C. Garcia-Martos, “Time Series Analysis - Forecasting
with ARIMA models,” Universidad Carlos III de Madrid, Universidad
captured the error rates. The epoch values varied between Politecnica de Madrid. 2012.
1 − 100 for each dataset. The reason for choosing 100 as [3] J. Brownlee, “How to Create an ARIMA Model for Time Se-
the upper level value for epochs was practical and feasibility ries Forecasting with Python,” [Link]
arima-for-time-series-forecasting-with-python/, 2017.
of the experiments. Computing error rates for epoch values [4] J. Brownlee, “Time Series Prediction with LSTM Recurrent Neural
between 1-100 took almost eight hours CPU time on a high Networks in Python with Keras,” [Link]
performance-computing cluster with Linux operating system. time-series-prediction-lstm-recurrent-neural-networks-python-keras/,
2016.
A pilot study on the data set with Epoch = 500 ran for one [5] G. Box, G. Jenkins, Time Series Analysis: Forecasting and Control, San
week on the cluster without any outputs. The results of the Francisco: Holden-Day, 1970.
 Fig. 1. The influence of the number of epochs: Economic time series data.

Fig. 2. The influence of the number of epochs: Financial time series data.
[6] A. Earnest, M. I. Chen, D. Ng, L. Y. Sin, “Using Autoregressive Influenza H5N1 Outbreaks,” BMC Bioinformatics, 15(1), 2014.
Integrated Moving Average (ARIMA) Models to Predict and Monitor [14] M. Khashei, M. Bijari, “A Novel Hybridization of Artificial Neural
the Number of Beds Occupied During a SARS Outbreak in a Tertiary Networks and ARIMA Models for Time Series forecasting,” in Applied
Hospital in Singapore,” in BMC Health Service Research, 5(36), 2005. Soft Computing 11(2): 2664-2675, 2011.
[7] T. Fischera, C. Kraussb, “Deep Learning with Long Short-term Memory [15] C. Krauss, X. A. Do, N. Huck, “Deep neural networks, gradient-
Networks for Financial Market Predictions,” in FAU Discussion Papers boosted trees, random forests: Statistical arbitrage on the S&P 500,”
in Economics 11, 2017. FAU Discussion Papers in Economics 03/2016, Friedrich-Alexander
[8] F. A. Gers, J. Schmidhuber, F. Cummins, “Learning to Forget: Continual University Erlangen-Nuremberg, Institute for Economics, 2016.
Prediction with LSTM,” in Neural Computation 12(10): 2451-2471, [16] S. I. Lee, S. J. Seong Joon Yoo, “A Deep Efficient Frontier Method
2000. for Optimal Investments,” Department of Computer Engineering, Sejong
[9] S. Hochreiter, J. Schmidhuber, “Long Short-Term Memory,” Neural University, Seoul, 05006, Republic of Korea, 2017.
Computation 9(8):1735-1780, 1997. [17] J. Patterson, Deep Learning: A Practitioner’s Approach, OReilly Media,
[10] N. Huck, “Pairs Selection and Outranking: An Application to the S&P 2017.
100 Index,” in European Journal of Operational Research 196(2): 819- [18] J. Schmidhuber, “Deep learning in neural networks: An overview,” in
825, 2009. Neural Networks, 61: 85-117, 2015.
[11] R. J. Hyndman, G. Athanasopoulos, Forecasting: Principles and Prac- [19] S. Siami-Namini, D. Muhammad, F. Fahimullah, “The Short and Long
tice. OTexts, 2014. Run Effects of Selected Variables on Tax Revenue - A Case Study,” in
[12] R. J. Hyndman, “Variations on Rolling Forecasts,” Applied Economics and Finance, 5(5), p. 23-32, 2018.
[Link] 2014. [20] S. Siami-Namini, D. Hudson, A. Trindade, “Commodity Price Volatility
[13] M. J. Kane, N. Price, M. Scotch, P. Rabinowitz, “Comparison of and U.S. Monetary Policy: The Overshooting Hypothesis of Agricultural
ARIMA and Random Forest Time Series Models for Prediction of Avian Commodity Prices,” in Agricultural and Applied Economics Association
 (AAEA) Annual Meeting, 2017.
[21] S. Siami-Namini, D. Hudson, “Inflation and Income Inequality in De-
veloped and Developing Countries,” in Southern Economics Association
Conference, 2017.
[22] , S. Siami-Namini, D. Hudson, “The Impacts of Sector Growth and
Monetary Policy on Income Inequality in Developing Countries” in
Available at SSRN, 2017.
[23] N. Tavakoli, D. Dai, Y. Chen, “Log-assisted straggler-aware I/O sched-
uler for high-end computing,” in 45th International Conference on
Parallel Processing Workshops (ICPPW), 2016.