See discussions, stats, and author profiles for this publication at: [Link]

net/publication/329641742

Impact of Data Normalization on Deep Neural Network for Time Series

Forecasting

Preprint · December 2018

CITATIONS READS
0 2,198

2 authors:

Samit Bhanja Abhishek Das

Government General Degree College, Singur Aliah University
10 PUBLICATIONS 43 CITATIONS 37 PUBLICATIONS 168 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Samit Bhanja on 08 November 2019.

The user has requested enhancement of the downloaded file.

 1

Impact of Data Normalization on Deep Neural

Network for Time Series Forecasting
Samit Bhanja and Abhishek Das*, Member, IEEE
*Email- adas@[Link]

Abstract—For the last few years it has been observed that the Deep Neural Networks (DNNs) has achieved an excellent success
in image classification, speech recognition. But DNNs are suffer great deal of challenges for time series forecasting because most
of the time series data are nonlinear in nature and highly dynamic in behaviour. The time series forecasting has a great impact
on our socio-economic environment. Hence, to deal with these challenges its need to be redefined the DNN model and keeping
this in mind, data pre-processing, network architecture and network parameters are need to be consider before feeding the data
into DNN models. Data normalization is the basic data pre-processing technique form which learning is to be done. The
effectiveness of time series forecasting is heavily depend on the data normalization technique. In this paper, different normalization
methods are used on time series data before feeding the data into the DNN model and we try to find out the impact of each
normalization technique on DNN to forecast the time series. Here the Deep Recurrent Neural Network (DRNN) is used to predict
the closing index of Bombay Stock Exchange (BSE) and New York Stock Exchange (NYSE) by using BSE and NYSE time series
data.

Index Terms— Neural Network, Deep Neural Network, Time Series, Data Normalization

◆ ——————————

1 INTRODUCTION

F ORECASTING the stock market price recently is gain-

ing much more attention because if the successful pre-
classification and prediction problems. Most of the shal-
low networks have one hidden layer, but Deep Neural
diction is achieved then the investors may be better Networks (DNN)[8]–[10] have many hidden layers, and
guided. The profitability of the investment in stock mar- these many hidden layer allow the user to build a network
ket is highly dependable on prediction. Moreover the pre- model to handle highly non-linear and dynamic data and
dicted trends also help the stock market regulator to make functions.
the corrective measure. Recently deep architectures are used in various fields
The Stock market is dynamic in nature and there are such as object recognition, speech recognition, natural
several complex factors that influence the market. There- language processing, physiological affect modelling, etc.,
fore the trends of the series are highly affected by these and they provide significant performance. Unfortunately,
factors. Many researches have proposed many fundamen- training deep architectures is a difficult task and the clas-
tal, technical or analytical models to predict the stock mar- sical methods are highly effective when applied to shal-
ket[1]–[3] and give more or less exact prediction. There is low architectures, but they are inefficient when they are
some other linear approach such as moving average, ex- applied on the deep architectures.
ponential smoothening, time series regression etc. One of The goal of this paper is to find out the effective data
the most common and popular linear method is the Auto- normalization method to predict the Indian stock market
regressive integrated moving average (ARIMA)[4] model. most efficiently.
Artificial Neural Network (ANN) is an efficient com- In this paper, the Deep Recurrent Neural Network
puting system whose central theme is borrowed from the (DRNN)[8] is used as DNN and multi attribute stock mar-
analogy of biological neural networks. The main objective ket data as Time Series data[4]. The historical stock mar-
of ANN[5] is to develop a system that can perform vari- ket data are applied on DRNN to train the DRNN. The
ous computational tasks faster than the traditional sys- DRNN acquire the knowledge by the training process and
tems. The ANN mimic the process of human’s brain and that knowledge is applied to predict the stock market.
solve the nonlinear problems, that’s why it widely used The organization of the paper is as follows: Section 2
for predicting and calculating the complicated task. provides the preliminary concept of DNN. Section 3, rep-
Nowadays, several Artificial Neural Network (ANN) resents the brief description of RNNs. In Section 4, de-
models such as Multilayer Perceptron (MLP) neural net- scription of time series data. Different normalization tech-
work, Back Propagation (BP)[6][7] neural networks are niques are presented in Section 5. Data set description and
used to predict the stock market price. proposed framework is represented in the Section 6. Sec-
Artificial neural networks with back propagation tion 7 describes the results and discussion and finally in
learning algorithm[6] are widely used in solving various Section 8, the conclusion is represented.
2

2 PRELIMINARY CONCEPT OF DEEP NEURAL which captures information about what has been calcu-
NETWORK lated so far. In theory RNNs can make use of information
in arbitrarily long sequences, but in practice they are lim-
The term deep neural network can have several mean- ited to looking back only a few steps. Here is what a typi-
ings, but one of the most common is to describe a neural cal RNN looks like:
network that has two or more layers of hidden processing The above diagram shows a RNN being unrolled (or un-
neurons. These multiple layers make the DNN so efficient folded) into a full network. The formulas that govern the
and effective to process the non-linear and highly dy-
computation happening in a RNN are as follows:
namic data. To process these types of dynamic infor-
𝑥𝑡 is the input at time step 𝑡.
mation, so many effective and efficient deep learning al-
𝑆𝑡 is the hidden state at time step 𝑡. It’s the “memory” of
gorithms have been developed. These deep architectures
the network. 𝑆𝑡 is calculated based on the previous hid-
are composed of many layers of no-linear processing
den state and the input at the current step: 𝑆𝑡 = 𝑓(𝑈𝑥𝑡 +
stages, where each lower layer’s output is fed into the in-
𝑊𝑆𝑡−1 ). The function 𝑓 usually is nonlinearity such
put of next higher layer.
Originally, the concept of deep learning was developed as 𝑡𝑎𝑛ℎ or𝑅𝑒𝐿𝑈.
from ANN research. Hence the MLPs or Feed-forward 𝑂𝑡 is the output at step 𝑡. 𝑂𝑡 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝑉𝑆𝑡 ).
neural network with many layers are the good example of Unlike a traditional deep neural network, which uses dif-
DNN[8]. The Back-propagation (in 1980’s) is a well- ferent parameters at each layer, an RNN shares the same
known algorithm for learning weights of these types of parameters (U, V, W above) across all steps. This reflects
DNNs. But, in practical it is not alone suitable for training the fact that the network performs the same task at each
the networks that contain more hidden layers. step, just with different inputs. This greatly reduces the
total number of parameters are needed to learn.

4 DESCRIPTION OF TIME SERIES DATA

A time series is a series of data collected in equal time in-
tervals such hour, day, month, year etc. This large volume
of time series data are analysed for the forecasting, such
as weather forecasting, stock market prediction, Sales
forecasting etc[12]–[14]. Here it is expected that value of
a data X in a given time is related to the previous values.
The time series is measured for a fixed units of time, so
the series of values can be expressed as
Fig. 1. A simple four layer DNN. 𝑋 = {𝑥(1), 𝑥(2), … . . , 𝑥(𝑡)}, where 𝑥(𝑡) is the most recent
value. For most of the time series problems, the goal is to
forecast 𝑥(𝑡 + 1) from previous values of the feature,
3 BRIEF DESCRIPTION OF RNNS where these values are directly related to the predicted
The RNNs[11] are one type deep generative architecture value.
and they are often used to model and generate the sequen-
tial data. They are first introduced in 1986. RNNs are very
5 DIFFERENT NORMALIZATION TECHNIQUES
powerful for modelling sequence data (e.g., speech or
text). The effectiveness of any learning algorithm is heavily de-
RNNs are called recurrent because they perform the same pendent on the normalization method[15]. The main ob-
task for every element of a sequence, with the output be- jective of the data normalization method to produce a
ing dependent on the previous computations. Another high quality of data that can feed into any learning algo-
way to think about RNNs is that they have a “memory” rithm. The time series data can have the wide range of val-
ues, so it need to be scale to a same range of values to
speed up the learning process. There are so many data
normalization techniques are available. Some of them are
as follows –

5.1 Min-Max Normalization

In this approach, the data scaled to a range of [0, 1] or [-1,
1]. This method convert the a input value 𝑥 of the attribute
𝑋 to 𝑥𝑛𝑜𝑟𝑚 the range[𝑙𝑜𝑤, ℎ𝑖𝑔ℎ], by using the formula-

(ℎ𝑖𝑔ℎ − 𝑙𝑜𝑤) ∗ (𝑥 − 𝑚𝑖𝑛𝑋)

𝑥𝑛𝑜𝑟𝑚 = (1)
𝑚𝑎𝑥𝑋 − 𝑚𝑖𝑛𝑋
Fig.2. A recurrent neural network and the unfolding in time of the com-
putation involved in its forward computation.
 3

Where 𝑚𝑖𝑛𝑋 and 𝑚𝑎𝑥𝑋 are the minimum and maxi- data are used for the training, validation and testing pur-
mum value of the attribute 𝑋 of the input data set. pose. The basic indices such as opening price, high price
and low price are taken for the input and closing price the
5.2 Decimal Scaling Normalization output.
In this technique, the decimal point of the values of the In this work, the BSE data from 1st January 2016 to 31st
attribute is moved. This movement of the decimal point is December 2017 (total 493 data points) are collected. Out
depends on the number of digits present in the maximum of these 493 data 70% of labelled data (345 data points) is
value of all the values of the attribute. Hence a value 𝑥 of used for the training, 15% of labelled data (74 data points)
the attribute 𝑋 is converted to 𝑥𝑛𝑜𝑟𝑚 by using the formula- is used for the validation and remaining 15% unlabelled
𝑥
𝑥𝑛𝑜𝑟𝑚 = 𝑑 (2) data (74 data points) are used for the testing purpose. The
10 NYSE data from 1st January 2016 to 31st December 2017
Where d is the smallest integer such that 𝑀𝑎𝑥(|𝑥𝑛𝑜𝑟𝑚 |) <
(total 503 data points) are also collected. Out of these 503
1
data points 70% of labelled data (352 data points) for train-
5.3 Z-Score Normalization ing purpose, 15% of labelled data (76 data points) for val-
Z-Score or Standard Score converts all the values of an at- idation purpose and remaining 15% of data (75 data
tribute to a common range of 0 and the standard deviation points) are used for the testing purpose.
of the attribute. In this method the value 𝑥 of the attribute The architecture of the proposed model depicted in
𝑋 is converted to 𝑥𝑛𝑜𝑟𝑚 by using the formula- the figure 3. In this model, BSE and NYSE stock market
𝑥 − 𝜇(𝑋) data are collected. Before applying these data into the
𝑥𝑛𝑜𝑟𝑚 = (3) forecasting model, the data pre-processing is done. In pre-
𝛿(𝑋)
processing phase different normalization techniques are
Where 𝜇(𝑋) is the mean value and 𝛿(𝑋) is the standard
applied to scale the data into a certain range. Then these
deviation of the attribute X.
normalized data set is used in the forecasting model for
5.4 Median Normalization the training, validation and testing purpose. In this work,
In this method each input value of an attribute is normal- deep recurrent neural network is used as the forecasting
ized by the median of all the values of that attribute. By model.
using the following formula, the normalized value 𝑥𝑛𝑜𝑟𝑚
of the value 𝑥 of the attribute 𝑋 can be calculated.
𝑥
𝑥𝑛𝑜𝑟𝑚 = (4)
𝑚𝑒𝑑𝑖𝑎𝑛(𝑋)
5.5 Sigmoid Normalization
Here sigmoid function is used for the data normalization.
The value 𝑥 othe the attribute 𝑋 can be normalized by the
following sigmoid function-
1
𝑥𝑛𝑜𝑟𝑚 = (5)
1 − 𝑒 −𝑥
5.6 Tanh estimators
This method is one of the most powerful and efficient nor-
malization technique. It is introduced by Hample. The
normalized value 𝑥𝑛𝑜𝑟𝑚 of the value 𝑥 of 𝑋 attribute of the
input data set can be calculated by the following formula-
0.01(𝑥 − 𝜇)
𝑥𝑛𝑜𝑟𝑚 = 0.5 [𝑡𝑎𝑛ℎ [ ] + 1] (6)
𝛿
Where µ and δ are mean value and standard deviation of
all the values of the 𝑋 attribute respectively.

6 DATA SET DESCRIPTION AND PROPOSED FRAME Fig.3. System Architecture

WORK
As the time series data, here the stock market data are 7 RESULT AND DISCUSSION
taken for the prediction. The stock market data are non-
linear and highly dynamic in nature, so it is a very chal- In this work, the stock indices like opening price, low
lenging task to successfully predict the market hypothe- price and high price are used for the input and closing
sis. price is used for the output of the network. The experi-
In this work, the Bombay Stock Exchange (BSE)[16] ment is done using the Matlab software. The data are pre-
and New York Stock Exchange (NYSE)[17] time series processed by the MinMax, Decimal scaling, Z-Score, Me-
4

dian, Sigmoid and Tanh Estimator normalization tech- MinMax 1.9097e-05 0.0033
niques. As the forecasting model here Deep Recurrent Decimal Scaling 1.6369e-06 9.5705e-04
Neural Network (DRNN) with one input layer, 20 hidden ZScore 2.2066e-04 0.0111
layer and output layer is used. The performance of DRNN Sigmoid 7.7942e-08 2.0090e-04
is measured for the different normalization technique Tanh Estimator 1.5486e-08 9.2140e-05
with respect to the Mean Squared Error (MSE) and Mean Median 1.3853e-06 8.6526e-04
Absolute Error (MAE).
Fig. 4 and Fig 5 show the graph of the predicted value of The Table 1 and Table2 shows the prediction errors
the closing price versus the actual value of BSE and NYSE such as MSE and MAE for the different normalization
stock market for the different normalization techniques process for the BSE and NYSE stock market data respec-
respectively. tively.

8 CONCLUSION
In this paper, the DRNN is applied to predict the sock
index of BSE and NYSE stock market for the different
normalization methods.
From the Fig.4. and Fig.5., it is observed that all the nor-
malization techniques produced fair results.
Also, from the Table 1 and Table 2, it can be observed
that all the normalization techniques produced very
small amount prediction errors, but the Tanh Estimator
normalization produce lesser MSE and MAE.
Fig. 4. Forecasting of closing price of BSE
Thus it can be concluded that the Tanh Estimator nor-
malization technique is an effective normalization
method for time series prediction of Deep Recurrent
Neural Network.

ACKNOWLEDGEMENT
The Authors will like to thank Hewlett Packard (HP) for
their generous grant of a HP Workstation Z4 G4 Pike
along with two units of 24' Display devices to the Princi-
pal Investigator for conducting the research.

Fig. 5. Forecasting of closing price of NYSE

REFERENCES
TABLE 1 [1] M. Farshchian and M. V. Jahan, “Stock market prediction
PREDICTION ERROR OF BSE DATA FOR THE DIFFERENT NOR- with Hidden Markov Model,” 2015 Int. Congr. Technol.
MALIZATION METHODS Commun. Knowl., no. Ictck, pp. 473–477, 2015.
[2] S. D. Bekiros, “Sign Prediction and Volatility Dynamics
BSE Data Set with Hybrid Neurofuzzy Approaches,” vol. 22, no. 12, pp.
Normalization Prediction Errors 2353–2362, 2011.
Techniques MSE MAE [3] F. Zhai, Q. Wen, Z. Yang, and Y. Song, “Hybrid Forecasting
Model Research on Stock Data Mining,” New Trends Inf. Sci.
MinMax 2.9079e-05 0.0040 Serv. Sci., pp. 630–633, 2010.
Decimal Scaling 1.0823e-07 2.4352e-04 [4] R. Adhikari and R. K. Agrawal, “An Introductory Study on
ZScore 3.0020e-04 0.0130 Time Series Modeling and Forecasting,” arXiv1302.6613 [cs,
Sigmoid 2.9824e-08 1.2972e-04 stat], 2013.
Tanh Estimator 1.1537e-08 8.1233e-05 [5] V. L. Shahpazov, V. B. Velev, and L. A. Doukovska, “Design
and application of Artificial Neural Networks for
Median 2.1359e-06 0.0011 predicting the values of indexes on the Bulgarian Stock
market,” 2013 Signal Process. Symp., pp. 1–6, 2013.
TABLE 2 [6] P. J. Werbos, “Backpropagation Through Time: What It
PREDICTION ERROR OF NYSE DATA FOR THE DIFFERENT Does and How to Do It,” Proc. IEEE, vol. 78, no. 10, pp.
NORMALIZATION METHODS 1550–1560, 1990.
[7] T. S. Lee and N. J. Chen, “Investigating the information
BSE Data Set content of non-cash-trading index futures using neural
Normalization Prediction Errors networks,” Expert Syst. Appl., vol. 22, no. 3, pp. 225–234,
Techniques MSE MAE 2002.
 5

[8] J. Schmidhuber, “Deep Learning in neural networks: An of National Dean's List, USA. He is a Member of IEEE and a Honor-
overview,” Neural Networks, vol. 61, pp. 85–117, 2015. ary Senior Member of IACSIT Singapore. He has several publica-
[9] I. Chaturvedi, Y. S. Ong, and R. V. Arumugam, “Deep
tions in International and National Journals and peer-reviewed Con-
transfer learning for classification of time-delayed Gaussian
networks,” Signal Processing, vol. 110, pp. 250–262, 2015. ferences. He is also a TPC Committee Member of IEEE International
[10] T. Villmann, M. Biehl, A. Villmann, and S. Saralajew, conferences in South-East Asia and Editor of the Journal of Image
“Fusion of deep learning architectures, multilayer Processing and Pattern Recognition progress(A UGC recognised
feedforward networks and learning vector quantizers for Journal) and Special Issues Journal on Image Processing & Com-
deep classification learning,” in 12th International Workshop puter Vision of IGI Global Publishers, USA ( SCI and Scopus In-
on Self-Organizing Maps and Learning Vector Quantization,
dexed).
Clustering and Data Visualization, WSOM 2017 - Proceedings,
2017.
[11] R. J. Williams and D. Zipser, “A Learning Algorithm for
Continually Running Fully Recurrent Neural Networks,”
Neural Comput., vol. 1, no. 2, pp. 270–280, 1989.
[12] C. Hsu, “Forecasting Stock / futures Prices by Using Neural
Networks with Feature Selection,” 2011.
[13] Q. Tang and D. Gu, “Day-Ahead Electricity Prices
Forecasting Using Artificial Neural Networks,” 2009 Int.
Conf. Artif. Intell. Comput. Intell., vol. 2, pp. 511–514, 2009.
[14] M. R. Hassan and B. Nath, “Stock market forecasting using
Hidden Markov Model: A new approach,” in Proceedings -
5th International Conference on Intelligent Systems Design and
Applications 2005, ISDA ’05, 2005, vol. 2005, pp. 192–196.
[15] S. C. Nayak, B. B. Misra, and H. S. Behera, “Impact of Data
Normalization on Stock Index Forecasting,” Int. J. Comput.
Inf. Syst. Ind. Manag. Appl., vol. 6, pp. 2150–7988, 2014.
[16] “Bombay Stock Exchane.” [Online]. Available:
[Link]
?p=%5EBSESN.
[17] “New York Stock Exchange.” [Online]. Available:
[Link]
%5ENYA.

Mr. Samit Bhanja is a Ph.D scholar with the Dept. of Computer Sc.
& Engineering at Aliah University, Kolkata and also working as an
Assistant Professor, Dept. of Computer Science, Government Gen-
eral Degree College, Hooghly. He has a MTech in Computer Sci-
ence and Engineering.

Dr. Abhishek Das is currently working as the Head and Associate

Professor in the Dept. of Computer Sc. & Engineering at Aliah Uni-
versity, Kolkata. His Research Area is in Medical Image Processing,
Computer Vision and IoT. He has received a Post Doctoral Research
position from University of West Scotland, UK by a Fellowship re-
ceived from the European Commission. He has also worked as an
Assistant Professor in the Dept. of Information Technology, Tripura
University ( A Central University), India. He has worked previously as
a Reader in Computer Sc. & Engineering, Indian Institute of Space
Science & Technology (IIST) and Lecturer in Information Technology,
Bengal Engineering and Science University, Shibpur (now IIEST
Shibpur) His Ph.D is from the Dept. of Computer Sc. & Engineering,
Jadavpur University, India. His M.S. in Electrical & Computer Engi-
neering from Kansas State University, USA and [Link] in Computer
Science & Engineering from Kalyani University, India. He has worked
as a Business System Analyst in Blue Cross Blue Shield New York,
USA & as a Quality Analyst in Vensoft Inc. Phoenix, USA and also as
a Consultant Project Manager in the Software Industry. He is also a
Visiting Scientist at Indian Statistical Institute, Kolkata. He has also
worked in Technical Educational Administration as Regional Officer
and Asst. Director (on deputation) at AICTE(under MHRD, Govt. of
India). He is a NCERT National Scholar, New Delhi and a Tilford Dow
Scholar, USA. His Biography has been published in the 28th edition
6

View publication stats