SYSTEM IMPLEMENTATION

MODULES

 Feature Engineering
 Feature Evaluation
 RNN
 LSTM

MODULE DESCRIPTION:

Feature Engineering
Feature engineering is the art of extracting useful patterns from data to make it easier for
machine learning models to per-form its prediction. It can be considered one of the most
important skills to achieve good results for prediction tasks . It investigated the behaviour of
consistent top performers in Kaggle data mining competitions. The findings were that feature
engineering is often the most import-ant part. It is quite a subjective process requiring domain
knowledge to be effective. It is also considered an art. Engineered features should represent what
one is trying to each the network.
Feature Evaluation

Features must be evaluated once selected. The reason for this is dealing with too large of a
feature set will considerably increase training time. In addition, machine learning algorithms can
suffer from decreased accuracy if the number of variables is significantly higher than the optimal
number. Several methods of feature evaluation exist including filter based selection and wrapper
based selection. Filter based selectors filter features based on a particular statistical property of
the feature e.g. correlation. Wrapper based methods perform a heuristic search of solutions to a
classifier.

FILTER BASED SELECTION MODEL:

WRAPPER BASED SELECTION MODEL

The Boruta algorithm in R is one such wrapped based methods. This algorithm is a wrapper built
around the random forest classification algorithm. This is an ensemble classification method in
which classification is performed by voting of multiple classifiers. The algorithm works on a
similar principle as the random forest classifier. It adds randomness to the model and collects
results from the ensemble of randomized samples to evaluate attributes. This extra randomness
provides you with a clear view on which attributes are important . All features were deemed
important to the model based on the random forest, with 5 day and 10 days the highest
importance among the tested averages. The de-noised closing price was one of the most
important variables also.

The dimensionality reduction technique of principal component analysis (PCA) was also
explored. The result was four principal groups in which all attributes belonged to. The results of
the PCA was not included in the final model as computation was not an issue and the original
data performed reasonably well.
RNN: (RECURRENT NEURAL NETWORKS)

The recurrent neural network (RNN) was first developed by Elman . The RNN is structured
similarly to the MLP, with the exception that signals can flow both forward and backwards in an
[Link] design of deep learning models in terms of network parameters is
imperative to their success. The three main options available when choosing how to select
parameters for deep learning models are random search, grid search and heuristic search methods
such as genetic algorithms. As mentioned in the related work section manual grid search and
Bayesian optimization are utilized in this study. Grid search, implemented for the Elman RNN, is
the process of selecting two hyper parameters with a minimum and maximum for each. One then
searches that feature space looking for the best performing parameters. This approach was taken
for parameters which were unsuitable for Bayesian optimization.

RECURRENT NEURAL NETWORK

In addition to passing input between layers, the out-put of each layer is fed to the context layer
to be fed into the next layer with the next input. In this context, the state is overwritten at each
time step. This offers the benefit of allowing the network to assign particular weights to events
that occur in a series rather than the same weights to all input as with the MLP. This results in a
dynamic network. The length of the temporal window in a sense is the length of your networks
memory. While this addresses the temporal issue faced with a time series task, vanishing
gradient can still be an issue. In addition, some research has found that while RNN are capable of
handling long-term dependencies, in practice they often fail to learn due to the difficulties
between gradient descent and long term dependencies.

LSTM(LONG SHORT TERM MEMORY):

Similar to the RNN, Bayesian optimization was chosen for selecting parameters for this model
where possible. This is a heuristic search method which works by assuming the function was
sampled from a Gaussian process and maintains a posterior distribution for this function as the
results of different hyper parameter selections are observed. One can then optimise the expected
improvement over the best result to pick hyper parameters for the next experiment. The
performance of both the RNN and LSTM network are evaluated on validation data with
significant over fitting measures in place. Dropout is implemented in both layers. In addition, an
early stopper is programmed into the model to prevent over fitting. This stops the model if its
validation loss doesn’t improve for 5 epochs. In terms of temporal length, the LSTM is
considerably better at learning long term dependencies. As a result, picking a long window for
this parameter was less detrimental for the LSTM as the RNN. This process followed a similar
process to the RNN in which autocorrelation lag was used as a guideline. The LSTM performed
poorly on smaller window sizes. Its most effective length found was 100 days.
Long short term memory (LSTM) units address both of these issues. Developed by Hochreiter et
al. they allow the preservation of the weights that are forward and back-propagated through
layers. This is in contrast to the Elman RNN in which the state gets overwritten at each step.
They also allow the network to continue to learn over many time steps by maintaining a more
constant error. This allows the network to learn long term dependencies. A LSTM cell contains
forget and remember gates which allow the cell to decide what information to block or pass
based on its strength and importance. As a result, weak signals can be blocked which prevents
vanishing gradient.
CONCLUSION:

Deep learning models such as the RNN and LSTM are evidently effective learners on training
data with the LSTM more capable for recognising longer-term dependencies. However, a high
variance task of this nature make it difficult to transpire this into impressive validation results.
As a result it remains a difficult task. There is a fine line to balance between over fitting a model
and preventing it from learning sufficiently. Dropout is a valuable feature to assist in improving
this. However, despite using Bayesian optimization to optimize the selection of dropout it still
couldn’t guarantee good validation results. Despite the metrics of sensitivity, specificity and
precision indicating good performance, the actual performance of the ARIMA forecast based on
error was significantly worse than the neural network models. The LSTM out performed the
RNN marginally, but there was not significant difference in the results of both. However, the
LSTM takes considerably longer to train.

FUTURE ENHANCEMENT:

The ability to predict on streaming data would improve the model. Sliding window validation is
an approach not implemented here but this may be explored for future work. One problem that
will arise is that the data is inherently shrouded in noise.