Deep_Learning_Module_1_Lesson_2

January 3, 2024

1 Multilayer Perceptron: Market Timing in APPL Stock

Reading Time 60 minutes

Prior Knowledge MLP, Market timing, Python
Keywords MLP, Classification, Neural Network, Dropout

In this lesson we will start increasing the complexity of our Neural Networks by considering more
dense MLPs that incorporate features such as Dropout in order to enhance its performance. Also,
different from the regression example we used in the Machine Learning course on timing the Mo-
mentum factor, we will deal here with a classification problem. Specifically, we will try to design
a strategy that times the return of Apple stock (‘AAPL’). For that, we will employ an MLP net-
work that aims to predict whether the future return of AAPL is positive or negative. Based on the
predictions from the model, we will later on check how our strategy performs.

1.1 1. Data and sources

As with any other Deep Learning endeavor, the single most important and necessary ingredient is
data. Different from what we did in the Momentum timing example, in this case we will retrieve
Apple stock price data from Yahoo! Finance using the famous yfinance library in Python, which
you are already familiar with:

[ ]: import numpy as np
import pandas as pd
import yfinance as yf

We will retrieve daily stock price data from AAPL (Apple Inc.) from January 1st, 1980 to a recent
date such as April 11th, 2022 (once you have gone through the entire notebook you can check what
happens with the model when altering these times).
Once we have daily prices, we can compute daily returns, which will be used to construct the inputs
(and output) of our MLP:

[ ]: df = [Link]("AAPL", start="1980-01-01", end="2022-04-11")

df["Ret"] = df["Adj Close"].pct_change()

df.reset_index(inplace=True)

1
 name = "Ret" # We define the variable 'name' for simplicity in later use

[Link]()

1.2 2. Timing Apple stock with Multilayer Perceptron (MLPs)

As we have already mentioned, the purpose of our MLP is to predict the future return (we will
shortly see for which time horizon and how) of AAPL using past returns. Hence, one of the first
things we must do is decide and define the inputs and outputs of the model.

1.2.1 2.1 Inputs and outputs

As you already know, there are multiple choices to make when selecting the inputs of a MLP model.
Some of them are based on sound theory, but also some simply stem from a trial and error process.
In this case, we will apply a very similar approach to what we did when building our models for
factor momentum timing and consider as inputs in the network the stock returns from the past 25,
60, 90, 120 and 240 days. Once again, after going over and understanding the entire notebook here,
please feel free to play around with these time frames or even add more/less inputs into the MLP
to see how its performance changes.

[ ]: df["Ret25_i"] = df[name].rolling(25).apply(lambda x: 100 * ([Link](1 + x / 100)␣

,→- 1))

df["Ret60_i"] = df[name].rolling(60).apply(lambda x: 100 * ([Link](1 + x / 100)␣

,→- 1))

df["Ret90_i"] = df[name].rolling(90).apply(lambda x: 100 * ([Link](1 + x / 100)␣

,→- 1))

df["Ret120_i"] = df[name].rolling(120).apply(lambda x: 100 * ([Link](1 + x /␣

,→100) - 1))

df["Ret240_i"] = df[name].rolling(240).apply(lambda x: 100 * ([Link](1 + x /␣

,→100) - 1))

del df["Open"]
del df["Close"]
del df["High"]
del df["Low"]
del df["Volume"]
del df["Adj Close"]

df = [Link]()
[Link](10)

Defining the output: Classification

Finally, we need to define our output label/s. We have decided to focus on the +120 (trading) days
return for Apple stock. Remember that in this case we will perform a classification task with out
MLP model, so that we simply aim to predict whether, on a given time t, the return of AAPL from
t to t + 120 days will be positive or negative (note how a zero return, although unlikely, will also
be classified as negative).

2
 Therefore, we first investigate, at a given time t, what would be the 120-day return on AAPL. Then,
we calculate our output variable, keeping in mind that we will be running a classification task and,
hence, we need to convert our output variable to a 0, 1 variable (0 for negative 120 days return, 1
for positive):

[ ]: df["Ret120"] = df["Ret120_i"].shift(-120)
df["Output"] = df["Ret120"] > 0
df["Output"] = df["Output"].astype(int)
del df["Ret120"]
df = [Link]()
[Link](10)

It is always useful to see some summary statistics of the different variables in our model. We can
very easily observe them with the ‘.describe()’ feature in Python:

[ ]: [Link]()

1.2.2 2.2 Train-Test samples and Scaling

Next important step in building our model is in defining our train and test samples, together with
the scaling of variables. As we have usually done before, we will take 20% of observations and
devote them to testing, while 80% will be used for training the model. Please note, once again, the
importance of doing this in chronological order!

[ ]: ts = int(0.2 * len(df)) # Number of observations in the test sample

split_time = len(df) - ts # From this data we are in the test sample
test_time = [Link][split_time:, 0:1].values # Keep the test sample dates
Ret_vector = [Link][split_time:, 1:2].values
[Link]()

Now we are ready to use sklearn to formally define the input and output matrices for training
(X_train and y_train) and test (X_test and y_test). One more time, make sure you keep the
‘shuffle’ option set to ‘False’ !

[ ]: from sklearn.model_selection import train_test_split

Xdf, ydf = [Link][:, 2:-1], [Link][:, -1]

X = [Link]("float32")
y = [Link]("float32")

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=ts, shuffle=False
) # It is important to keep "shuffle=False"
print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)

The last ‘print’ in the previous chunk of code gives as an output very useful information specially
for a later time where we will move on to more complex models. In this case, we know that we have
8, 144 observations in training for 5 different input series. Just as a concept check, could you say
how many observations are in the test sample?

3
 • What about scaling?
As you see, we have not implemented scaling here. Scaling of variables is actually a very good
practice that we recommend you always follow. In this case, since we are working with returns from
the same stock, the impact that scaling of inputs will have is presumably very limited. (To be clear,
we use ‘scaled’ data by taking prices that adjust for any dividends and stock splits). Nonetheless,
we leave this task to you. Once you have completed the whole notebook, come back here and scale
input variables to see whether model performance changes by a lot. Feel free to go back to the last
module of Machine Learning, where we introduced scaling.

1.2.3 2.3 Model and Training

Finally, it is time to set up our model. In this case, we will use 3 hidden layers with 25, 15 and
10 units respectively, and a final single-unit output layer. For all the hidden layers, we will use a
ReLU activation function, whereas we will opt for a sigmoid activation in the output layer.
• Dropout
Novel to MLP models we have used before (in the Machine Learning course), we are here introducing
a dropout layer after each of the hidden layers. As you already know, dropout randomly sets some
units of a hidden layer to zero. The question of how many units (i.e., frequency rate) is up to the
user to define. In this case, we set n_dropout = 0.2 to shut down 20% of the units in the
layer. You can find more info on how the dropout layers work in the official Keras documentation:
[Link]
• Loss function?
So far, we have dealt with regression problems where we used loss functions based on Mean Squared
Error (MSE) or Mean Absolute Error (MAE). But now we are in a classification problem with 2
labels (0, 1), so we select a loss function, binary cross-entropy that is essentially a log-likelihood:

N
1 X
H(yi ) = − yi log(p(yi )) + (1 − yi ) log(1 − p(yi ))
N
i=1

For more information on how this loss function works, check the official documentation in Keras:
[Link]
A simple explanation of binary cross-entropy may be found here:
[Link]
a3ac6025181a
• Metric for loss function?
Still, one question remains about which metric would we use for our loss function based on binary
cross-entropy. A metric is simply a function that is used to judge the performance of your model.
See all metrics available in Keras here: [Link]
Here, we choose to judge our model performance using ‘accuracy’, which in this case corresponds
to binary accuracy: [Link]

4
[ ]: import tensorflow as tf

[Link].clear_session() # We clear the backend to reset the random␣

,→seed process

[Link].set_seed(
1234
) # We will set a random seed so that results obtained are somewhat replicable

act_fun = "relu"
hp_units = 25
hp_units_2 = 15
hp_units_3 = 10
n_dropout = 0.2

model = [Link]()
[Link]([Link](units=hp_units, activation=act_fun))
[Link]([Link](n_dropout))
[Link]([Link](units=hp_units_2, activation=act_fun))
[Link]([Link](n_dropout))
[Link]([Link](units=hp_units_3, activation=act_fun))
[Link]([Link](n_dropout))
[Link]([Link](units=1, activation="sigmoid"))

hp_lr = 1e-5 # Learning rate

adam = [Link](learning_rate=hp_lr) # Adam optimizer

[Link](optimizer=adam, loss="binary_crossentropy", metrics=["accuracy"])

1.2.4 2.3.1 Validation and callbacks (Earlystopping)

We have already built our model, but there are still a few questions that need an answer when
training it: how many epochs do we train for? when does the model stop?
To solve these questions, we will use EarlyStopping, a method you are already familiar with (feel
free to revisit the documentation for Module 7 of Machine Learning). In this case the variable that
we will monitor will be model accuracy in the validation set. Naturally, we will aim to obtain the
maximum accuracy in validation, and set the patience to 20, so that model training will stop after
20 epochs in which there is no improvement in the accuracy that the model weights yield in the
validation set.

[ ]: es = [Link](
monitor="val_accuracy",
mode="max",
verbose=1,
patience=20,
restore_best_weights=True,
)

5
 1.2.5 2.3.2 Classification on imbalance data: class_weight
When we work on classification tasks, there is always the possibility that one of the labels we are
trying to predict is underrepresented in the training sample. Ideally you would want the model
to give a heavier weight to underrepresented labels so that you do not overlook this in future
prediction. For example, suppose that you are trying to predict corporate default, but your data
has very small percentage of observation when a firm actually defaults. You may want the model
to pay more attention to these observations where the actual action occurs. For that, we can use
class_weights.
Class-weights essentially consist on passing Keras a weight for each class in the sample, so that
we can make the model focus on a particular class more than it will based on its representation in
the sample.
Here you have a complete tutorial on how to perform these kind of tasks that use imbalance data in
Keras. Please note that his tutorial covers a wide range of topics that we will eventually cover as well.
In order not to get lost in details that are not important for now, we suggest you focus on the ‘Class
weights’ section: [Link]
In this case, we would like the model to pay more attention to labels of 0, which correspond to days
in which the 120-day-ahead return is negative. There are many ways to assign and modify class
weights, we follow here a simple one, completely discretionary:

[ ]: class_weight = {0: ([Link](y_train) / 0.5) * 1.2, 1: 1.0}

print(class_weight)

Finally, we are ready to train our model with all these ingredients!

[ ]: history = [Link](
X_train,
y_train,
validation_split=0.2,
epochs=500,
batch_size=32,
verbose=2,
callbacks=[es],
class_weight=class_weight,
)

As usual, let’s also inspect the summary of the model layers, which in the presence of dropout makes
much more sense to have clear in mind:

[ ]: [Link]()

Note that the presence of the dropout layer does not interfere with the number of parameters in
the model. This is because dropout neither adds nor reduces the number of model parameters (or
units) as whole, as units are randomly shut down during training epochs, but of course the unit
affected is not always the same!

This section concludes all relative to the construction and training of the MLP classification model.

6
 Now, as we have usually done, let’s explore if we could use this model, which achieves a decent
accuracy in the validation set, to build a trading strategy:All rights reserved WQU WorldQuant
University QQQQ

1.3 3. Financial performance of the model

As we have done before, let’s now evaluate the use of this model for a financial strategy. It is very
important that we clearly understand that evaluating the financial performance is a completely
separate thing from evaluating the predictive performance of the model. Obviously, the two
tasks are interrelated, but they may offer completely different conclusions after a thorough analysis.
We have already trained our model and assess its fit. Now, we will check if the predictions delivered
in the test sample are valid for developing a trading strategy!

1.3.1 3.1 Model performance in test sample

We will start by obtaining the predictions of the model in the test sample and its accuracy. Re-
member, we are evaluating how well our model predicts, using past AAPL returns for 5 different
windows, whether the next 120 days returns is positive or negative. The way we will operate is
defining model prediction as a 1 (i.e., positive return) if the probability assigned by the model is
higher than 0.5; and 0 (i.e., negative return) else.

[ ]: import [Link] as plt

import seaborn as sns
from sklearn import metrics

y_prob = [Link](X_test)
y_pred = [Link](y_prob > 0.50, 1, 0)

acc = [Link](X_test, y_test)

print("Model accuracy in test: ", acc)

As you can see, we obtain a very decent accuracy of the model in the test sample. But we would
also like to see when does the model ‘miss’, and when it is mostly ‘right’ about a prediction. For
that, we come to our old friend, the confusion matrix:

[ ]: cm = metrics.confusion_matrix(y_test, y_pred)
[Link](figsize=(9, 9))
ax = [Link]()
[Link](cm, annot=True, fmt="g", ax=ax)
# annot=True to annotate cells, ftm='g' to disable scientific notation

# labels, title and ticks

ax.set_xlabel("Predicted labels")
ax.set_ylabel("True labels")
ax.set_title("Confusion Matrix")
[Link].set_ticklabels(["DOWN", "UP"])
[Link].set_ticklabels(["DOWN", "UP"]);

7
 1.3.2 3.2 Trading strategy based on model predictions
Next, let’s evaluate the extent to which our model predictions can act as the foundation of a trading
strategy. To that end, we will perform a backtest in the same spirit of Module 7 in Machine
Learning:

[ ]: df_predictions = [Link](
{
"Date": test_time.flatten(),
"Pred": y_pred.flatten(),
"Ret": (Ret_vector.flatten()),
}
)
df_predictions.tail()

[ ]: df_predictions.Date = pd.to_datetime(df_predictions.Date, format="%YYYY-%mm-%dd")

df = df_predictions
[Link]()

We will define the positions that our trading strategy will take as long (+1) if the prediction of the
model is higher than 0.5, and short (−1) if less. (Note, nonetheless, that in practice our prediction
can only take values 0 or 1)
As in previous cases, we will backtest 3 trading strategies:
• A long/short strategy that will take a long or short position when model prediction indicate
so.
• A long-only strategy that will go to cash (return = 0) when model predicts a negative 120-day
return.
• A Buy-and-hold strategy that will buy the stock at the beginning of the test period and hold
it until the end of the period.

[ ]: df["Positions"] = [Link](df["Pred"] > 0.5, 1, -1)

df["Strat_ret"] = df["Positions"].shift(1) * df["Ret"]
df["Positions_L"] = df["Positions"].shift(1)
df["Positions_L"][df["Positions_L"] == -1] = 0
df["Strat_ret_L"] = df["Positions_L"] * df["Ret"]
df["CumRet"] = df["Strat_ret"].expanding().apply(lambda x: [Link](1 + x) - 1)
df["CumRet_L"] = df["Strat_ret_L"].expanding().apply(lambda x: [Link](1 + x) -␣
,→1)

df["bhRet"] = df["Ret"].expanding().apply(lambda x: [Link](1 + x) - 1)

Final_Return_L = [Link](1 + df["Strat_ret_L"]) - 1

Final_Return = [Link](1 + df["Strat_ret"]) - 1
Buy_Return = [Link](1 + df["Ret"]) - 1

print("Strat Return Long Only =", Final_Return_L * 100, "%")

print("Strat Return =", Final_Return * 100, "%")

8
 print("Buy and Hold Return =", Buy_Return * 100, "%")

[ ]: import [Link] as plt

fig = [Link](figsize=(12, 6))

ax = [Link]()
[Link](x="Date", y="bhRet", label="Buy&Hold", ax=ax)
[Link](x="Date", y="CumRet_L", label="Strat Only Long", ax=ax)
[Link](x="Date", y="CumRet", label="Strat Long/Short", ax=ax)
[Link]("date")
[Link]("Cumulative Returns")
[Link]()
[Link]()

[Link]()

1.4 4. Conclusion
In this lesson, we have worked with TensorFlow Keras on a MLP model for classification purposes.
Specifically, we designed a MLP model that aims to predict the forward 120 day return of Apple
stock. In doing so, we have introduced some novel features of Deep Learning models, such as
dropout and class weights, which will also come in handy later on for improving the performance
of our models. Still, there is much to be done in terms of improving model performance by, for
example, optimally selecting some of the hyperparameters used in the model such as the learning
rate. That is what we will do in the remainder of the module introducing hyperparameter tuning.
For that, see you in the next lesson!

Copyright 2023 WorldQuant University. This content is licensed solely for personal use. Redistri-
bution or publication of this material is strictly prohibited.