0% found this document useful (0 votes)
13 views58 pagesKNN Regression and Data Scaling in Python
The document discusses techniques for scaling data and using K-nearest neighbors (KNN) regression for machine learning models applied to finance data in Python. It covers removing weekday features, using scikit-learn's scaler to standardize features, and making histograms to visualize the effects of scaling. It then discusses implementing a basic neural network using Keras and evaluating its performance on the data. Finally, it covers techniques for addressing overfitting like dropout and ensembling multiple models to improve predictions.
Uploaded by
david AbotsitseCopyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
13 views58 pagesKNN Regression and Data Scaling in Python
The document discusses techniques for scaling data and using K-nearest neighbors (KNN) regression for machine learning models applied to finance data in Python. It covers removing weekday features, using scikit-learn's scaler to standardize features, and making histograms to visualize the effects of scaling. It then discusses implementing a basic neural network using Keras and evaluating its performance on the data. Finally, it covers techniques for addressing overfitting like dropout and ensembling multiple models to improve predictions.
Uploaded by
david AbotsitseCopyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
Scaling data and
KNN Regression
MACHINE LEARNING FOR FINANCE IN PYTHON
Nathan George
Data Science Professor
MACHINE LEARNING FOR FINANCE IN PYTHON
Feature selection: remove weekdays
print(feature_names) print(feature_names[:-4])
['10d_close_pct', ['10d_close_pct',
'14-day SMA', '14-day SMA',
'14-day RSI', '14-day RSI',
'200-day SMA', '200-day SMA',
'200-day RSI', '200-day RSI',
'Adj_Volume_1d_change', 'Adj_Volume_1d_change',
'Adj_Volume_1d_change_SMA', 'Adj_Volume_1d_change_SMA']
'weekday_1',
'weekday_2',
'weekday_3',
'weekday_4']
MACHINE LEARNING FOR FINANCE IN PYTHON
Remove weekdays
train_features = train_features.iloc[:, :-4]
test_features = test_features.iloc[:, :-4]
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
Scaling options
Scaling options:
min-max
standardization
median-MAD
map to arbitrary function (e.g. sigmoid, tanh)
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
sklearn's scaler
from [Link] import scaler
sc = scaler()
scaled_train_features = sc.fit_transform(train_features)
scaled_test_features = [Link](test_features)
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
Making subplots
# create figure and list containing axes
f, ax = [Link](nrows=2, ncols=1)
# plot histograms of before and after scaling
train_features.iloc[:, 2].hist(ax=ax[0])
ax[1].hist(scaled_train_features[:, 2])
[Link]()
MACHINE LEARNING FOR FINANCE IN PYTHON
Scale data and use
KNN!
MACHINE LEARNING FOR FINANCE IN PYTHON
Neural Networks
MACHINE LEARNING FOR FINANCE IN PYTHON
Nathan George
Data Science Professor
MACHINE LEARNING FOR FINANCE IN PYTHON
Neural networks have potential
Neural nets have:
non-linearity
variable interactions
customizability
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
MACHINE LEARNING FOR FINANCE IN PYTHON
Implementing a neural net with keras
from [Link] import Sequential
from [Link] import Dense
MACHINE LEARNING FOR FINANCE IN PYTHON
Implementing a neural net with keras
from [Link] import Sequential
from [Link] import Dense
model = Sequential()
[Link](Dense(50,
input_dim=scaled_train_features.shape[1],
activation='relu'))
[Link](Dense(10, activation='relu'))
[Link](Dense(1, activation='linear'))
MACHINE LEARNING FOR FINANCE IN PYTHON
Fitting the model
[Link](optimizer='adam', loss='mse')
history = [Link](scaled_train_features,
train_targets,
epochs=50)
MACHINE LEARNING FOR FINANCE IN PYTHON
[Link]([Link]['loss'])
[Link]('loss:' + str(round([Link]['loss'][-1], 6)))
[Link]('epoch')
[Link]('loss')
[Link]()
MACHINE LEARNING FOR FINANCE IN PYTHON
Checking out performance
from [Link] import r2_score
# calculate R^2 score
train_preds = [Link](scaled_train_features)
print(r2_score(train_targets, train_preds))
0.4771387560719418
MACHINE LEARNING FOR FINANCE IN PYTHON
Plot performance
# plot predictions vs actual
[Link](train_preds, train_targets)
[Link]('predictions')
[Link]('actual')
[Link]()
MACHINE LEARNING FOR FINANCE IN PYTHON
Make a neural net!
MACHINE LEARNING FOR FINANCE IN PYTHON
Custom loss
functions
MACHINE LEARNING FOR FINANCE IN PYTHON
Nathan George
Data Science Professor
MACHINE LEARNING FOR FINANCE IN PYTHON
MSE with directional penalty
If prediction and target direction match:
∑(y − y^)2
If not:
∑(y − y^)2 ∗ penalty
MACHINE LEARNING FOR FINANCE IN PYTHON
Implementing custom loss functions
import tensorflow as tf
MACHINE LEARNING FOR FINANCE IN PYTHON
Creating a function
import tensorflow as tf
# create loss function
def mean_squared_error(y_true, y_pred):
MACHINE LEARNING FOR FINANCE IN PYTHON
Mean squared error loss
import tensorflow as tf
# create loss function
def mean_squared_error(y_true, y_pred):
loss = [Link](y_true - y_pred)
return tf.reduce_mean(loss, axis=-1)
MACHINE LEARNING FOR FINANCE IN PYTHON
Add custom loss to keras
import tensorflow as tf
# create loss function
def mean_squared_error(y_true, y_pred):
loss = [Link](y_true - y_pred)
return tf.reduce_mean(loss, axis=-1)
# enable use of loss with keras
import [Link]
[Link].mean_squared_error = mean_squared_error
# fit the model with our mse loss function
[Link](optimizer='adam', loss=mean_squared_error)
history = [Link](scaled_train_features, train_targets, epochs=50)
MACHINE LEARNING FOR FINANCE IN PYTHON
Checking for correct direction
[Link](y_true * y_pred, 0)
Correct direction:
neg * neg = pos
pos * pos = pos
Wrong direction:
neg * pos = neg
pos * neg = neg
MACHINE LEARNING FOR FINANCE IN PYTHON
Using [Link]()
# create loss function
def sign_penalty(y_true, y_pred):
penalty = 10.
loss = [Link]([Link](y_true * y_pred, 0),
penalty * [Link](y_true - y_pred),
[Link](y_true - y_pred))
MACHINE LEARNING FOR FINANCE IN PYTHON
Tying it together
# create loss function
def sign_penalty(y_true, y_pred):
penalty = 100.
loss = [Link]([Link](y_true * y_pred, 0),
penalty * [Link](y_true - y_pred),
[Link](y_true - y_pred))
return tf.reduce_mean(loss, axis=-1)
# enable use of loss with keras
[Link].sign_penalty = sign_penalty
MACHINE LEARNING FOR FINANCE IN PYTHON
Using the custom loss
# create the model
model = Sequential()
[Link](Dense(50,
input_dim=scaled_train_features.shape[1],
activation='relu'))
[Link](Dense(10, activation='relu'))
[Link](Dense(1, activation='linear'))
# fit the model with our custom 'sign_penalty' loss function
[Link](optimizer='adam', loss=sign_penalty)
history = [Link](scaled_train_features, train_targets, epochs=50)
MACHINE LEARNING FOR FINANCE IN PYTHON
The bow-tie shape
train_preds = [Link](scaled_train_features)
# scatter the predictions vs actual
[Link](train_preds, train_targets)
[Link]('predictions')
[Link]('actual')
[Link]()
MACHINE LEARNING FOR FINANCE IN PYTHON
Create your own
loss function!
MACHINE LEARNING FOR FINANCE IN PYTHON
Overfitting and
ensembling
MACHINE LEARNING FOR FINANCE IN PYTHON
Nathan George
Data Science Professor
MACHINE LEARNING FOR FINANCE IN PYTHON
Simplify your model
MACHINE LEARNING FOR FINANCE IN PYTHON
Neural network options
Options to combat over ing:
Decrease number of nodes
Use L1/L2 regulariation
Dropout
Autoencoder architecture
Early stopping
Adding noise to data
Max norm constraints
Ensembling
MACHINE LEARNING FOR FINANCE IN PYTHON
Dropout
MACHINE LEARNING FOR FINANCE IN PYTHON
Dropout in keras
from [Link] import Dense, Dropout
model = Sequential()
[Link](Dense(500,
input_dim=scaled_train_features.shape[1],
activation='relu'))
[Link](Dropout(0.5))
[Link](Dense(100, activation='relu'))
[Link](Dense(1, activation='linear'))
MACHINE LEARNING FOR FINANCE IN PYTHON
Test set comparison
R2 values on AMD without dropout:
train: 0.91
test: -0.72
With dropout:
train: 0.46
test: -0.22
MACHINE LEARNING FOR FINANCE IN PYTHON
Ensembling
MACHINE LEARNING FOR FINANCE IN PYTHON
Implementing ensembling
# make predictions from 2 neural net models
test_pred1 = model_1.predict(scaled_test_features)
test_pred2 = model_2.predict(scaled_test_features)
# horizontally stack predictions and take the average across rows
test_preds = [Link]([Link]((test_pred1, test_pred2)), axis=1)
MACHINE LEARNING FOR FINANCE IN PYTHON
Comparing the ensemble
Model 1 R2 score on test set:
-0.179
model 2:
-0.148
ensemble (averaged predictions):
-0.146
MACHINE LEARNING FOR FINANCE IN PYTHON
Dropout and
ensemble!
MACHINE LEARNING FOR FINANCE IN PYTHON