0% found this document useful (0 votes)

89 views11 pages

LSTM Implementation in Python

Q: What does the backward pass in an LSTM's training involve, and how is it crucial for the learning process?

The backward pass in an LSTM's training involves computing the gradient of the loss with respect to each parameter by traversing through the network in the reverse order of the forward pass. This process uses stored intermediate values from the forward pass, computing derivatives for each gate and state in the LSTM cell, crucially enabling the model to learn by adjusting its parameters in a manner that minimizes the output error. Without an accurate backward pass, the LSTM would not be able to adapt its weights effectively to the patterns in the input data .

Q: How does an LSTM handle both short-term and long-term dependencies in input sequences, and why is this important?

An LSTM handles both short-term and long-term dependencies through its memory cell and gating mechanisms. The cell state acts as a conveyor belt that maintains the memory, allowing information to persist over long periods. The gates (input, output, and forget gates) regulate the flow of information into, out of, and within the cell state. This capability is crucial because it enables the LSTM to selectively remember or forget information based on the importance of inputs at different timesteps, which is necessary for tasks where dependencies across varying lengths are critical, like language modeling or time-series prediction .

Q: Evaluate the impact of batch size on training an LSTM on time-series data.

Batch size significantly impacts the efficiency and effectiveness of training an LSTM. A larger batch size accelerates learning by providing more gradient estimates per update, hence stabilizing gradient descent by reducing variance. However, it requires more memory and can lead to poor convergence with noisy gradients, causing the model to miss the optimal solution. A smaller batch size, as used in the document, often results in better convergence but is computationally intensive and increases the training time. The choice of batch size is a key hyperparameter that balances convergence speed and training stability .

Q: Discuss the significance of early stopping in training LSTM networks.

Early stopping is significant in training LSTM networks because it helps prevent overfitting by monitoring the model's performance on a validation set and stopping training once the performance deteriorates or does not improve significantly after a certain number of epochs (patience). This approach ensures that the model does not over-optimize on the training data to the detriment of generalization to new data .

Q: Describe the process involved in updating the parameters of an LSTM network as implemented in this document.

The process of updating the parameters involves calculating the gradients of the loss function with respect to each of the learnable parameters through backpropagation. These gradients are then used to adjust the parameters in the opposite direction of the gradient to minimize the loss. The adjustment is scaled by a learning rate. This includes updating weights such as wf, wi, wo, wc, why, and biases such as bf, bi, bo, bc, by for the LSTM network. Gradient clipping is applied before updating to keep the gradients within a defined range, ensuring stable convergence .

Q: Explain the purpose and process of gradient clipping in LSTM networks.

Gradient clipping is used in LSTM networks to prevent the problem of exploding gradients, which can destabilize the training process by causing large updates to the weights. During backpropagation, before updating the weights, each gradient is compared against a predefined threshold (clip value). If a gradient exceeds this threshold, it is scaled down to adhere to the threshold limit. This ensures that the magnitude of the updates remains manageable and prevents the gradients from becoming too large .

Q: How does the weight initialization method impact the performance of an LSTM network?

The weight initialization method impacts the performance of an LSTM network by affecting how well the network learns during training. An appropriate initialization, like the 'xavier' method mentioned, helps to ensure that the initial weights are set to values that are neither too large nor too small, which helps in maintaining gradients of the right scale, thereby aiding effective learning. Without proper initialization, the gradients can either vanish or explode, impacting the convergence of the model .

Q: What is the importance of input reshaping for training LSTM models on time-series data?

Input reshaping is critical for training LSTM models on time-series data as it ensures data compatibility with the expected input format of LSTMs, which require data to be structured as a 3D array: [samples, timesteps, features]. This format helps the LSTM to understand the sequential nature of the input data, facilitating the use of its recurrent structures to learn temporal dependencies effectively. Without appropriate reshaping, the model cannot leverage its designed advantages to process sequences, leading to suboptimal learning .

Q: What is the role of intermediate values cached during the forward pass in an LSTM model?

The intermediate values cached during the forward pass in an LSTM model are crucial for performing backpropagation. These caches store the computed values of the cell state and hidden state along with the activations and gates at each timestep. During backpropagation, these cached values are used to compute the gradients of the weights and biases with respect to the loss, enabling the network to update its parameters correctly .

Q: How does the TimeSeriesDataset class facilitate time-series analysis using LSTM models, particularly with stock data?

The TimeSeriesDataset class facilitates time-series analysis by providing a structured way to load and preprocess stock data. It performs min-max normalization to scale the data, splits it into training and testing sets according to a specified proportion, and organizes it into sequences that are compatible with LSTM input requirements. The look_back parameter allows the dataset to create input-output pairs by considering a specified number of previous timesteps, thereby setting up the data for effective sequential learning by the LSTM model .

Uploaded by

Neel24787bose

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

89 views11 pages

LSTM Implementation in Python

Uploaded by

Neel24787bose

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

LSTM

May 6, 2024

1 LSTM From Scratch in Python

By Cristian Leo

[1]: import numpy as np

import pandas as pd
import [Link] as plt

# Custom classes (built from scratch)

from [Link] import WeightInitializer
from [Link] import PlotManager, EarlyStopping

[2]: class LSTM:

"""
Long Short-Term Memory (LSTM) network.

Parameters:
- input_size: int, dimensionality of input space

1
- hidden_size: int, number of LSTM units
- output_size: int, dimensionality of output space
- init_method: str, weight initialization method (default: 'xavier')
"""
def __init__(self, input_size, hidden_size, output_size,␣
↪init_method='xavier'):

self.input_size = input_size
self.hidden_size = hidden_size
self.output_size = output_size
self.weight_initializer = WeightInitializer(method=init_method)

# Initialize weights
[Link] = self.weight_initializer.initialize((hidden_size, hidden_size␣
↪+ input_size))
[Link] = self.weight_initializer.initialize((hidden_size, hidden_size␣
↪+ input_size))
[Link] = self.weight_initializer.initialize((hidden_size, hidden_size␣
↪+ input_size))
[Link] = self.weight_initializer.initialize((hidden_size, hidden_size␣
↪+ input_size))

# Initialize biases
[Link] = [Link]((hidden_size, 1))
[Link] = [Link]((hidden_size, 1))
[Link] = [Link]((hidden_size, 1))
[Link] = [Link]((hidden_size, 1))

# Initialize output layer weights and biases

[Link] = self.weight_initializer.initialize((output_size,␣
↪hidden_size))

[Link] = [Link]((output_size, 1))

@staticmethod
def sigmoid(z):
"""
Sigmoid activation function.

Parameters:
- z: [Link], input to the activation function

Returns:
- [Link], output of the activation function
"""
return 1 / (1 + [Link](-z))

@staticmethod

2
def dsigmoid(y):
"""
Derivative of the sigmoid activation function.

Parameters:
- y: [Link], output of the sigmoid activation function

Returns:
- [Link], derivative of the sigmoid function
"""
return y * (1 - y)

@staticmethod
def dtanh(y):
"""
Derivative of the hyperbolic tangent activation function.

Parameters:
- y: [Link], output of the hyperbolic tangent activation function

Returns:
- [Link], derivative of the hyperbolic tangent function
"""
return 1 - y * y

def forward(self, x):

"""
Forward pass through the LSTM network.

Parameters:
- x: [Link], input to the network

Returns:
- [Link], output of the network
- list, caches containing intermediate values for backpropagation
"""
caches = []
h_prev = [Link]((self.hidden_size, 1))
c_prev = [Link]((self.hidden_size, 1))
h = h_prev
c = c_prev

for t in range([Link][0]):
x_t = x[t].reshape(-1, 1)
combined = [Link]((h_prev, x_t))

f = [Link]([Link]([Link], combined) + [Link])

3
i = [Link]([Link]([Link], combined) + [Link])
o = [Link]([Link]([Link], combined) + [Link])
c_ = [Link]([Link]([Link], combined) + [Link])

c = f * c_prev + i * c_
h = o * [Link](c)

cache = (h_prev, c_prev, f, i, o, c_, x_t, combined, c, h)

[Link](cache)

h_prev, c_prev = h, c

y = [Link]([Link], h) + [Link]
return y, caches

def backward(self, dy, caches, clip_value=1.0):

"""
Backward pass through the LSTM network.

Parameters:
- dy: [Link], gradient of the loss with respect to the output
- caches: list, caches from the forward pass
- clip_value: float, value to clip gradients to (default: 1.0)

Returns:
- tuple, gradients of the loss with respect to the parameters
"""
dWf, dWi, dWo, dWc = [np.zeros_like(w) for w in ([Link], [Link], self.
↪wo, [Link])]

dbf, dbi, dbo, dbc = [np.zeros_like(b) for b in ([Link], [Link], self.

↪bo, [Link])]

dWhy = np.zeros_like([Link])
dby = np.zeros_like([Link])

# Ensure dy is reshaped to match output size

dy = [Link](self.output_size, -1)
dh_next = [Link]((self.hidden_size, 1)) # shape must match␣
↪hidden_size

dc_next = np.zeros_like(dh_next)

for cache in reversed(caches):

h_prev, c_prev, f, i, o, c_, x_t, combined, c, h = cache

# Add gradient from next step to current output gradient

dh = [Link]([Link].T, dy) + dh_next
dc = dc_next + (dh * o * [Link]([Link](c)))

4
df = dc * c_prev * [Link](f)
di = dc * c_ * [Link](i)
do = dh * [Link]([Link](c))
dc_ = dc * i * [Link](c_)

dcombined_f = [Link]([Link].T, df)

dcombined_i = [Link]([Link].T, di)
dcombined_o = [Link]([Link].T, do)
dcombined_c = [Link]([Link].T, dc_)

dcombined = dcombined_f + dcombined_i + dcombined_o + dcombined_c

dh_next = dcombined[:self.hidden_size]
dc_next = f * dc

dWf += [Link](df, combined.T)

dWi += [Link](di, combined.T)
dWo += [Link](do, combined.T)
dWc += [Link](dc_, combined.T)

dbf += [Link](axis=1, keepdims=True)

dbi += [Link](axis=1, keepdims=True)
dbo += [Link](axis=1, keepdims=True)
dbc += dc_.sum(axis=1, keepdims=True)

dWhy += [Link](dy, h.T)

dby += dy

gradients = (dWf, dWi, dWo, dWc, dbf, dbi, dbo, dbc, dWhy, dby)

# Gradient clipping
for i in range(len(gradients)):
[Link](gradients[i], -clip_value, clip_value, out=gradients[i])

return gradients

def update_params(self, grads, learning_rate):

"""
Update the parameters of the network using the gradients.

Parameters:
- grads: tuple, gradients of the loss with respect to the parameters
- learning_rate: float, learning rate
"""
dWf, dWi, dWo, dWc, dbf, dbi, dbo, dbc, dWhy, dby = grads

[Link] -= learning_rate * dWf

[Link] -= learning_rate * dWi

5
[Link] -= learning_rate * dWo
[Link] -= learning_rate * dWc

[Link] -= learning_rate * dbf

[Link] -= learning_rate * dbi
[Link] -= learning_rate * dbo
[Link] -= learning_rate * dbc

[Link] -= learning_rate * dWhy

[Link] -= learning_rate * dby

[3]: class LSTMTrainer:

"""
Trainer for the LSTM network.

Parameters:
- model: LSTM, the LSTM network to train
- learning_rate: float, learning rate for the optimizer
- patience: int, number of epochs to wait before early stopping
- verbose: bool, whether to print training information
- delta: float, minimum change in validation loss to qualify as an␣
↪improvement

"""
def __init__(self, model, learning_rate=0.01, patience=7, verbose=True,␣
↪delta=0):

[Link] = model
self.learning_rate = learning_rate
self.train_losses = []
self.val_losses = []
self.early_stopping = EarlyStopping(patience, verbose, delta)

def train(self, X_train, y_train, X_val=None, y_val=None, epochs=10,␣

↪batch_size=1, clip_value=1.0):

"""
Train the LSTM network.

Parameters:
- X_train: [Link], training data
- y_train: [Link], training labels
- X_val: [Link], validation data
- y_val: [Link], validation labels
- epochs: int, number of training epochs
- batch_size: int, size of mini-batches
- clip_value: float, value to clip gradients to
"""
for epoch in range(epochs):
epoch_losses = []

6
for i in range(0, len(X_train), batch_size):
batch_X = X_train[i:i + batch_size]
batch_y = y_train[i:i + batch_size]
losses = []

for x, y_true in zip(batch_X, batch_y):

y_pred, caches = [Link](x)
loss = self.compute_loss(y_pred, y_true.reshape(-1, 1))
[Link](loss)

# Backpropagation to get gradients

dy = y_pred - y_true.reshape(-1, 1)
grads = [Link](dy, caches,␣
↪clip_value=clip_value)

[Link].update_params(grads, self.learning_rate)

batch_loss = [Link](losses)
epoch_losses.append(batch_loss)

avg_epoch_loss = [Link](epoch_losses)
self.train_losses.append(avg_epoch_loss)

if X_val is not None and y_val is not None:

val_loss = [Link](X_val, y_val)
self.val_losses.append(val_loss)

if epoch % 10 == 0:
print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.
↪5f}, Val Loss: {val_loss:.5f}')

# Check early stopping condition

self.early_stopping(val_loss)
if self.early_stopping.early_stop:
print("Early stopping")
break
else:
print(f'Epoch {epoch + 1}/{epochs} - Loss: {avg_epoch_loss:.
↪5f}')

def compute_loss(self, y_pred, y_true):

"""
Compute mean squared error loss.
"""
return [Link]((y_pred - y_true) ** 2)

def validate(self, X_val, y_val):

7
"""
Validate the model on a separate set of data.
"""
val_losses = []
for x, y_true in zip(X_val, y_val):
y_pred, _ = [Link](x)
loss = self.compute_loss(y_pred, y_true.reshape(-1, 1))
val_losses.append(loss)
return [Link](val_losses)

[11]: class TimeSeriesDataset:

"""
Dataset class for time series data.

Parameters:
- ticker: str, stock ticker symbol
- start_date: str, start date for data retrieval
- end_date: str, end date for data retrieval
- look_back: int, number of previous time steps to include in each sample
- train_size: float, proportion of data to use for training
"""
def __init__(self, start_date, end_date, look_back=1, train_size=0.67):
self.start_date = start_date
self.end_date = end_date
self.look_back = look_back
self.train_size = train_size

def load_data(self):
"""
Load stock data.

Returns:
- [Link], training data
- [Link], testing data
"""
df = pd.read_csv('data/[Link]')
df = df[(df['Date'] >= self.start_date) & (df['Date'] <= self.end_date)]
df = df.sort_index()
df = [Link][self.start_date:self.end_date]
df = df[['Close']].astype(float) # Use closing price
df = [Link]([Link]) # Convert DataFrame to numpy array
train_size = int(len(df) * self.train_size)
train, test = df[0:train_size,:], df[train_size:len(df),:]
return train, test

def MinMaxScaler(self, data):

"""

8
Min-max scaling of the data.

Parameters:
- data: [Link], input data
"""
numerator = data - [Link](data, 0)
denominator = [Link](data, 0) - [Link](data, 0)
return numerator / (denominator + 1e-7)

def create_dataset(self, dataset):

"""
Create the dataset for time series prediction.

Parameters:
- dataset: [Link], input data

Returns:
- [Link], input data
- [Link], output data
"""
dataX, dataY = [], []
for i in range(len(dataset)-self.look_back):
a = dataset[i:(i + self.look_back), 0]
[Link](a)
[Link](dataset[i + self.look_back, 0])
return [Link](dataX), [Link](dataY)

def get_train_test(self):
"""
Get the training and testing data.

Returns:
- [Link], training input
- [Link], training output
- [Link], testing input
- [Link], testing output
"""
train, test = self.load_data()
trainX, trainY = self.create_dataset(train)
testX, testY = self.create_dataset(test)
return trainX, trainY, testX, testY

[12]: # Instantiate the dataset

dataset = TimeSeriesDataset('2005-01-01', '2020-12-31', train_size=0.7,␣
↪look_back=1)

trainX, trainY, testX, testY = dataset.get_train_test()

9
# Plot the data
# Combine train and test data
combined = [Link]((trainY, testY))

# Plot the data

[Link](figsize=(14, 5))
[Link](combined, label='Google Stock Price', linewidth=2, color='dodgerblue')
[Link]('Google Stock Price', fontsize=20)
[Link]('Time', fontsize=16)
[Link]('Normalized Stock Price', fontsize=16)
[Link](True)
[Link](fontsize=14)
[Link](fontsize=12)
[Link](fontsize=12)
[Link]()

[13]: # Reshape input to be [samples, time steps, features]

trainX = [Link](trainX, ([Link][0], [Link][1], 1))
testX = [Link](testX, ([Link][0], [Link][1], 1))

look_back = 1 # Number of previous time steps to include in each sample

hidden_size = 256 # Number of LSTM units
output_size = 1 # Dimensionality of the output space

lstm = LSTM(input_size=1, hidden_size=hidden_size, output_size=output_size)

# Create and train the LSTM using LSTMTrainer

trainer = LSTMTrainer(lstm, learning_rate=1e-3, patience=50, verbose=True,␣
↪delta=0.001)

[Link](trainX, trainY, testX, testY, epochs=1000, batch_size=32)

10
Epoch 1/1000 - Loss: 0.24629, Val Loss: 0.53735
Epoch 11/1000 - Loss: 0.07889, Val Loss: 0.11416
Epoch 21/1000 - Loss: 0.06242, Val Loss: 0.05693
Epoch 31/1000 - Loss: 0.05286, Val Loss: 0.03841
Epoch 41/1000 - Loss: 0.04575, Val Loss: 0.02845
Epoch 51/1000 - Loss: 0.04046, Val Loss: 0.02222
Epoch 61/1000 - Loss: 0.03655, Val Loss: 0.01831
Epoch 71/1000 - Loss: 0.03364, Val Loss: 0.01598
Epoch 81/1000 - Loss: 0.03149, Val Loss: 0.01472
Epoch 91/1000 - Loss: 0.02989, Val Loss: 0.01420
Epoch 101/1000 - Loss: 0.02870, Val Loss: 0.01417
Epoch 111/1000 - Loss: 0.02782, Val Loss: 0.01444
Epoch 121/1000 - Loss: 0.02717, Val Loss: 0.01490
Epoch 131/1000 - Loss: 0.02668, Val Loss: 0.01546
Early stopping

[15]: plot_manager = PlotManager()

# Inside your training loop

plot_manager.plot_losses(trainer.train_losses, trainer.val_losses)

# After your training loop

plot_manager.show_plots()

Common questions