Intro to Deep Learning
Module 2
© 2024, ZAKA AI, Inc. All Rights Reserved.
Outline
● Introduction to Deep
Learning and Neural
Networks
● Data Preparation for Deep
Learning Models
● Training a Neural Network
with Keras and Tensorflow
● Advanced Deep Learning
Concepts
○ Understanding Model
Behaviour
○ Reusable Models
© 2024, ZAKA AI, Inc. All Rights Reserved.
Intro to Neural
Networks &
Deep Learning
© 2024, ZAKA AI, Inc. All Rights Reserved.
What is Deep
Learning?
© 2024, ZAKA AI, Inc. All Rights Reserved.
Machine Learning
Machine Learning is a type of Artificial Intelligence that provides
computers with the ability to learn without being explicitly
programmed.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Deep Learning
● Part of the machine learning field
● Exceptionally effective at identifying and learning
patterns.
● Utilizes learning algorithms that derive meaning out of data
by using a hierarchy of multiple layers that mimic the neural
networks of our brain.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Deep Learning vs Machine Learning
Deep Learning
Performance
Older Learning
Techniques
Amount of Data
© 2024, ZAKA AI, Inc. All Rights Reserved.
Applications of Deep Learning
© 2024, ZAKA AI, Inc. All Rights Reserved.
Optical Character Recognition
© 2024, ZAKA AI, Inc. All Rights Reserved.
Face Recognition
© 2024, ZAKA AI, Inc. All Rights Reserved.
Virtual Assistants
© 2024, ZAKA AI, Inc. All Rights Reserved.
Large Language Models
© 2024, ZAKA AI, Inc. All Rights Reserved.
Gaming
[Link]
al-characters-with-ace-for-games/
© 2024, ZAKA AI, Inc. All Rights Reserved.
Neural
Networks
© 2024, ZAKA AI, Inc. All Rights Reserved.
Inspired by the brain
Our brain has lots of neurons connected together and the
strength of the connections between neurons represents long
term knowledge.
© 2024, ZAKA AI, Inc. All Rights Reserved.
- Dendrite: It receives signals from other neurons.
- Soma (cell body): It sums all the incoming signals to generate input.
- Axon: When the sum reaches a threshold value, neuron fires and the signal travels down the
axon to the other neurons.
- Synapses: The point of interconnection of one neuron with other neurons. The amount of
signal transmitted depend upon the strength (synaptic weights) of the connections.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Artificial Neural Networks (ANN)
© 2024, ZAKA AI, Inc. All Rights Reserved.
How does it learn?
© 2024, ZAKA AI, Inc. All Rights Reserved.
How does it learn?
© 2024, ZAKA AI, Inc. All Rights Reserved.
How to find the perfect weights?
The neural network learns by adjusting its weights iteratively to
yield the desired output. BUT HOW?
- Brute force
- Random selection
- Gradient descent
- Particle swarm
- Genetic algorithms
- etc…
© 2024, ZAKA AI, Inc. All Rights Reserved.
© 2024, ZAKA AI, Inc. All Rights Reserved.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Gradient Descent
© 2024, ZAKA AI, Inc. All Rights Reserved.
Backpropagation
Efficient method of computing gradients in neural networks.
© 2024, ZAKA AI, Inc. All Rights Reserved.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Backpropagation
© 2024, ZAKA AI, Inc. All Rights Reserved.
Bias
● Extra neurons added to each
layer
● Store the value of 1
● They also have weights
attached to them
● Weights are learned during
backpropagation
output = sum (weights * inputs) + bias
© 2024, ZAKA AI, Inc. All Rights Reserved.
To summarize
In neural networks, we use
backpropagation to calculate the error
contribution of each neuron after a batch
of data.
Training a neural network basically
means calibrating all of the “weights” by
repeating two key steps, forward
propagation and backward propagation.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Data
Preparation for
Deep Learning
Models
© 2024, ZAKA AI, Inc. All Rights Reserved.
Normalization
It’s about setting a range for the data, usually between 0 and 1
○ StandardScaler
Drug Sample Molar Mass (g/mol) Half Life (hrs) Molar Mass (g/mol) Half Life (hrs) -
Drug Sample
- Normalized Normalized
Methotrexate 454.44 7
Methotrexate 0.9718 0.1591
Phenytoin 252.268 27
Phenytoin 0.5394 0.6136
Methadone 309.445 43 Methadone 0.6617 0.9773
Buprenorphine 467.64 44 Buprenorphine 1 1
© 2024, ZAKA AI, Inc. All Rights Reserved.
Standardization
Numerical values (age,
height, price, etc..) => Set the
distribution right!
○ StandardScaler
© 2024, ZAKA AI, Inc. All Rights Reserved.
Encoding Categorical Data
● Categorical data, such as words or labels, does not have a
numerical representation by default, making it impossible
for a model to process it directly.
● Encoding methods:
○ Label Encoding
○ One-hot Encoding
© 2024, ZAKA AI, Inc. All Rights Reserved.
Label Encoding
You can easily encoder categorical labels using the LabelEncoder()
class from scikit-learn library
data = [Link]({
'Category': ['A', 'B', 'C', 'A', 'B', 'C']
})
# Initialize the LabelEncoder
label_encoder = LabelEncoder()
# Apply LabelEncoder to the categorical column
data['Category_Encoded'] =
label_encoder.fit_transform(data['Category'])
© 2024, ZAKA AI, Inc. All Rights Reserved.
One-Hot Encoding
© 2024, ZAKA AI, Inc. All Rights Reserved.
Split Your Dataset
We need the network to be able to
generalize.
Split the training dataset:
- Train set
- Test set
© 2024, ZAKA AI, Inc. All Rights Reserved.
Training A Neural
Network with
Keras and
Tensorflow
© 2024, ZAKA AI, Inc. All Rights Reserved.
A Step-by-Step Approach
Below is an overview of the 5 steps in the neural network model
life-cycle in Keras:
1. Define Network
1. Define Network.
2. Compile Network. 2. Compile Network
3. Fit (Train) Network.
4. Evaluate Network.
3. Fit Network
5. Make Predictions.
4. Evaluate Network
5. Make Predictions
© 2024, ZAKA AI, Inc. All Rights Reserved.
Define the Network
The first layer in the network must define the number of inputs to
expect.
Different activation functions for the output layer:
● Regression: Linear activation function, or linear and the number
of neurons matching the number of outputs.
● Binary Classification (2 classes): Logistic activation function, or
sigmoid, and one neuron in the output layer.
● Multiclass Classification (>2 classes): Softmax activation
function, or softmax, and one output neuron per class value,
assuming a one hot encoded output pattern.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Keras Sequential vs Functional
The sequential API :
● Create models layer-by-layer
● It is limited (can’t create models that share layers or have multiple
input or output layers)
The functional API:
● An alternate way of creating models
● Offers a lot more flexibility, including creating more complex models.
Models are defined by creating instances of layers and connecting them
directly to each other in pairs, then defining a Model that specifies the
layers to act as the input and output to the model.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Sequential vs. Functional
#sequential
model = Sequential()
[Link](Dense(10, input_dim = 3, activation = 'relu'))
[Link](Dense(10, activation = 'relu'))
[Link](Dense(1, activation = 'sigmoid'))
#functional
visible = Input(shape=(3,))
hidden1 = Dense(10, activation = 'relu')(visible)
hidden2 = Dense(10, activation = 'relu')(hidden1)
outlayer = Dense(1, activation = 'sigmoid')(hidden2)
model = Model(inputs = visible, outputs = outlayer)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Sequential vs. Functional
© 2024, ZAKA AI, Inc. All Rights Reserved.
Compile the Network
Compilation transforms the simple sequence of layers that we defined into a highly
efficient series of matrix transforms in a format intended to be executed on your
GPU or CPU.
Compilation requires a number of parameters to be specified:
● The optimization algorithm to use to train the network
● The loss function used to evaluate the network that is minimized by the
optimization algorithm.
[Link](optimizer=”sgd”, loss=”mean_squared_error”)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Compile the Network
Different loss functions:
● Regression: Mean Squared Error or mean_squared_error.
● Binary Classification: Logarithmic Loss, also called cross-entropy or
binary_crossentropy.
● Multiclass Classification: Multiclass Logarithmic Loss or
categorical_crossentropy.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Train the Network
Fitting the network requires training data: inputs X
and the outputs Y.
model.fit(X, Y, batch_size=10, epochs=100)
The network is trained using the backpropagation
algorithm and optimized according to the
optimization algorithm and loss function specified
when compiling the model.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Evaluate the Network
After the model is trained, we need to evaluate it to check its
performance. This is done on a SEPARATE DATASET!
loss = [Link](X_test, Y_test)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Evaluate the Network
- Underfitting
- Overfitting
- Good fitting
© 2024, ZAKA AI, Inc. All Rights Reserved.
Make predictions
Training Data Deep Model
Learning
Unseen Data Fit Model Prediction
predictions = [Link](X)
● Regression: predictions returned in the format of the problem directly
● Binary classification: prediction is a probability for the first class that can
be converted to a 1 or 0 by rounding.
● Multiclass classification: an array of probabilities that need to be
converted to a single class output prediction using the argmax() function
© 2024, ZAKA AI, Inc. All Rights Reserved.
Final Step: Make modifications
● Try a different model architecture
● Fine tune models
● Perform feature engineering
● Experiment with ensemble methods
○ Bagging, boosting, or stacking
© 2024, ZAKA AI, Inc. All Rights Reserved.
Hands-on:
DL Models in Keras
& PyTorch
© 2024, ZAKA AI, Inc. All Rights Reserved.
Advanced Deep
Learning
Concepts
© 2024, ZAKA AI, Inc. All Rights Reserved.
Understanding
the Model’s
Behavior
© 2024, ZAKA AI, Inc. All Rights Reserved.
Cross Validation
© 2024, ZAKA AI, Inc. All Rights Reserved.
Scikit-Learn & Keras
How to wrap Keras models so that they can be used with the
scikit-learn machine learning library?
The Keras library provides a convenient wrapper for deep learning
models to be used as classification or regression estimators in
scikit-learn.
KerasClassifier and KerasRegressor
Define a function that creates a model and pass it to the Keras
wrapper
© 2024, ZAKA AI, Inc. All Rights Reserved.
Scikit-learn Cross Validation (Implementation)
1. Define function create_model()
2. Create KerasClassifier model
model = KerasClassifier(build_fn=create_model, epochs=150,
batch_size=10, verbose=0)
3. Use StratifiedKFold class from the scikit-learn library
# define 10-fold cross validation test harness
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
4. Evaluate using 10-fold cross validation
results = cross_val_score(model, X, Y, cv=kfold)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Keras Callbacks
A callback is a function to be applied at given stages of the
training procedure.
● Get a view on internal states and statistics of the model
during training.
● You can pass a list of callbacks to the .fit() method.
● The relevant methods of the callbacks will then be called at
each stage of the training.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Understand Model Behavior
History: Callback that records events into a History object.
This callback is automatically applied to every Keras model. The
History object gets returned by the fit() method of models.
We can use the data collected to create plots and reveal useful
things about the training of the model such as:
● Its speed of convergence over epochs (slope).
● Whether the model may have already converged (plateau of
the line).
● Whether the model may be over-learning the training data
(inflection for validation line).
© 2024, ZAKA AI, Inc. All Rights Reserved.
Understand Model Behavior
© 2024, ZAKA AI, Inc. All Rights Reserved.
Model Checkpoint
● Keep the best models during training with Checkpointing.
● The ModelCheckpoint instance can then be passed to the
training process when calling the fit() function on the model
© 2024, ZAKA AI, Inc. All Rights Reserved.
Early Stopping
● Early stopping is a technique allows us to stop training once
the value being monitored (e.g validation loss or val_loss) has
stopped improving.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Early Stopping
The Keras EarlyStopping Callback function allows us to stop
training once the value being monitored (e.g val_loss) has
stopped getting better.
The following is an example with the main parameters it can take:
© 2024, ZAKA AI, Inc. All Rights Reserved.
Learning Rate
© 2024, ZAKA AI, Inc. All Rights Reserved.
Learning Rate Selection
© 2024, ZAKA AI, Inc. All Rights Reserved.
Learning Rate Schedules
Adapting the learning rate for your stochastic gradient descent
optimization procedure can increase performance and reduce
training time!
1. Time-Based Learning Rate Schedule
2. Drop-Based Learning Rate Schedule
© 2024, ZAKA AI, Inc. All Rights Reserved.
Time-Based Learning Rate Schedule
Decay argument
Example: LR = 0.1 / decay = 0.001
Epoch Learning Rate
1 0.1
2 0.0999000999
3 0.0997006985
4 0.09940249103
5 0.09900646517
© 2024, ZAKA AI, Inc. All Rights Reserved.
Drop-Based Learning Rate Schedule
Systematically drop the learning rate at
specific times during training.
LearningRateScheduler Callback
Example:
LR = 0.1 & drop it by a factor of 0.5 (half it)
every 10 epochs
© 2024, ZAKA AI, Inc. All Rights Reserved.
Hands-on:
Understanding the
Model’s Behavior
© 2024, ZAKA AI, Inc. All Rights Reserved.
Model
Reusability
© 2024, ZAKA AI, Inc. All Rights Reserved.
Save/Load Models With Serialization
➔ HDF5 Format
➔ to_ json()
➔ save_weights()
➔ model_from_ json()
➔ load_weights()
© 2024, ZAKA AI, Inc. All Rights Reserved.
Transfer Learning
Transfer learning adapts pretrained models
from one task to another by reusing learned
representations and leveraging knowledge from
a related task
● Utilizes pretrained models as feature
extractors or fine-tunes them for new tasks
● Reduces data needs and computational
resources for training new models
● Beneficial for domains with related but
different datasets
© 2024, ZAKA AI, Inc. All Rights Reserved.
Hands-on:
Regression
© 2024, ZAKA AI, Inc. All Rights Reserved.
Assignment
© 2024, ZAKA AI, Inc. All Rights Reserved.
THANK
YOU
[Link]
© 2024, ZAKA AI, Inc. All Rights Reserved.