Deep Learning

Anteneh Yehalem (MSc)

 Mode of course Delivery

Lecture Lab ( please install these tools)

Project/Exam/Grading
Outline

1. What is Neural Network?

2. Why Deep Learning?

3. Three reasons to go Deep

4. Your choice of Deep Net

5. An old problem: The Vanishing Gradient

 ML-related search
results

Top ML researcher Top conference of AI/ML

The Most cited paper in 2020

Top programming language for ML

AI vs ML vs DL
• AI : broader concept (science+Engineering ) to create
intelligent machines that can simulate human thinking
capability.

• ML : subset of AI that allows machines to learn from

data without being programmed explicitly.

• DL: subset of ML, that uses the neural networks to

analyze different factors with a structure that is similar to
the human neural system.
an Introduction to:
Deep Learning
1. what exactly is deep learning ?

And, 2. why is it generally better than other methods on

image, speech and certain other types of data?

The short answers

1. ‘Deep Learning’ means using a neural network
with several layers of nodes between input and output

2. the series of layers between input & output do

feature identification and processing in a series of stages,
just as our brains seem to.
Why Were Deep Networks Hard to Train Before?

1. Vanishing and Exploding Gradients:

• When backpropagating error signals in deep networks, gradients

often shrink (vanish) or grow uncontrollably (explode).
• This made weight updates ineffective, especially for earlier
layers.
2. Poor Weight Initialization:
• If weights were initialized incorrectly, neurons would become saturated
(stuck in extreme activation values), making learning extremely slow.

3. Computational Limitations:
• Training deep networks required more computational resources than
were available in the past.
hmmm… OK, but:
3. multilayer neural networks have been around for
25 years. What’s actually new?

we have always had good algorithms for learning the

weights in networks with 1 hidden layer

but these algorithms are not good at learning the weights for
networks with more hidden layers

what’s new is: algorithms for training many-layer networks

longer answers
1. reminder/quick-explanation of how neural
network weights are learned;
2. the idea of unsupervised feature learning
(why ‘intermediate features’ are important
for difficult classification tasks, and how
NNs seem to naturally learn them)
3. The ‘breakthrough’ – the simple trick for
training Deep neural networks
-0.06

W1

-2.5 W2
f(x)
W3

1.4
-0.06

2.7

-2.5 -8.6
f(x)
0.002 x = -0.06×2.7 + 2.5×8.6 + 1.4×0.002 = 21.34

1.4
A dataset
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Training the neural network
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Training data
Fields class Initialise with random weights
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Training data
Fields class Present a training pattern
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 1.4
etc …
2.7

1.9
Training data
Fields class Feed it through to get output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 1.4
etc …
2.7 0.8

1.9
Training data
Fields class Compare with target output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 1.4
etc …
2.7 0.8
0
1.9 error 0.8
Training data
Fields class Adjust weights based on error
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 1.4
etc …
2.7 0.8
0
1.9 error 0.8
Training data
Fields class Present a training pattern
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8

1.7
Training data
Fields class Feed it through to get output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8 0.9

1.7
Training data
Fields class Compare with target output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8 0.9
1
1.7 error -0.1
Training data
Fields class Adjust weights based on error
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8 0.9
1
1.7 error -0.1
Training data
Fields class And so on ….
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8 0.9
1
1.7 error -0.1

Repeat this thousands, maybe millions of times – each time

taking a random training instance, and making slight
weight adjustments
Algorithms for weight adjustment are designed to make
changes that will reduce the error
The decision boundary perspective…
Initial random weights
The decision boundary perspective…
Present a training instance / adjust the weights
The decision boundary perspective…
Present a training instance / adjust the weights
The decision boundary perspective…
Present a training instance / adjust the weights
The decision boundary perspective…
Present a training instance / adjust the weights
The decision boundary perspective…
Eventually ….
The new way to train multi-layer NNs…
The new way to train multi-layer NNs…

Train this layer first

The new way to train multi-layer NNs…

Train this layer first

then this layer
The new way to train multi-layer NNs…

Train this layer first

then this layer
then this layer
The new way to train multi-layer NNs…

Train this layer first

then this layer
then this layer
then this layer
The new way to train multi-layer NNs…

Train this layer first

then this layer
then this layer
then this layer
finally this layer
 Why Deep Learning?

• Deeper Networks:

• Modern neural networks have many layers ("depth"), allowing them to learn
complex patterns. For example, a 100-layer network (like ResNet) can
recognize intricate details in images that shallow networks cannot.

• Hierarchical Feature Learning: Each layer in a deep network learns

increasingly abstract features. For instance, in image recognition:

• Early layers detect edges/textures.

• Middle layers recognize shapes/parts.

• Final layers identify entire objects (e.g., "cat" or "car").

 Cont....

Architectural Innovations

1. CNNs (Convolutional Neural Networks): Specialized for grid-like data (e.g

images). They use "convolutions" to scan for patterns efficiently (e.g used in facial
recognition).

2. RNNs (Recurrent Neural Networks): Designed for sequences (e.g., text, speech).
They have "memory" to process inputs over time. Variants like LSTM and GRU
fix issues with long-term memory.

3. Transformers: A breakthrough architecture using "attention" to weigh the

importance of different inputs (e.g., GPT-3 for text generation, BERT for language
understanding).
 Cont....

Improved Training Techniques

1. Activation Functions: Functions like ReLU (Rectified Linear Unit) help networks
learn non-linear patterns without saturating.

2. Optimization Algorithms: Methods like Adam adjust learning rates dynamically to

train models faster and more reliably.

3. Regularization: Techniques like dropout (randomly turning off neurons) prevent

overfitting (memorizing data instead of learning patterns).

4. Weight Initialization: Better ways to set initial parameters (e.g He initialization)

help networks train faster.
 Cont....

• Massive Computational Power and Data Availability

• Hardware Advances: GPUs and TPUs (specialized chips) speed up training, making it
feasible to train huge models.

• Big Data: The internet provides vast labeled datasets (e.g., ImageNet for images,
Wikipedia for text), which are fuel for training accurate models.

• End-to-End Learning and Automation

• Instead of hand-coding rules or features (e.g., manually defining edges in images), deep
learning systems automatically learn to map inputs to outputs.

For example: A self-driving car’s AI learns to go from raw pixels to steering commands.
 Three Reason to go Deep
1. Automatic Feature Extraction

• Traditional Machine Learning:

– Relies on manual feature engineering that requires domain expertise.

Deep Learning Advantage:

• Automatic Learning: Models learn features directly from raw data.

– Hierarchical Representations:
– Early Layers: Detect simple patterns like edges, textures, and colors.
• Deeper Layers: Combine these into complex patterns (e.g., facial features, object
parts).
Examples:
– Facial Recognition: Early layers capture outlines and basic shapes; deeper layers
differentiate unique facial attributes.
– Object Detection: Lower layers identify edges; higher layers recognize complete
objects.
 Cont…
2. Handling Big Data

• Scalability:
– Deep learning models improve in accuracy and robustness with increasing amounts of data.
• High-Dimensional Data:
– Designed to handle large, diverse datasets efficiently (e.g., millions of images
or vast text corpora).
• Parallel Processing:
– Utilizes modern hardware (like GPUs and TPUs) to process big data faster.
• Real-World Applications:
– Social Media Analysis: Processing vast amounts of image and text data.
– Medical Imaging: Analyzing high-resolution scans for disease detection.
– Sensor Data: Monitoring and analyzing data from IoT devices.
 Cont…
3. Versatility Across Domains

Wide Range of Applications:

– Computer Vision: Image classification, object detection, and scene recognition.
– Natural Language Processing: Machine translation, sentiment analysis, chatbots.
– Speech Recognition: Voice assistants and transcription services.
– Reinforcement Learning: Robotics, game playing, autonomous systems.

• Adaptability:
– Pre-trained models can be fine-tuned for specific tasks, reducing training time and data
requirements.

• Case Studies:
– Self-Driving Cars: Utilize deep learning for real-time object detection, lane tracking, and
decision-making.
– Virtual Assistants: Leverage NLP and speech recognition to understand and respond to
user commands.
 Cont…
4. Superior Performance

• Benchmark Achievements:
– Deep learning models have set new standards in accuracy and performance on
tasks like image classification and language translation.

• Innovative Architectures:
– Examples: ResNet, Inception, BERT, GPT each pushing the boundaries in their
respective domains.

• Transfer Learning:
– Leverages knowledge from pre-trained models to excel in niche applications.

• Competitive Edge:
– Dominance in competitions (e.g., ImageNet, Kaggle challenges) reinforces the
superior performance of deep learning over traditional methods.
 Your choice of Deep Net
Deep learning has revolutionized various fields, but selecting the optimal network architecture is crucial for
success. .

Factors Influencing Model Selection:

1. Data Type and Task:

● Images & Videos: Convolutional Neural Networks (CNNs) excel at capturing spatial
hierarchies through convolutions and pooling. Think of how they identify objects within a
picture by analyzing edges, shapes, and textures.

● Sequential Data (Text, Speech, Time Series): Recurrent Neural Networks (RNNs),
LSTMs, and Transformers are designed to handle temporal dependencies. They remember
past information to understand context and predict future elements in a sequence. Consider
how they power language translation or speech recognition.

● Multimodal Data: Hybrid models, combining CNNs, RNNs, and attention mechanisms, are
essential for tasks like image captioning where understanding both visual and textual
information is crucial.
Cont..
2. Network Depth and Complexity:

● Deeper Networks: Offer increased capacity to learn complex patterns, but come with
higher computational costs and training challenges (vanishing/exploding gradients).

● Modern Architectures: ResNets and Highway Networks mitigate these issues by enabling
efficient gradient flow, making training deeper networks more manageable.

3. Pre-trained Models and Transfer Learning:

● Leveraging Existing Knowledge: Starting with a pre-trained model (e.g., VGG, ResNet,
BERT) and fine-tuning it on your data saves time and resources.

● Benefits: Particularly useful with limited data or when leveraging large-scale pre-training
on benchmark datasets.
 Cont..
4. Resource Availability:

● Computational Constraints: Deep networks can be resource-intensive. Consider memory

and processing power limitations.
● Optimization Techniques: Model pruning, quantization, and knowledge distillation can
reduce the resource footprint.

Practical Considerations:

1. Accuracy vs. Complexity:

● The Trade-off: Marginal accuracy gains from a more complex model might not justify the
increased computational burden. Strive for efficiency.

2. Training Time:

● Patience is Key: Deep networks often require careful tuning of hyperparameters (learning
rates, regularization) and training schedules.

3. Application Constraints:

● Real-time Requirements: Inference speed can be as critical as accuracy for applications

with time constraints. Prioritize efficient models.
 An Old Problem: The Vanishing Gradient

What is the Vanishing Gradient Problem?

• Backpropagation in Deep Networks:

• Gradients are propagated from the output layer back to the input layer.

• These gradients indicate how to update each weight.

• Multiplication of Small Gradient Values:

• Deep networks involve many layers.

• Activation functions like sigmoid or tanh squash outputs into a narrow range, producing
small gradients.

• Exponential Shrinkage:

• Multiplying many small numbers causes the gradient values to shrink exponentially.

• As a result, early layers receive almost no learning signal.

Cont..
Why Is It a Problem?
• Slow or No Learning in Early Layers:

• Early layers capture essential low-level features.

• When gradients vanish, these layers update extremely slowly.

• Ineffective learning in early layers compromises overall network performance.

• Difficulty in Capturing Long-Range Dependencies:

• Tasks such as language modeling or time-series prediction require

understanding relationships between distant inputs.

• Vanishing gradients hinder the network's ability to learn these long-term

dependencies.
Cont...
Implications and Considerations
Impact on Model Convergence:

• Uneven learning across layers can lead to suboptimal training outcomes.

Strategies to Mitigate:

• Activation Functions: Use alternatives like ReLU that do not squash gradients
as severely.

• Network Architecture: Consider techniques such as residual connections.

• Weight Initialization: Proper initialization methods can help maintain gradient

flow.

Overall Takeaway:

• The vanishing gradient problem is a critical challenge in training very deep

networks and requires careful design choices to overcome.
?