Module 1_BAI701

Module 1
Introduction to Deep Learning Introduction, Shallow Learning, Deep
Learning, Why to use Deep Learning, How Deep Learning Works,Deep
Learning Challenges,. How Learning Differs from Pure Optimization,
Challenges in Neural Network Optimization.
Introduction to Deep Learning Introduction
1.1 Introduction to Machine Learning (ML)
What is Machine Learning?
Machine Learning (ML) is a branch of Artificial Intelligence (AI) that allows
machines (computers) to learn from experience (data) and improve their
performance without being directly programmed.
Imagine you are learning to ride a bicycle. You fall a few times, but gradually
you learn how to balance, steer, and stop. Similarly, a machine learns from
past data and makes better decisions in the future.
ML systems can have shallow (simple) or deep (complex, multi-layered)
structures, and they learn from raw data by finding patterns and useful
information that help in making decisions.
Three Main Types of Learning in Machine Learning
1. Supervised Learning
Think of it like a teacher teaching a student.
• The data used in supervised learning has both:
o Input (attributes or features): This is the information you give to
the model.
o Output (label): This is the answer or result you expect.
Example:
If you're building a machine learning model to identify fruits:
• Input: [Red, Round, Small]
• Label: Apple
The goal is to train the system so that when it sees a new fruit with similar
features, it can predict the correct label (e.g., Apple).
Supervised learning can handle both:
• Continuous data: like temperature, weight, etc.
• Discrete data: like categories (apple, orange, banana).
2. Unsupervised Learning

1
 Module 1_BAI701

This is like exploring without a map.

• In this method, we give the machine only input data – there are no
labels or answers provided.
• The machine has to find patterns or groups in the data on its own.
Key Goals:
1. Clustering: Grouping similar data points together.
o Example: Grouping customers with similar buying habits.
2. Density Estimation: Understanding how data is distributed.
3. Visualization: Showing data in a way we can see and understand, often
by reducing it to 2D or 3D plots.
3. Semi-Supervised Learning
Half teacher, half self-learning.
• This is a mix of supervised and unsupervised learning.
• Some data has labels, and some doesn’t.
• First, the system learns from the unlabeled data to understand the
structure.
• Then it uses the labeled data to improve accuracy.
Example:
You have 1000 photos of animals, but only 200 are labeled (dog, cat, horse,
etc.).
Semi-supervised learning helps the machine make use of all 1000 images, not
just the 200 labeled ones.
4. Reinforcement Learning
Learning by reward and punishment – like training a pet.
• The system learns from trial and error by getting feedback (reward or
penalty) from its actions.
• It does not get exact answers, but instead, it gets a signal of how well it
is doing.
Real-life Examples:
• Teaching a robot to walk.
• Optimizing routes in mobile networks.
• Choosing the best marketing strategy.
• Controlling machines in a factory.

2
 Module 1_BAI701

• Ranking search results for webpages.

Summary Table

Type of
Input Data Output/Label Goal
Learning

Supervised Input + Label Known Predict the correct label

Unsupervised Input only Not known Find patterns or groups

Semi- Learn from both labeled and

Mix of both +
Supervised unlabeled data

Feedback- Learn by actions and

Reinforcement No labels
based feedback

1.2 Shallow Learning

In the world of Machine Learning, there are two types of models based on their
structure: shallow architectures and deep architectures.
What are Shallow Architectures?
Shallow architectures are models that have only one or two layers between
the input (what you give) and the output (what you want). These types of
models are:
• Simple and easy to understand.
• Faster to train on data.
• Work well for many common problems like spam detection, loan
approval, or predicting house prices.
Because they are efficient and reliable, shallow models are still used in most
real-world applications today.
Why Do We Need Deep Architectures?
Even though shallow models are useful, they often struggle with complex
problems. For example:
• Understanding the meaning of a sentence in natural language.
• Recognizing and classifying objects in images or videos.
• Detecting emotions in speech or faces.
These tasks involve many layers of patterns and deeper relationships in the
data. Shallow models cannot capture this complexity well.

3
 Module 1_BAI701

So, researchers and engineers have turned to deep architectures (also called
deep learning), which are models with many layers. These deep models are
better at:
• Learning detailed patterns.
• Handling large and complex data.
• Solving real-world problems that need more "intelligence" and depth of
understanding.
Summary:
• Shallow models = Good for simple problems, fast, and widely used.
• Deep models = Needed for complex problems like image analysis and
natural language processing.
• That’s why today there is growing interest in deep learning, especially
in areas where shallow models do not perform well.
1.3 Deep Learning
Understanding Deep Learning in Simple Words
Deep learning is a modern and advanced part of machine learning. It has
become very popular in recent years because it helps solve more complex
problems like image recognition and understanding human language.
What is Deep Learning?
Deep learning uses special types of models called “deep networks.” These
networks have many hidden layers between the input and the output. Each
of these layers learns something different from the data. The main idea is to
understand the data step by step, going from simple things to more complex
ideas.
Imagine you are learning to recognize a face:
• First, you look at lines and curves (like eyebrows or lips),
• Then you recognize parts of the face (like eyes or nose),
• Finally, you recognize the entire face.
Deep learning works in a similar way. It learns simple features first and builds
on them to learn more complex patterns.
How is Deep Learning Different from Traditional Machine Learning?
In traditional machine learning:
• You cannot give raw data (like images or text) directly to the system.

4
 Module 1_BAI701

• You first need a feature extractor – a tool or algorithm that picks out
important information from the data (like the edges or color of an object
in an image).
• These features are designed by human experts using their knowledge
of the problem. This is called handcrafting.
But in deep learning:
• You can give raw data directly to the system (like the pixel values of an
image).
• The system automatically learns what features are important during
the training process.
• It removes the need for manual effort to extract features.
• It learns features that are often better than human-designed features.
How Does Deep Learning Work?
Think of it as a layered learning system:
• Input Layer: The raw data is given here (for example, an image with all
its pixel values).
• Hidden Layers: These layers do the actual learning. Each layer looks for
different things:
o First layers detect edges.
o Next layers detect corners and shapes.
o Higher layers understand object parts like eyes, wheels, or hands.
o Even higher layers understand complete objects like faces,
animals, or cars.
• Output Layer: This layer gives the final result – like the name of the
object or category of the input (for example, "Cat" or "Car").
Each layer uses the output of the previous layer as its input – like building
blocks. That’s why this is called hierarchical learning – learning in steps, one
on top of the other.
Why Is Deep Learning Powerful?
• It removes the need for hand-designed features.
• It learns everything from the data itself, step-by-step.
• It is very good at dealing with complex data like images, audio, and
natural language.

5
 Module 1_BAI701

• It creates a mapping between the input and output using many layers,
making it very effective.
In Summary:

Traditional ML Deep Learning

Needs feature engineering by Learns features from raw data

experts automatically

Simple models Deep networks with many layers

Struggles with raw data Handles raw data well

Fixed features Learns features during training

Slower and less accurate in High accuracy in tasks like vision,

complex tasks speech, and NLP

Why to use Deep Learning

Deep learning is a modern and advanced part of machine learning. It has
become very popular in recent years because it helps solve more complex
problems like image recognition and understanding human language.
What is Deep Learning?
Deep learning uses special types of models called “deep networks.” These
networks have many hidden layers between the input and the output. Each
of these layers learns something different from the data. The main idea is to
understand the data step by step, going from simple things to more complex
ideas.
Imagine you are learning to recognize a face:
• First, you look at lines and curves (like eyebrows or lips),
• Then you recognize parts of the face (like eyes or nose),
• Finally, you recognize the entire face.
Deep learning works in a similar way. It learns simple features first and builds
on them to learn more complex patterns.
How is Deep Learning Different from Traditional Machine Learning?
In traditional machine learning:
• You cannot give raw data (like images or text) directly to the system.
• You first need a feature extractor – a tool or algorithm that picks out
important information from the data (like the edges or color of an object
in an image).

6
 Module 1_BAI701

• These features are designed by human experts using their knowledge

of the problem. This is called handcrafting.
But in deep learning:
• You can give raw data directly to the system (like the pixel values of an
image).
• The system automatically learns what features are important during
the training process.
• It removes the need for manual effort to extract features.
• It learns features that are often better than human-designed features.
How Does Deep Learning Work?
Think of it as a layered learning system:
• Input Layer: The raw data is given here (for example, an image with all
its pixel values).
• Hidden Layers: These layers do the actual learning. Each layer looks for
different things:
o First layers detect edges.
o Next layers detect corners and shapes.
o Higher layers understand object parts like eyes, wheels, or hands.
o Even higher layers understand complete objects like faces,
animals, or cars.
• Output Layer: This layer gives the final result – like the name of the
object or category of the input (for example, "Cat" or "Car").
Each layer uses the output of the previous layer as its input – like building
blocks. That’s why this is called hierarchical learning – learning in steps, one
on top of the other.
Why Is Deep Learning Powerful?
• It removes the need for hand-designed features.
• It learns everything from the data itself, step-by-step.
• It is very good at dealing with complex data like images, audio, and
natural language.
• It creates a mapping between the input and output using many layers,
making it very effective.
In Summary:

7
 Module 1_BAI701

Traditional ML Deep Learning

Needs feature engineering by Learns features from raw data

experts automatically

Simple models Deep networks with many layers

Struggles with raw data Handles raw data well

Fixed features Learns features during training

Slower and less accurate in High accuracy in tasks like vision,

complex tasks speech, and NLP

1.5 How Deep Learning Works.

A deep learning network is a model that takes input and gives output (or
prediction), like recognizing whether an image is of a cat or a dog. But it
doesn’t do this in one step — it goes through many layers, each performing a
transformation to help the system understand the data better.
Layers and Transformations
Each layer in a deep learning network:
• Takes the output of the previous layer as input.
• Applies a transformation using some mathematical operations.

8
 Module 1_BAI701

• These transformations are controlled by weights — just numbers that

tell the network how important certain inputs are.
Think of each layer like a filter:
• Some filters look for edges,
• Others look for shapes,
• And finally, some layers can detect entire objects.
The final layer gives the result or prediction (e.g., “This is a dog”).
What is Learning in Deep Learning?
"Learning" means:
• Finding the best set of weights for all layers so that the network can
give the correct output for a given input.
The goal is to adjust these weights so that:
• The prediction made by the network is as close as possible to the
actual correct answer.
This is done by using something called a loss function.
What is a Loss Function?
A loss function (also called an objective function) measures how wrong the
network’s prediction is.
• If the network gives the wrong answer, the loss will be high.
• If the network gives the right answer, the loss will be low.
This number tells the network how well or badly it performed on one
training example.
Adjusting the Weights — The Role of the Optimizer
To reduce the loss and improve performance, we need to adjust the weights
of the network.
This is done by an algorithm called backpropagation (short for “backward
propagation of errors”).
Here's how it works step-by-step:
1. Start with random weights (the network has no idea what it’s doing).
2. Give an input example (like a picture of a cat).
3. The network gives an output (maybe it says “dog”).
4. The loss function checks how far this output is from the correct
answer (“cat”).

9
 Module 1_BAI701

5. Based on this error, the backpropagation algorithm adjusts the

weights — it tries to make the network better at predicting the correct
result.
6. Repeat this process for many examples.
Each time, the weights are slightly adjusted in a way that helps the network
get closer to the right answer next time.
This cycle is repeated thousands of times over thousands of examples, and
the network gradually learns.
When Does a Network "Learn"?
A network is said to have learned well when:
• The difference between the predicted outputs and the actual targets
becomes very small.
• The weights are now at values that give accurate predictions for new,
unseen data.
Summary Table

Concept Simple Meaning

Deep Network A model with many layers that process data step-by-step

Numbers that control how much importance to give to

Weights
certain features

Adjusting weights so the network gives the correct

Learning
answers

Measures how far the prediction is from the correct

Loss Function
answer

Backpropagation The algorithm that updates weights to reduce error

Optimizer The tool that helps find the best weights

1.6 Deep Learning Challenges.

While deep learning networks have become very popular and powerful
today, they were not always easy to use. In fact, for many years, their
problems were bigger than their benefits.
Why Was Deep Learning Not Used Earlier?
• Training deep networks used to be very slow.
• Computers were not powerful enough to handle the large amount of
calculations needed.

10
 Module 1_BAI701

• Other machine learning methods like kernel methods (e.g., SVM)

performed better with fewer resources.
What Changed?
With time, technology improved:
• GPUs (Graphics Processing Units) became faster and cheaper. GPUs
are great at doing the kind of math deep learning needs.
• Distributed computing (using multiple computers together) made
training much faster.
• Large labeled datasets (like images with tags) became easily available.
Because of these improvements, deep learning made a strong comeback and
started giving excellent results.
But Deep Learning Still Has Challenges
Even with modern technology, training deep learning models is very tricky.
Let’s see why:
1. Too Many Parameters
• A deep learning model can have millions of parameters (weights).
• Finding the right values for all of them is a very complex mathematical
problem.
• If you change one weight, it might affect others – it’s all connected.
2. Hard to Train Properly
• Many researchers are working on better ways to train deep networks.
• Some improvements include:
o Better optimizers (like Adam or RMSProp) – they help find good
weights faster.
o Smart initialization – starting with good guesses for weights.
o Activation functions based on local competition – these help the
model focus on important patterns.
o Skip connections – connecting layers in a way that allows better
flow of information through the network (used in ResNets).
Even with these techniques, deep networks still struggle, especially because
they use many layers of complex transformations that can sometimes make
learning unstable.
Too Much Data Needed
• Deep learning learns by seeing a lot of examples.

11
 Module 1_BAI701

• It performs best when large amounts of labeled data are available.

• But in many fields (like medical imaging or robotics), such data is
hard to collect.
In such cases:
• We need models that can learn well even with less data.
• This is still an ongoing challenge.
Problem of Reusability
• A deep learning model trained for one problem (like identifying cats
and dogs) works very well for that problem.
• But if we slightly change the task (like identifying lions or tigers), we
often need to retrain the entire model again.
• This takes time and effort.
Researchers are trying to build models that can:
• Handle multiple tasks at once, or
• Learn one task and transfer knowledge to another (called transfer
learning or multi-task learning),
but more progress is needed in this area.
Summary Table

Challenge What It Means

Slow Training Deep learning was too slow to be practical before GPUs
(Earlier) and big data

Too Many
Hard to find the best values for millions of weights
Parameters

Complex
Difficult math behind training makes learning unstable
Optimization

Needs Big Data Works best only with large datasets

Models can’t easily handle new but similar problems

Hard to Reuse
without retraining

✅ In Short:
While deep learning has come a long way and is very powerful today, it still
has challenges. These include training complexity, high data requirements,
and poor flexibility across tasks. Researchers are constantly working to
improve deep learning so it becomes faster, more reliable, and more reusable.

12
 Module 1_BAI701

8.1 How Learning Differs from Pure Optimization

When we train a deep learning model, our goal is to make accurate
predictions, like correctly recognizing objects in images or translating a
sentence.
To do this, we need to find the best values of the model’s parameters (weights).
This is where optimization algorithms come in.
But optimization in deep learning is a bit different from traditional
optimization.
What Do We Want to Achieve?
In machine learning, we care about a performance measure, usually called P
— this could be:
• How accurate the model is on new data (test set),
• How well it generalizes to unseen examples.
But the problem is:
• We can’t directly optimize P because we don’t have access to all possible
future data.
• So instead, we try to minimize a different function, called the cost
function (denoted as J(θ)), with the hope that minimizing J will also
improve P.
In short:
We don’t directly optimize for performance (P).
We optimize a loss or cost function (J) that indirectly helps improve
performance.
What is the Cost Function J(θ)?
This cost function tells us how wrong our model is on average.
In deep learning:
• θ (theta) represents all the weights or parameters in the model.
• J(θ) is calculated by averaging the error (loss) over all training examples.
• We want to minimize J(θ) by adjusting the parameters (weights).
Here’s what the function looks like:

Let’s break it down in simple terms:

13
 Module 1_BAI701

Symbol Meaning

x Input (e.g., image, sentence)

y Actual output or label (e.g., "cat")

f(x; θ) Prediction made by the model using weights θ

L(f(x; θ), y) Loss function: shows how wrong the prediction is

𝔼 (E) Expected value, or average over all training examples

p^data The training data we are using (empirical distribution)

So, we are saying:

“Calculate the loss for each example in the training set, average them, and try
to reduce this average error by adjusting the model's weights.”
Empirical Risk Minimization (ERM)
The method we just described — minimizing the average loss over training
data — is called:
Empirical Risk Minimization (ERM)
• "Empirical" means based on data we have (the training set).
• "Risk" means the error or loss.
• So, ERM means we are minimizing the error we see on the training set.
But in reality, we care about performance on all possible data (called the true
data distribution, denoted as Pdata— not just on the training set.
Ideal Goal vs Reality
• Ideal Goal: Minimize the loss over all data that could possibly exist (real-
world data).

• In Reality: We only have the training data, so we minimize the average

loss over the training set instead (this is the empirical version).
Summary

Concept Simplified Explanation

Optimization Adjusting the model’s weights to reduce error

Performance (P) How well the model does on real-world, unseen data

14
 Module 1_BAI701

Concept Simplified Explanation

Cost function (J(θ)) A measurable function we minimize during training

Loss function (L) Error for one example (how wrong the model is)

ERM Minimizing average error over the training data

p̂_data The training data we have

p_data The true data distribution (all possible real examples)

8.1.1 Empirical Risk Minimization

What is the Goal of a Machine Learning Algorithm?
The main goal of any machine learning algorithm is to perform well on new,
unseen data — not just the training data. This is called reducing the:
Generalization Risk
(It means: "How much error the model makes on real-world data.")
Ideal Scenario: Minimizing True Risk
In a perfect world:
• We would know the true data distribution (denoted as pdata(x,y)— this
means knowing exactly how real data is generated.
• If we knew this, we could treat machine learning as a pure optimization
problem and use algorithms to find the model that minimizes the
expected error (or risk) over all possible data.
BUT…

❌ In reality, we don’t know the full distribution of real-world data.

✅ We only have a limited dataset (training set), which is just a small sample
from the real world.
What Do We Do Instead?
Since we can’t optimize the true risk, we try to minimize the average error on
the training set — this is called:
Empirical Risk Minimization (ERM)
We replace the unknown real distribution p(x,y) with the known empirical
distribution (denoted p^(x,y), which is based on our training data.
How Is Empirical Risk Calculated?
If you have m training examples, the empirical risk is the average of the loss
(error) on all training examples:

15
 Module 1_BAI701

Symbol Meaning

m Total number of training examples

x₁, x₂,…, xₘ Input examples

y₁, y₂,…, yₘ True labels (targets)

f(xᵢ; θ) Model’s prediction for input xᵢ with parameters θ

L( … ) Loss function (how wrong the prediction is)

So, we are averaging the error for all training examples and trying to reduce
this average error.
What’s the Problem With ERM?
1. Overfitting:
o A model with too much "learning power" (high capacity) can just
memorize the training data.
o It may do very well on the training set, but fail on new data.
o This means it has low empirical risk, but high generalization risk.
2. Optimization Issues:
o Some common loss functions like 0-1 loss (used for classification,
where 0 = correct, 1 = wrong) are not smooth.
o These functions have no useful derivative, so optimization
algorithms like gradient descent cannot be used effectively.
What Do We Do in Deep Learning?
In deep learning:
• We don’t usually apply empirical risk minimization directly, because of
the two issues above:
1. Risk of overfitting,
2. Difficulty in optimization with non-differentiable loss functions.
Instead:
• We use alternative loss functions (like cross-entropy for classification),

16
 Module 1_BAI701

• And apply regularization and other techniques to improve

generalization,
• So the quantity we actually optimize is often different from the quantity
we truly care about (true risk).
Summary in Simple Terms

Term Simple Meaning

The error on new, unseen data (what we care

Generalization Risk
about most)

Empirical Risk The average error on the training data

ERM (Empirical Risk Training the model by minimizing average

Minimization) training error

When the model remembers training data too

Overfitting
well and fails on new data

A function that tells how wrong the model’s

Loss Function
prediction is

Gradient Descent A method to adjust weights to reduce the loss

Why ERM isn't ideal in Because of overfitting and hard-to-optimize

deep learning loss functions

In Short:
In deep learning, we want our model to do well on real-world data (generalize
well), but we only have training data to work with. So we optimize the loss on
the training set (empirical risk). But we must be careful not to overfit, and we
often use alternative, smoother loss functions that can be optimized with
gradient-based methods.
8.1.2 Surrogate Loss Functions and Early Stopping
Why We Use Surrogate Loss Functions in Machine Learning
In machine learning, when we train a model (like a classifier), we want it to
make as few mistakes as possible. That is, we want to reduce classification
error – how often the model predicts the wrong label. This kind of error is
usually measured using a 0-1 loss:
• 0-1 loss means:
o If the prediction is correct → loss is 0
o If the prediction is wrong → loss is 1

17
 Module 1_BAI701

But here’s the problem:

Trying to minimize this 0-1 loss directly is very hard (mathematically
intractable), especially when the data has many features. Even for simple
models like linear classifiers, minimizing 0-1 loss exactly can take too much
time (exponential time). This makes it impractical.
The Solution: Use a Surrogate Loss Function
Instead of trying to minimize 0-1 loss directly, we use a surrogate loss
function. This is a replacement loss that is:
• Easier to compute and optimize
• Smooth and differentiable (good for gradient-based optimization)
A common surrogate loss is:
• Negative log-likelihood (NLL) or cross-entropy loss
What does this do?
• It encourages the model to give high probability to the correct class
• The better it gets at this, the more accurate the model becomes
So, even though we aren’t directly minimizing 0-1 loss, we are improving
accuracy indirectly by training the model to be more confident and correct.
Can Surrogate Loss Actually Help More?
Yes!
Imagine your model is already perfect on the training data (0-1 loss is zero),
but:
• If you keep training using the surrogate loss (like log-likelihood),
• The model will start pushing the class predictions further apart
• This makes the model more confident and robust
This helps improve accuracy on new unseen data (the test set). So, using
surrogate loss doesn’t just make training easier – it can improve
generalization too!
When Do We Stop Training?
In optimization problems (like math), you usually stop when the gradient
becomes very small – this means you’ve reached a local minimum.
But in machine learning:
• We don’t wait for the surrogate loss to fully minimize
• We use early stopping – a rule that tells us when to stop training before
overfitting begins

18
 Module 1_BAI701

We check the validation set using the real loss we care about (like 0-1 loss),
and:
• If performance starts to get worse → we stop training
So, we often stop training while the surrogate loss is still decreasing, because
going further might cause overfitting.
Summary
• Directly minimizing 0-1 loss is too hard, so we use surrogate losses like
log-likelihood.
• These are easier to optimize and can even help the model learn better.
• Training doesn’t go until full convergence. We use early stopping to
prevent overfitting.
• This way, we get efficient training and better generalization to new data.
8.1.3 Batch and Minibatch Algorithms
Why Optimization in Machine Learning Is Special
In machine learning, we often train models using algorithms that are similar
to optimization algorithms, but with some key differences. One major
difference is that:
The loss function (objective function) we try to minimize is a sum over many
training examples.
For example, if we’re training a model to do classification, we might want to
maximize the likelihood that the model predicts the correct label. When we
write this in terms of log-likelihood (which is easier to optimize), it becomes a
sum of losses for each example.
Calculating the Gradient – Full Dataset vs. Mini Samples
To train the model, we need to compute the gradient of the loss function —
this tells us in which direction to change the model’s weights to reduce error.
• Exact method: Use the whole training dataset to calculate the gradient
(called batch gradient descent).
• Practical method: Use just a few samples (a minibatch) to estimate the
gradient.
Why do we use minibatches?
• It’s much faster
• Gives a good enough estimate of the true gradient
• Saves memory and computation

19
 Module 1_BAI701

This approach is called Stochastic Gradient Descent (SGD) or Mini-batch

SGD, and it's widely used in deep learning.
Why Not Use the Whole Dataset Every Time?
Let’s compare two cases:
• One uses 100 examples to compute the gradient
• The other uses 10,000 examples
While the second gives a more accurate estimate, it’s 100x slower, but only
10x better in terms of reducing noise. That’s a poor trade-off.
So, using a small sample (like 32, 64, or 128 examples) often gives the best
balance between speed and accuracy.
Redundancy in the Training Data
If many training examples are similar, they all give almost the same
contribution to the gradient. So using all of them adds repetition.
Using just one of those similar examples might be enough. That’s another
reason why small samples work well.
Types of Gradient Descent
1. Batch Gradient Descent: Uses the entire training set each time.
2. Stochastic Gradient Descent (SGD): Uses one training example at a
time.
3. Mini-batch Gradient Descent (commonly used): Uses a small batch (like
32 or 64 examples).
The term batch can be confusing:
• Batch gradient descent = full dataset
• Mini-batch = small number of examples
Choosing the Right Mini-Batch Size
The mini-batch size affects many things:

Mini-batch size Effect

More accurate gradient, but slower and uses more

Larger batch
memory

Smaller batch Faster, uses less memory, but noisier gradient

Very small batch Can act like regularization (adds noise), but may need
(e.g., 1) smaller learning rates

20
 Module 1_BAI701

On GPUs, it's better to use powers of 2 (like 32, 64, 128), as these sizes often
run faster.
Different Hardware and Algorithms
• Some methods (like second-order methods that use the Hessian matrix)
need very accurate gradients, so they require larger batch sizes.
• Other methods (like regular SGD) work well with small batches.
Importance of Shuffling
If you always take training examples in order, it can hurt performance.
Example:
• A medical dataset may have 5 samples from patient A, then 5 from
patient B, and so on.
• If a minibatch has only patient A's data, it's biased.
Solution: Shuffle the data before training so that each mini-batch has a mix
of different kinds of examples.
For huge datasets, shuffle the data once, save it, and reuse the same order
for multiple passes.
Online Learning and Generalization
In online learning, the model sees one new example at a time, like a person
learning from each new experience. This approach:
• Matches the true data distribution
• Helps reduce the generalization error (error on new data)
This works best if each example is used only once (no repeats). But often, we
go through the training data multiple times (called epochs) for better learning.
Only the first pass gives an unbiased estimate of the generalization error. After
that, we start reusing examples, but it's usually okay.
In Big Data: Underfitting > Overfitting
For very large datasets:
• Overfitting becomes less of a problem.
• The challenge is to learn fast enough before we run out of time or
computing power.
So in such cases:
• We often go through the dataset just once.
• Efficiency is more important than squeezing out the last bit of accuracy.

21
 Module 1_BAI701

Summary
• Machine learning optimization deals with loss functions summed over
examples.
• Instead of using the whole dataset to compute the gradient, we use
minibatches (small samples) for faster learning.
• Minibatch SGD balances accuracy, speed, and memory usage.
• Shuffling the data is important to avoid bias.
• Smaller batches add useful randomness, but may need smaller learning
rates.
• In huge datasets, going through the data only once may be best.
• Using the right batch size and strategy helps train models efficiently
and effectively.
8.2 Challenges in Neural Network Optimization
Why Optimization in Deep Learning Is Hard
Optimization is the process of finding the best solution to a problem — in
machine learning, that usually means finding the best model parameters that
minimize a loss function (or maximize accuracy).
But here’s the truth:
Optimization is a very difficult task in general.
In Traditional Machine Learning: We Use Convex Optimization
In earlier machine learning methods (like logistic regression or support vector
machines), experts would carefully design the objective (loss) functions and
constraints so that the optimization problem would be convex.
What is a convex problem?
• A convex function has one global minimum — like a smooth bowl.
• It’s easy to find the bottom using optimization algorithms.
• No matter where you start, you’ll end up at the best solution.
So convex problems are easier and safer to solve.
In Deep Learning: The Problem Is Non-Convex
Neural networks are complex models with many layers and parameters. When
we train them:
• The loss function becomes non-convex
• That means the loss function has many hills and valleys

22
 Module 1_BAI701

• There are many local minima, saddle points, and flat regions
So:
• It’s hard to find the best solution
• Optimization becomes much more challenging
Even Convex Optimization Isn’t Always Simple
Even when the problem is convex (like in some simpler models), it's not always
easy:
• There might still be practical issues, like how fast it converges
• Or how well it handles large datasets and limited computing resources
What This Section Is About
This part of the book will now discuss:
• The main challenges we face when trying to optimize deep learning
models
• Why these problems are harder than traditional machine learning
• And how we try to deal with these challenges using different techniques
Summary
• Optimization is the process of tuning model parameters to minimize
error.
• Traditional ML models used convex problems to make optimization
easier.
• Deep learning models create non-convex problems that are much
harder.
• Even convex optimization has its own practical difficulties.
• The following section will explore the biggest optimization challenges in
training deep models.
8.2.1 Ill-Conditioning
Challenges in Optimization — Even for Convex Functions
Even when the function we are trying to optimize is convex (which is supposed
to be easier), we can still face serious problems. One of the biggest issues is
something called:
Ill-conditioning of the Hessian matrix
What Is "Ill-Conditioning"?
Let’s break this down:

23
 Module 1_BAI701

• The Hessian matrix is used to describe the curvature of the loss

function (how steep or flat the surface is in different directions).
• Ill-conditioning means that the surface of the loss function is shaped in
a very uneven way — like a long, narrow valley.
• This makes it hard to move in the right direction during training.
In such cases, even if you have a strong gradient (which tells you which way
to go), if you take a small step in that direction, it might actually increase the
loss instead of decreasing it!
Why Ill-Conditioning Affects Learning
The training process often uses gradient descent — which updates weights by
using the gradient (slope) of the loss function. But when ill-conditioning
happens, the gradient descent step behaves poorly.
Let’s say we take a small step during training. The change in the cost function
(loss) depends on two terms:
Change in loss ≈ - ε gᵗg + ½ ε² gᵗHg
• gᵗg is the square of the gradient norm (strength of the slope).
• gᵗHg is related to curvature from the Hessian.
If the curvature term (gᵗHg) becomes larger than the gradient term (gᵗg), then:
• The cost may increase even with small steps.
• So, the learning algorithm needs to reduce the learning rate (step size)
to avoid overshooting.
• This makes training very slow, even though the model could be
improving fast.
What Does It Look Like in Practice?
In real training situations, like when training a convolutional neural network
for object detection, we can observe:
• The gradient norm (gᵗg) actually increases over time instead of
decreasing.
• This seems strange, because we expect the gradient to get smaller if the
model is converging to a good solution.
• But even though the gradient grows, the model still performs well — the
validation classification error goes down.
This means that:
• The model is still learning successfully.

24
 Module 1_BAI701

• But due to ill-conditioning, the training takes longer and needs smaller
learning steps.
Why Common Solutions Don’t Always Work for Neural Networks
In other optimization problems (like in engineering or physics), people often
use a powerful method called Newton’s Method to deal with ill-conditioning.
Newton’s Method uses the Hessian matrix to adjust the steps in the correct
directions and solve the problem faster.
But in deep learning:
• Newton’s Method can’t be directly applied.
• The Hessian matrix is too large to compute for big neural networks.
• It needs major changes before we can use it in neural network training.
Summary
• Ill-conditioning of the Hessian is a big challenge even in convex
optimization.
• It means the training surface is shaped in a way that makes gradient
descent inefficient.
• The training may slow down, even if the model is capable of learning
well.
• Neural network training often suffers from this issue.
• Standard fixes, like Newton’s Method, don’t work easily with deep
networks without major changes.
8.2.2 Local Minima
Convex vs. Non-Convex Optimization in Deep Learning
When we try to train a model using optimization, we’re basically trying to find
the best settings (parameters) that minimize a loss function — a
mathematical way of saying, “How wrong is the model?”
In Convex Optimization (Simpler Case)
A convex function is one where:
• It has only one lowest point (called the global minimum).
• If you find any local minimum, it’s also the global minimum.
• Some convex functions might have a flat region at the bottom, but any
point in that flat region is a good solution.
So, if you're optimizing a convex function, you're safe — as long as you find
any critical point (where the slope is zero), you're good.

25
 Module 1_BAI701

In Deep Learning: We Deal with Non-Convex Functions

Neural networks have non-convex loss functions, which means:
• There can be many local minima (small dips in the loss landscape).
• The loss surface is bumpy and irregular.
• This makes optimization more difficult.
But here's the good news:
Even though there are many local minima, they are often not a big problem.
Why So Many Local Minima in Neural Networks?
This happens because of something called model identifiability.
A model is identifiable if a big enough training set gives you only one unique
solution for the parameters.
But in neural networks:
• The same model behavior can come from many different sets of
parameters.
• For example, you can swap two hidden units and still get the exact same
output.
This is called weight space symmetry — and it creates many local minima that
are mathematically different but functionally identical.
Also:
• In networks like ReLU or Maxout, you can scale the weights going into
a unit and adjust the weights coming out, and the output remains the
same.
• This creates a huge number of equivalent solutions.
These don't hurt performance — they just mean the loss function has many
identical dips.
When Are Local Minima Actually a Problem?
Sometimes, local minima can be bad, if they have a higher loss than the best
possible solution.
• In very small neural networks, researchers have found examples where
the model gets stuck in a bad local minimum with high loss.
• But in large networks (the kind used in real applications), this seems to
be rare.
So far, research shows that:

26
 Module 1_BAI701

• Most local minima in big networks are actually pretty good.

• It's not necessary to find the perfect (global) minimum.
• It's more important to find a good enough solution with low loss.
How to Tell if Local Minima Are a Problem?
If you're training a neural network and it's not improving, many people blame
local minima.
But here’s a simple way to check:
Plot the gradient norm (size of the slope) over time.
• If the gradient gets smaller and smaller, the model is converging to a
solution (maybe a local minimum).
• But if the gradient is still large, then:
o You're not stuck in a local minimum
o Something else might be the problem
Important Note: In high-dimensional spaces (like neural networks), many
structures (not just local minima) can make gradients small — so it's hard to
be 100% sure what’s causing the issue.
Summary
• In convex problems, any local minimum is a good solution.
• Neural networks have non-convex loss functions with many local
minima.
• But most of these local minima are not harmful — they lead to similar
performance.
• Issues like weight symmetry and scaling tricks create many
mathematically different but equivalent solutions.
• Only rarely do local minima have high loss and cause serious problems.
• You can use the gradient norm to check if local minima are slowing
down training.
• Modern research shows that finding a global minimum is not necessary
— a good enough local minimum is usually fine for training deep
models.
8.2.3 Plateaus, Saddle Points and Other Flat Regions
Why Saddle Points Are a Challenge in Deep Learning Optimization

27
 Module 1_BAI701

When training deep neural networks, the optimization process can get stuck—
not because of local minima (which we often fear)—but due to something
called saddle points.
What Are Saddle Points?
• A saddle point is a point in the loss (or cost) function where the slope
(gradient) is zero.
• But unlike a local minimum (a small valley) where all directions lead
uphill, at a saddle point:
o Some directions lead to lower cost (downhill).
o Other directions lead to higher cost (uphill).
Imagine sitting on a horse saddle: the saddle goes down along the direction
you’re sitting (forward/backward), but it curves up along the sides (left/right).
That’s how the cost function behaves near a saddle point.
Hessian Matrix and Saddle Points
• The Hessian matrix tells us about the curvature of the cost function.
• At a saddle point, this matrix has both positive and negative values,
meaning:
o Some directions curve upward (bad),
o Some curve downward (good).
This mix of directions makes it hard for optimization algorithms to know what
to do.
Saddle Points vs. Local Minima
• In low-dimensional problems (few features or weights), local minima are
common.
• But in high-dimensional problems (like deep networks with millions of
parameters), saddle points are far more common than local minima.
• Why? Think of each dimension like a coin flip:
o To get a local minimum, you need all “heads” (positive curvature).
o But getting all heads in, say, 1000 coin flips is almost impossible.
Hence, saddle points (a mix of heads and tails) dominate.
Key Insight: Not All Saddle Points Are Bad
Interestingly, researchers found that:
• Saddle points found at lower cost values are more likely to lead to a
good solution.

28
 Module 1_BAI701

• Saddle points with very high cost values are more like traps or barriers
that make learning hard.
Real Neural Networks Also Have Many Saddle Points
• Theoretical and experimental work by researchers (e.g., Dauphin et al.
2014, Goodfellow et al. 2015) shows that real deep networks have:
o A large number of saddle points (especially early in training),
o But gradient descent (like SGD) often escapes from them
effectively.
• Visualizations show that even when the gradient is small near a saddle
point, the training can continue and eventually find better solutions.
What About Optimization Algorithms?
1. Gradient Descent (GD / SGD)
• Works by following the gradient (slope) downward.
• Doesn’t need to find exact points where the gradient is zero.
• Tends to avoid saddle points naturally, though it might slow down near
them.
2. Newton’s Method
• Tries to find where the gradient is exactly zero.
• Problem: If it lands on a saddle point, it might mistakenly treat it like a
minimum and get stuck!
• Solution: A modified version called “saddle-free Newton method” has
been proposed, but it is hard to use for large networks due to its
complexity.
Other Difficult Areas in Optimization
Besides saddle points, there are:
• Local maxima: Also have zero gradient but are rare in high dimensions.
• Flat regions (plateaus): Areas where the cost doesn't change much.
o Gradient = 0
o Hessian = 0
o Hard for algorithms to know where to go.
These flat regions slow learning, especially in general (non-convex)
optimization problems.
Summary: Why Saddle Points Matter

29
 Module 1_BAI701

• In deep learning, saddle points are common and important.

• They make optimization hard because they confuse algorithms—
especially those that rely on curvature (like Newton’s method).
• Stochastic Gradient Descent (SGD) usually handles them well, which is
why it's widely used.
• Researchers are still exploring smarter second-order methods to deal
with saddle points more efficiently.
8.2.4 Cliffs and Exploding Gradients
Problem: Steep Regions or "Cliffs" in Deep Neural Networks
When we train deep neural networks (especially those with many layers or
recurrent neural networks), we often face a major challenge — something
called "cliffs" in the cost function.
What Are These "Cliffs"?
Imagine you're walking on a hilly path, and suddenly, the path drops very
steeply like the edge of a cliff. That’s what happens in the cost landscape of
deep neural networks. These cliffs are areas where the cost (or error) value
changes very suddenly with just a small change in the weights of the
network.
Why does this happen?
👉 Because in deep networks, the output depends on many weight
multiplications. When you multiply large weights over and over, the values
can grow extremely fast, making the cost function very steep in some places.
What Happens Near These Cliffs?
• When gradient descent (the algorithm used to reduce cost) is applied
near such cliffs, it uses the slope (gradient) to decide how much to
change the weights.
• If the slope is very steep, it might suggest a very large step.
• But that large step can jump off the cliff, taking the model into a
completely different region of the cost function.
• This can undo much of the progress made during training — like falling
down a hill after climbing up for hours.
How to Fix This: Gradient Clipping
To prevent this dangerous situation, a simple trick called gradient clipping is
used.
Gradient Clipping:
• It checks how big the gradient step is.

30
 Module 1_BAI701

• If the step is too large, it reduces it to a safe size.

• This ensures the step remains within a small, manageable area —
where the gradient still gives a reliable direction to reduce cost.
Think of it like using a seatbelt — you still move forward, but you’re protected
from moving too far too fast.
Why Recurrent Neural Networks (RNNs) Face This More
RNNs are especially vulnerable to cliffs because:
• They repeat the same operations across many time steps.
• This leads to many weight multiplications over time.
• The more time steps, the steeper the cliffs become.
So for RNNs handling long sequences (like in language modeling or speech
recognition), gradient clipping becomes very important to avoid unstable
training.
Summary
• Deep and recurrent networks can have very steep "cliff-like" regions in
their cost functions.
• These cliffs cause sudden and dangerous jumps in weight values during
training.
• Gradient clipping is a safety method to limit the step size, preventing
the algorithm from "falling off" the cliff.
• This helps ensure stable and effective learning, especially in RNNs.
8.2.5 Long-Term Dependencies
Vanishing and Exploding Gradients in Deep and Recurrent Neural
Networks
When training deep neural networks, especially recurrent neural networks
(RNNs), there's a common problem that makes learning difficult. This problem
is called the vanishing and exploding gradient problem.
Let’s break it down step by step:
What Are Deep Computational Graphs?
• A computational graph shows how inputs flow through operations (like
layers in a neural network) to produce an output.
• In deep feedforward networks, there are many layers, so the graph
becomes very deep.

31
 Module 1_BAI701

• In recurrent neural networks (RNNs), the same operation is repeated for

each time step in a sequence (like processing one word after another in
a sentence). So, even a small RNN can become very deep over time —
one layer repeating many times.
Why Is This a Problem?
Let’s imagine that each layer in the network involves multiplying by a matrix
W.
Now, if you repeat this multiplication over t steps (like in an RNN), it becomes:

This is like repeatedly applying a transformation.

Understanding via Eigenvalues
Suppose we use linear algebra to simplify things.
Any square matrix W can be broken down into:

Where:
• λ are the eigenvalues (they tell us how the matrix stretches space),
• V is a matrix of eigenvectors (directions in space).
If we now multiply W with itself many times:

That means we are raising each eigenvalue to the power of t.

Vanishing vs Exploding Gradients
• If an eigenvalue is less than 1, then λ^t becomes very small as t
increases.
o This causes the vanishing gradient problem — the signal becomes
weaker and weaker.
• If an eigenvalue is greater than 1, then λ^t becomes very large as t
increases.
o This causes the exploding gradient problem — the signal becomes
too strong and unstable.
This is called the vanishing and exploding gradient problem. It happens
because we keep multiplying gradients by the same matrix repeatedly.

32
 Module 1_BAI701

Why Is This a Big Deal?

• Vanishing gradients mean that the model can’t learn long-term
dependencies — it forgets earlier steps in a sequence.
• Exploding gradients make training unstable, jumping around too
much.
• In both cases, it becomes hard to train the network properly.
Where Do These Problems Occur?
• RNNs are most affected because they use the same matrix W repeatedly
for each time step in a sequence.
• Feedforward networks, even when deep, use different matrices in each
layer, so they are less affected.
Interesting Insight
This behavior is similar to a method in math called the Power Method, which
finds the strongest direction (or dominant eigenvector) of a matrix. So over
time, all input directions except the strongest one are forgotten — just like in
vanishing gradients.
What’s the Fix?
We’ll talk more about how to fix this (like using gradient clipping, LSTM units,
or better initialization) later in the chapter on recurrent networks.
Summary for Students
• Deep networks (especially RNNs) suffer from vanishing and exploding
gradients due to repeated matrix multiplication.
• This makes it hard to train the network because the gradients either
become too small or too large.
• It's one of the biggest challenges in training RNNs and requires special
techniques to fix.
8.2.6 Inexact Gradients
Understanding Gradient and Optimization in Deep Learning
Most optimization algorithms — like Gradient Descent — are based on the
idea that we can calculate the exact gradient (which tells us the direction to
improve a model). Some advanced methods also use the Hessian matrix,
which gives more information about how the function curves (like second
derivatives in calculus).
But here’s the catch:
In Real Life, Things Are Not Perfect

33
 Module 1_BAI701

What If the Function Itself Is Too Hard?

Sometimes, the function we want to minimize (called the objective function,
like a loss function) is so complicated that:
• We cannot compute it directly.
• And even worse, we cannot compute its gradient either.
This situation usually comes up in advanced models — for example:
• In Boltzmann Machines (a type of deep model), the objective is based
on log-likelihood, which is often intractable (too hard to compute).
So how do we deal with this?
Approximation Techniques
One method is called Contrastive Divergence. It helps:
• Estimate the gradient of the intractable loss function.
• Even though it’s not exact, it gives us a good enough approximation to
keep learning going.
Designing Smart Optimization Algorithms
Deep learning researchers know that these imperfections exist. So, many
optimization algorithms are designed to work well:
• Even when the gradient is noisy or inaccurate.
• By making the algorithms robust and adaptive.
Use Surrogate Loss Functions
When the true loss is too difficult to work with, another smart idea is to use
a surrogate loss function.
What is it?
A surrogate loss is a simpler version of the actual loss that:
• Is easier to compute.
• Still gives useful direction for learning.
Summary

Concept Simple Meaning

Exact gradient Perfect direction to minimize the loss

Noisy/Biased
Rough estimate due to using mini-batches
gradient

34
 Module 1_BAI701

Concept Simple Meaning

Intractable function Too complicated to compute directly

Contrastive
A trick to approximate gradients for complex models
divergence

A simpler loss that helps guide learning when the real

Surrogate loss
loss is too hard

8.2.7 Poor Correspondence between Local and Global Structure

Why Optimization in Deep Learning Is Hard
Let’s break down the big ideas from the text:
1. Problems at a Single Point in the Loss Function
The loss function (denoted as J(θ)) tells us how "bad" our model is. When we
train a neural network, we want to reduce this loss by adjusting the model’s
parameters (θ).
But sometimes, the function’s shape at a single point can make things hard:
• Poor conditioning: The landscape is too steep in some directions and
too flat in others, making it hard to find a good step size.
• Cliffs: A small step might suddenly cause a huge increase in loss.
• Saddle points: These are flat areas where gradients are small or zero,
even though better values exist nearby.
These problems make it hard to decide how and where to move next.
2. Direction of Movement Might Be Misleading
Even if we manage to overcome the above issues, there’s a bigger picture
problem:
The direction we move in (based on local gradients) might not lead us to the
lowest point (global minimum) of the loss function.
Imagine this:
• You're hiking down a mountain and always choosing the steepest path
(local gradient).
• But you might end up walking around a huge mountain, taking a long,
curved path instead of a direct shortcut through it.
That’s what Figure 8.2 is trying to show — the learning path takes a long
detour around high regions.

35
 Module 1_BAI701

3. It’s Not Always About Local Minima

People often worry about models getting stuck in local minima or saddle
points, but in practice, neural networks often don’t get stuck there.
• In fact, many loss functions don’t have a true minimum.
• For example, in classification using soft max and cross-entropy loss,
the loss keeps getting smaller as the model gets more confident — but
it never exactly reaches zero.
• Similarly, in regression with Gaussian outputs, the model might keep
reducing error forever, trying to make predictions more precise, causing
the loss to go to negative infinity.
So the challenge is not just getting stuck — it's also that the loss surface can
be weird.
4. Why Initialization Matters
Even if there are no local minima or saddle points, you can still get stuck on
the wrong side of the "mountain" (as shown in Figure 8.4).
• If the starting point is far from the best path, gradient-based methods
can never reach the correct solution, because they only take small, local
steps.
• In high-dimensional space (like deep neural networks), the model can
sometimes find a way around the obstacle, but that might take a very
long time, increasing training time.
So, a bad starting point leads to:
• Longer training
• Poor solutions
• Wasted computation

36
 Module 1_BAI701

5. Most Optimizers Use Local Moves

Almost all deep learning optimization algorithms — like Gradient Descent,
Adam, etc. — are based on making small, local steps in the direction that
reduces loss.
But this strategy has its limits:
• We can only estimate the direction approximately using mini-batches,
so we don't even follow the ideal path.
• The area around us might be too flat, jagged, or poorly shaped.
• Our steps (like δ or ε) may be too small to make progress in such
regions.
This results in:
• Many tiny steps being needed
• Slow learning
• Sometimes, no progress at all
6. Future Research Directions
There are two main ideas to make training better:
a. Better Initialization
• If we can start training in a “good” region of the loss landscape, we might
avoid many of these issues.
• So, a lot of research is focusing on how to choose better starting points.
b. Smarter Algorithms
• Instead of just making local moves, can we design optimizers that take
bigger leaps?
• Or use global knowledge of the landscape?
These are still open research questions.
Summary Table

Concept Simple Meaning

Local gradient At one point, the function may be flat, steep, or oddly
problems shaped

Local direction May not lead to the best solution far away

Long learning path The optimizer may take a long route around obstacles

37
 Module 1_BAI701

Concept Simple Meaning

In some models, the loss keeps improving but never

No true minimum
stops

Local optimization Only uses nearby info; can get stuck or slow

Initialization Starting in a good place can help a lot

Future goal Create better starting points and smarter optimizers

8.2.8 Theoretical Limits of Optimization

Why Theory Can’t Always Predict Optimization in Neural Networks
When training neural networks, we often rely on optimization algorithms like
gradient descent to adjust the model's parameters and minimize the loss. But
there’s an important question researchers ask:
Can we prove that these optimization algorithms will always work well?
1. Some Theoretical Results Say: Optimization Is Hard!
Several research papers have shown that:
• There are mathematical limits on how well any optimization algorithm
can perform when training neural networks.
• In some cases, finding the best solution (minimum loss) is
mathematically impossible or takes too long (intractable).

🧠 Examples of such research:

• Blum and Rivest (1992)
• Judd (1989)
• Wolpert and MacReady (1997)
These studies prove that, in some theoretical settings, training neural
networks is very hard.
2. But In Practice, Things Are Different
Even though the theory seems discouraging, these results often don’t match
what we see in real-world training. Why?
Let’s look at the reasons:
3. Theoretical Limits Apply to Special Cases
a. Discrete Outputs:
Some theoretical results apply only when neural networks output discrete
values (like 0 or 1).

38
 Module 1_BAI701

• But in practice, most modern neural networks use continuous (smooth)

activation functions (like ReLU or sigmoid).
• This makes optimization easier because we can apply gradient-based
methods smoothly.
b. Intractable Problem Classes:
Some problems are provably too hard to solve (called intractable).
• But when we train neural networks, we usually don’t know whether our
task belongs to these hard classes.
• So, the theory doesn’t always help us judge practical difficulty.
4. Bigger Networks Make Training Easier
Some theoretical results say it’s hard to find a solution in a small network.
But in real life:
• We often use larger networks.
• Larger networks have more parameters, which gives more flexibility.
• That means there are more possible settings of weights that give a good
enough solution.
So, we can often find acceptable solutions even if perfect solutions are hard
to find.
5. We Don’t Need the Exact Minimum
In theory, optimization is about finding the exact lowest value (global
minimum) of the loss function.
But in deep learning, our goal is different:
We just want the model to perform well enough on new data (called
generalization).
So we don’t need the absolute best solution—just a solution that gives low
enough loss to make good predictions.
6. Theoretical Proofs Are Still Difficult
It’s very hard to mathematically prove that an algorithm will always reduce
the loss enough to achieve good generalization in all situations.
That’s why:
• We still don’t fully understand the theory behind training neural
networks.
• This is an active research area in machine learning.

39
 Module 1_BAI701

Summary Table

Concept Simple Meaning

Some problems are proven to be too hard for

Theoretical limits
optimization algorithms

Discrete vs. smooth Theory often applies to discrete outputs, but real
outputs models use smooth ones

Some problems are unsolvable, but we may never face

Intractable problems
them in practice

More parameters = more chances to find a good

Larger networks
solution

Goal in practice We don’t need a perfect solution, just a good one

We still don’t have complete mathematical

Open research
explanations for training success

40