BAI701, DL & RL

MODULE 1: Introduction to Deep Learning

1.1 Introduction

Machine Learning (ML) is a branch of Artificial Intelligence (AI) that enables systems to learn
from data and make decisions or predictions without being explicitly programmed. It allows
machines to improve their performance with experience. The fundamental goal of ML is to
allow a machine to learn useful information from raw data and improve its prediction or
classification capabilities over time, much like a human.

Types of Machine Learning

Machine learning techniques are broadly categorized into four types:
a) Supervised Learning

Supervised learning adapts a system such that for a given input data it produces a target output.
The learning data is made up of tuples (attributes, label) where “attributes” represent the input
data and “label” represents the target output. The goal here is to adapt the system so that for a
new input the system can predict the target output. Supervised learning can use both continuous
and discrete types of input data.
b) Unsupervised Learning
Unsupervised learning involves data that comprises of input vectors without any target output.
There are different objectives in unsupervised learning, such as clustering, density estimation,
and visualization. The goal of clustering is to discover groups of similar data items on the basis
of measured or perceived similarities between the data items. The purpose of density estimation
is to determine the distribution of the data within the input space. In visualization, the data is
projected down from a high-dimensional space to two or three dimensions to view the similar
data items.
c) Semi-supervised Learning
Semi-supervised learning first uses unlabeled data to learn a feature representation of the input
data and then uses the learned feature representation to solve the supervised task. The training
dataset can be divided into two parts: the data samples with corresponding labels and the data
samples where the labels are not known. Semisupervised learning can involve not providing
with an explicit form of error at each time but only a generalized reinforcement is received
giving indication of how the system should change its behavior, and this is sometimes referred

Mrs. Pooja R Rao, CSE(DS), RNSIT 1

 BAI701, DL & RL

to as reinforcement learning. Reinforcement learning has been successful in applications as

diverse as autonomous helicopter flight, robot legged locomotion, cell-phone network routing,
marketing strategy selection, factory control and efficient webpage indexing.
1.2 Shallow Learning
Shallow architectures are well understood and perform good on many common machine
learning problems, and they are still used in a vast majority of today’s machine learning
applications. However, there has been an increased interest in deep architectures recently, in
the hope to find means to solve more complex real-world problems (e.g., image analysis or
natural language understanding) for which shallow architectures are unable to learn models
adequately.

1.3 Deep Learning

Deep learning is a newarea of machine learning which has gained popularity in recent past.
Deep learning refers to the architectures which contain multiple hidden layers (deep networks)
to learn different features with multiple levels of abstraction. Deep learning algorithms seek to
exploit the unknown structure in the input distribution in order to discover good
representations, often at multiple levels, with higher level learned features defined in terms of
lower level features.

Conventional machine learning methods rely heavily on manual feature extraction,

requiring domain expertise to convert raw data into suitable feature vectors. In traditional
systems, engineers design specific algorithms or extractors to transform raw data (like image
pixels) into usable representations for classifiers.

Deep learning eliminates the need for manual feature extraction by allowing raw data
to be directly fed into the model. Deep learning models automatically learn the right set of
features during training, often more effectively than manually coded ones. Deep learning builds
a hierarchy of features, where simple concepts learned in lower layers are combined into more
complex concepts in higher layers. When processing an image, lower layers detect edges,
intermediate layers detect corners and contours, and higher layers recognize object parts using
these basic shapes. All layers and features are learned automatically from data, not handcrafted
by human engineers. The output layer uses all the learned features from previous layers to
classify the input and produce the final output label. Each layer builds on the outputs of

Mrs. Pooja R Rao, CSE(DS), RNSIT 2

 BAI701, DL & RL

previous layers, forming a structured, layered learning process. Figure 1.1 visually contrasts
traditional ML (with hand-engineered features) versus deep learning (with learned hierarchical
representations).

Fig. 1.1 a Conventional machine learning using hand-designed feature extraction

algorithms. b deep learning approach using hierarchy of representations that are learnt
automatically.

Difference from Conventional Machine Learning

Aspect Conventional ML Deep Learning

Feature Extraction Manual, domain-specific Automatic, learned from data
Input Requires feature vectors Raw data (e.g., pixels, audio, text)
Layer Depth One or two layers (shallow) Multiple layers (deep)
Learning Process Handcrafted feature design Hierarchical feature learning

Connection with Artificial Intelligence

In deep learning, meaningful representations of input data are learned by constructing complex
mappings through a sequence of simpler ones. The term "deep" signifies the use of multiple
successive layers, each learning increasingly abstract representations. The depth of the model
is determined by the number of these layers. Unlike traditional (shallow) machine learning

Mrs. Pooja R Rao, CSE(DS), RNSIT 3

 BAI701, DL & RL

approaches that learn only one or two layers of features, deep learning can involve tens or
even hundreds of layers. Both deep learning and machine learning are subfields of Artificial
Intelligence (AI), and their relationship is visually represented in Figure 1.2.
Fig. 1.2 Relationship between AI, machine learning, and deep learning

In deep learning, successive layers of representations are often learned using submodels,
which are structured as layers stacked on top of each other. Deep learning networks typically
have more layers and parameters, which gives them the potential to represent more complex
inputs effectively.

Evolution of Deep Learning

Although deep learning has existed since the 1980s, it remained unpopular for many years
due to inadequate computational infrastructure (both hardware and software) and the limited
size of available datasets. As traditional neural networks lost popularity, deep networks re-
emerged recently after showing spectacular results in applications like speech recognition and
computer vision. Two major factors that contributed to the success and evolution of deep
learning are:

o Improved computational resources that allow processing of massive datasets and

training of larger models.
o Automatic feature extraction, which reduces the need for manual effort in designing
feature extractors.

Mrs. Pooja R Rao, CSE(DS), RNSIT 4

 BAI701, DL & RL

The term artificial neural networks has its roots in neuroscience, but deep learning networks
are not exact models of the human brain. Deep learning models are only inspired by the
understanding of biological brain functions and are not direct imitations of it. Not all
components of deep learning models are neuroscience-inspired; many are based on empirical
research, mathematical theory, and practical intuition. The neural activity in the human brain
is significantly more complex than what is modeled using artificial neurons.

Learning Mechanism in Deep Networks

The working of a deep learning model begins with feeding the raw input data—such as
pixel values of an image—into the input layer. The data then passes through multiple hidden
layers, each performing mathematical transformations to extract increasingly complex features.
Each layer uses the output of the previous layer to construct higher-level concepts. Finally, the
transformed data reaches the output layer, which provides the final prediction or classification.
For example, if the task is to recognize handwritten digits, the output layer will assign the
image to one of the digit classes from 0 to 9. Figure 1.3 illustrates a deep learning network used
for digit classification, and Figure 1.4 shows how different layers learn progressively abstract
representations of the input digit.

Fig. 1.3 A deep learning network for digit classification

Mrs. Pooja R Rao, CSE(DS), RNSIT 5

 BAI701, DL & RL

Fig. 1.4 Representations learnt by a deep network for digit classification during the first
pass. Network structural changes can be incorporated that result in desired representations
at various layers

While deep learning draws inspiration from the human brain and its neural structure, it
is not an exact biological model. The artificial neural networks used in deep learning are
mathematical constructs that mimic certain aspects of brain function, such as learning from
experience through the adjustment of weights and biases. The learning mechanism in these
models, though effective, is far simpler than the intricate activity of neurons in a biological
brain.
1.4 Why to Use Deep Learning

The choice of features used to represent a dataset significantly impacts the performance of any
machine learning system. In traditional machine learning, domain experts and data scientists
must spend extensive time and effort to manually identify and extract useful features from raw
data. This process is not only time-consuming but also heavily reliant on human expertise.
Moreover, conventional algorithms struggle when dealing with high-dimensional data or
complex tasks, as they lack the flexibility to automatically determine which aspects of the input
are most relevant.

 Deep learning eliminates the need for manual feature extraction by learning directly
from raw data.

Mrs. Pooja R Rao, CSE(DS), RNSIT 6

 BAI701, DL & RL

 It can efficiently process complex inputs such as images, audio, and natural language
that traditional machine learning struggles with. 
 Deep networks often outperform traditional models in accuracy and can generalize well
to unseen data.
 Unlike traditional models that rely heavily on domain expertise, deep learning reduces
human effort by learning features automatically. 
 Deep learning has led to major advances in tasks like speech recognition, image
classification, and machine translation. 
 Its success has been driven by the availability of large datasets and advanced computing
infrastructure.
 Companies like Google, Amazon, Facebook, and Microsoft have successfully deployed
deep learning for real-world applications. 
 Deep learning powers chatbots, autonomous vehicles, digital assistants,
recommendation systems, and more.
 It has achieved results comparable to or better than humans in areas like Go, Shogi, and
speech recognition. 
 Deep learning models can be applied across various domains with minimal
customization, increasing their versatility and value. 
 Deep learning gained prominence after 2007 due to its success in perceptual tasks like
vision and speech, where traditional methods failed.
 It has successfully addressed complex problems that shallow models and earlier ML
methods could not solve. 
 Deep learning is now used in medicine, finance, and manufacturing to guide important
decisions based on data.
 Techniques like regularization and improved optimizers allow deep models with
millions of parameters to be trained effectively. 
 Deep learning is expected to revolutionize fields such as scientific discovery, software
automation, and intelligent systems development when combined with other AI tools. 

Deep learning has made good impact in historically difficult areas of machine learning:
o Near-human-level image classification;
o Near-human-level speech recognition;
o Near-human-level handwriting transcription;

Mrs. Pooja R Rao, CSE(DS), RNSIT 7

 BAI701, DL & RL

o Improved self-driving cars;

o Digital assistants such as Google Now,Microsoft Cortana, Apple’s Siri, and
Amazon Alexa;
o Improved ad targeting, as used by Google, Baidu, and Bing;
o Improved search results on the web;
o Ability to answer natural language questions; and
o Superhuman Go, Shogi, and Chess playing.

In the future, deep learning is expected to support advances in many fields including software
development, scientific discovery, and automation. When combined with other AI
technologies, deep learning may lead to revolutionary breakthroughs, transforming industries
and reshaping the way humans interact with intelligent systems.
1.5 How Deep Learning Works

 Deep learning networks map input data to output targets through a sequence of layered
transformations.
 Each layer applies transformations that are controlled by weights, which are numerical
parameters learned during training.
 Learning in deep networks involves finding the optimal values of these weights so that
the network can correctly map inputs to their corresponding outputs.
 A deep network contains thousands of parameters, and finding the right combination is
challenging because changes in one weight can affect others.
 The network evaluates its performance using a loss function (or objective function),
which measures the difference between the predicted output and the actual target.
 The goal of training is to minimize the loss function by adjusting the weights so that
predictions become more accurate.
 The error or loss value acts as a feedback signal used to update the weights in the right
direction.
 This weight update process is handled by the backpropagation algorithm, which is the
core optimization method in deep learning. 
 Initially, weights are assigned random values, and the network’s output is usually far
from correct, resulting in a high loss. 

Mrs. Pooja R Rao, CSE(DS), RNSIT 8

 BAI701, DL & RL

 As more training examples are fed into the network, weights are iteratively adjusted to
reduce the loss, and the network is considered to have "learned" once the outputs closely
match the expected targets.
1.6 Deep Learning Challenges

 Initially, deep learning was limited by slow training speed and insufficient
computational resources, making it impractical for general use.
 Traditional methods like kernel-based approaches outperformed deep networks due to
these limitations.
 The revival of deep learning was enabled by the growth of GPU computing and the
availability of large labeled datasets.
 Training deep networks remains a complex and ill-posed optimization problem due to
the huge number of parameters involved.
 Researchers have developed solutions like better optimizers, smart initialization
techniques, activation functions with local competition, and skip connections to
improve training.
 Despite these advances, the stacking of many nonlinear transformations still causes
training difficulties and inefficiencies.
 Deep learning models often require large volumes of data, which may not be available
in all application areas.
 There's a need for more flexible models that can learn well even with limited or small
datasets.
 Deep learning networks are generally designed to solve one specific problem, and
adapting them to similar tasks often requires retraining and architectural changes.
 More research is needed to develop models capable of multitasking without the need
for complete reconfiguration or retraining.

1.7 How Learning Differs from Pure Optimization

Optimization plays a crucial role in many aspects of deep learning, particularly in training
neural networks. Among various optimization tasks, neural network training is the most
challenging, often requiring days to months of computation on hundreds of machines for a

Mrs. Pooja R Rao, CSE(DS), RNSIT 9

 BAI701, DL & RL

single model. Due to its complexity and cost, specialized optimization techniques have been
developed specifically for this purpose.

Pure Optimization focuses on gradient-based optimization used to minimize a cost function

J(θ), where θ represents the parameters of the neural network. The cost function typically
includes both a performance metric and regularization terms.

Optimization algorithms used for training deep learning models differ significantly from
traditional optimization approaches. In deep learning, we do not directly optimize the actual
performance measure P, which is defined over the test set and is often intractable. Instead, we
minimize a different, surrogate cost function J(θ), hoping that doing so will indirectly improve
P. This differs from conventional optimization, where the goal is to minimize a known and
directly relevant cost function.

Deep learning cost functions are usually tailored to reflect the structure of machine learning
tasks. A common formulation of the cost function is as an average over the training set,
particularly in supervised learning. This is represented by the empirical distribution p^data ,
and the per-example loss function L evaluates the difference between predicted and true
outputs.

The typical objective function in supervised learning is:

…………..(Eq 1)

Where:

 f(x;θ): the model’s prediction for input x using parameters θ

 y: the true label
 L: the loss function (e.g., mean squared error, cross-entropy)

Ideally, we would prefer to minimize the expected cost over the true data distribution pdata ,
which leads to the theoretical objective:

…................... (Eq 2)

Mrs. Pooja R Rao, CSE(DS), RNSIT 10

 BAI701, DL & RL

However, since the true distribution is unknown, we work with the empirical version based on
available training data. This framework can also be extended to include regularization or to
accommodate unsupervised learning scenarios by modifying the arguments of the loss function
L.

1.7.1 Empirical Risk Minimization

The primary objective of a machine learning algorithm is to minimize the expected

generalization error, also referred to as risk, which is represented by the expected loss over the
true data distribution pdata(x,y). This is described by Equation 8.2:

………(Eq 2)

If the true distribution pdata were known, minimizing risk would be a standard optimization
problem. However, in practice, we do not have access to the true distribution and only observe
a finite training dataset, turning the task into a machine learning problem.

To address this, the common strategy is to minimize the expected loss over the empirical
distribution defined by the training data, which leads to empirical risk minimization (ERM).
The empirical risk is the average loss over the training samples:

……..(Eq 3)

Here:

 m is the number of training examples.

 (xi,yi)are the training samples.

ERM transforms the learning task into an optimization problem based on available data.
However, ERM comes with two major limitations:

1. Overfitting: High-capacity models may memorize training data instead of learning

general patterns.

Mrs. Pooja R Rao, CSE(DS), RNSIT 11

 BAI701, DL & RL

2. Non-differentiability: Many useful loss functions (e.g., 0-1 loss) lack useful gradients,
making gradient-based optimization difficult.

As a result, in deep learning, pure empirical risk minimization is rarely used. Instead, we
optimize alternative, differentiable loss functions that approximate the original objective while
being compatible with optimization algorithms like gradient descent.

1.7.2 Surrogate Loss Functions and Early Stopping

In many machine learning tasks, the true loss function of interest—such as 0-1 classification
error—is difficult or impossible to optimize directly due to its computational intractability,
even for simple models like linear classifiers. To address this, practitioners use a surrogate loss
function, which serves as a proxy that is easier to optimize. A common surrogate is the negative
log-likelihood, which enables the model to estimate class probabilities. By accurately modeling
these probabilities, the system can minimize expected classification errors effectively.

Interestingly, using a surrogate loss can sometimes lead to better learning outcomes. For
instance, when using the log-likelihood surrogate, the model may continue improving its test
set performance even after achieving zero training error. This happens because the model
becomes more robust and confident, further separating class boundaries and extracting more
useful information from the data.

Another key difference between general optimization and optimization in training algorithms
is the stopping condition. In machine learning, training often uses early stopping, where
training halts based on validation performance (e.g., 0-1 loss), not necessarily when the
surrogate loss gradient is near zero. This prevents overfitting and ensures better generalization.
Therefore, training may stop while the surrogate loss still has large gradients, which contrasts
with pure optimization, where convergence typically requires very small gradients.

1.7.3 Batch and Minibatch Algorithms

In machine learning, the objective function often decomposes over individual training
examples, making it feasible to estimate gradients using only a subset of the data—known as a
minibatch—rather than evaluating the entire dataset. This is especially relevant in optimization
problems like maximum likelihood estimation, which in log-space can be expressed as:

Mrs. Pooja R Rao, CSE(DS), RNSIT 12

 BAI701, DL & RL

…….(Eq 4)

This is equivalent to maximizing the expected log-likelihood under the empirical distribution:

……(Eq 5)

The gradient of the cost function J(θ) is also an expectation over the training data:

……..(Eq 6)
Computing this gradient exactly requires evaluating the model on every training example,
which is computationally expensive for large datasets. Hence, in practice, we estimate it using
minibatch sampling, where only a small group of examples is used to compute the average
gradient. This strategy is more efficient and is justified statistically: the standard error of the
mean reduces only as σ/√𝑛, meaning that even a small sample provides a reasonably accurate
gradient estimate.

 Types of Gradient Descent in Optimization

1. Batch Gradient Descent:

o Uses the entire training set for each update.
o Also called deterministic gradient methods.
o Processes all examples simultaneously.
o The term “batch” here refers to full dataset, not a subset.

2. Minibatch Gradient Descent:

o Uses a small subset of training examples (e.g., 32, 64, 128).
o Most deep learning algorithms use this method.
o Originally called minibatch stochastic gradient descent.
o Now commonly referred to as stochastic methods.

3. Stochastic Gradient Descent (SGD):

Mrs. Pooja R Rao, CSE(DS), RNSIT 13

 BAI701, DL & RL

o Updates the model using only one training example at a time.

o Also called online learning, especially when examples come from a continuous
data stream.
o Suitable when data is received sequentially and not from a fixed dataset.
4. Terminology Clarification:
o "Batch size" usually refers to the number of examples in a minibatch, not the
full dataset.
o So, batch gradient descent ≠ batch size in minibatch SGD.

 Factors Influencing Minibatch Size in Deep Learning

1. Gradient Estimation Accuracy

o Larger batches give more accurate gradient estimates.
o However, the improvement is less than linear—doubling the batch size does not
halve the error.
2. Hardware Utilization
o Very small batches underutilize multicore CPUs and GPUs.
o There is a minimum practical batch size below which processing time does not
decrease.

3. Memory Constraints
o When processing batches in parallel, memory usage increases with batch size.
o Hardware memory limits often restrict how large the batch size can be.

4. Hardware Optimization
o Some hardware (especially GPUs) works more efficiently with power-of-2
batch sizes.
o Common efficient sizes: 32, 64, 128, 256.
o For large models, batch size 16 is sometimes used.
5. Regularization Effect of Small Batches
o Smaller batches introduce noise into the training process.
o This can act as a form of regularization, helping reduce overfitting.
o Batch size 1 often gives the lowest generalization error, but:
 Needs a smaller learning rate for stable training.
 Takes more time to train due to more steps required.

Mrs. Pooja R Rao, CSE(DS), RNSIT 14

 BAI701, DL & RL

 Gradient Estimation, Minibatch Use, and Generalization in Deep Learning

1. Algorithm Sensitivity to Batch Size

o Some algorithms are more sensitive to sampling errors due to their use of complex
or higher-order information.
o First-order methods (use gradient g only) are more robust and work well with
smaller batches (~100 samples).
o Second-order methods (use Hessian H and compute updates like H-1g) require
much larger batches (~10,000) to reduce estimation noise.
2. Error Amplification in Second-Order Methods
o If H is poorly conditioned, even small errors in g get amplified, leading to inaccurate
updates.
o Since H is also approximated, the final update H-1g may contain significant errors.
3. Importance of Random Sampling
o For unbiased gradient estimates, samples in a minibatch must be independent.
o Highly ordered datasets (e.g., patient-wise medical records) may result in biased
minibatches.
o Shuffling the dataset before training avoids such bias. For very large datasets, one-
time shuffling is usually sufficient.
4. Parallel & Asynchronous Updates
o Many ML objectives decompose well enough to allow parallel updates using
separate minibatches.
o This forms the basis for asynchronous distributed optimization.
5. SGD and Generalization Error
o Minibatch SGD can minimize the true generalization error as long as training
examples are not reused.
o On the first epoch, minibatches give unbiased gradient estimates.
o From the second epoch onward, estimates become biased, but still help reduce
training error.
6. Generalization Error and Gradient (Discrete Case)
o Generalization error is:

………….( Eq 7)

Mrs. Pooja R Rao, CSE(DS), RNSIT 15

 BAI701, DL & RL

o Exact gradient:

……….(Eq. 8)

7. Unbiased Gradient Estimate from Minibatch

o Given minibatch {x(1),...,x(m)}, gradient estimate:

………(Eq. 9)

o Updating θ using g performs SGD on generalization error.

8. When to Use Multiple Epochs

o Although only the first epoch follows an unbiased gradient, further epochs reduce
training error.
o The trade-off is a potential increase in the gap between training and test error
(overfitting risk).

9. Large Datasets: New Strategy

o In very large datasets, reusing data is inefficient and overfitting is rare.
o It is now common to do only one pass or partial passes through such datasets for
computational efficiency.
o The main challenge becomes underfitting and efficient use of resources.

1.8 Challenges in Neural Network Optimization

Neural network optimization is challenging due to the non-convex nature of the objective
functions. Unlike traditional machine learning where convexity ensures convergence to global
minima, deep learning involves navigating complex landscapes with saddle points, plateaus,
and ill-conditioned regions.

i. Ill-Conditioning

 A key issue even in convex optimization is ill-conditioning of the Hessian matrix H.

 In neural networks, this can lead to slow learning, as SGD steps may increase the
cost function.

Mrs. Pooja R Rao, CSE(DS), RNSIT 16

 BAI701, DL & RL

 The Taylor series expansion for cost function:

 Ill-conditioning becomes problematic when:

>

 Figure 1.5 below Shows gradient norm increasing over training time (left), yet
validation error decreases (right), indicating learning continues despite gradient
growth.

Figure 1.5: Gradient descent often does not arrive at a critical point of any
kind.

ii. Local Minima

 In non-convex functions, many local minima may exist due to weight space symmetry
and non-identifiability.
 Example: Swapping hidden units or scaling input/output weights doesn’t change the
output, leading to many equivalent minima.
 Most local minima in practice are not problematic, as they have low cost.
 A gradient norm test can help identify if the model is stuck in a true minimum.

iii. Saddle Points & Flat Regions

Mrs. Pooja R Rao, CSE(DS), RNSIT 17

 BAI701, DL & RL

 Saddle points are more common than local minima in high-dimensional space.
 At saddle points, the Hessian matrix has both positive and negative eigenvalues.
 In higher dimensions, probability of encountering saddle points grows exponentially.
 Figure 1.6 Shows cost surface near a saddle point and the SGD trajectory curving
around it, avoiding the trap.
 Second-order methods like Newton’s method are more likely to get stuck at saddle
points without modification (e.g., Saddle-Free Newton).

Figure 1.6: A visualization of the cost function of a neural network.

iv. Cliffs & Exploding Gradients

 Sharp cliffs in the cost function arise due to large weight multiplications.
 Gradient descent can jump off these cliffs, undoing previous learning.
 Solution: Use Gradient Clipping—limit the size of update steps to stay within safe
descent regions.
 Figure 1.7 Illustrates how steep nonlinearities cause instability in deep/recurrent
networks.

Mrs. Pooja R Rao, CSE(DS), RNSIT 18

 BAI701, DL & RL

Figure 1.7: The objective function for highly nonlinear deep neural networks

v. Long-Term Dependencies

 In very deep networks or RNNs, repeated matrix multiplications cause:

 Exploding gradients: when eigenvalues ∣λ∣>1
 Vanishing gradients: when ∣λ∣<1
 Equation:

This affects the learning signal across time steps or layers.

 Feedforward networks are less affected; RNNs are highly susceptible.

vi. Inexact Gradients

 Gradients are often approximated using minibatches, leading to noise.

 Some loss functions are intractable (e.g., Boltzmann machine’s log-likelihood).
 Contrastive Divergence is used to approximate intractable gradients.
 Surrogate loss functions simplify training by approximating the true objective.

vii. Poor Local-Global Correspondence

 The locally optimal direction may not lead to a globally optimal solution.

Mrs. Pooja R Rao, CSE(DS), RNSIT 19

 BAI701, DL & RL

 Much of training time is spent circumnavigating obstacles (e.g., plateaus, mountains).

 Figure 1.8, shows an example of being stuck on the wrong side of a “mountain”; no
minima or saddle points, but difficult to cross.
 Emphasizes the need for good initialization to start in regions from where local descent
is effective.

Figure 1.8: Optimization based on local downhill moves can fail if the local surface
does not point toward the global solution.

viii. Theoretical Limits of Optimization

 Some optimization problems are provably intractable (NP-hard).

 However, these limits are often not practical constraints:
o Neural networks use smooth outputs, allowing local search.
o Overparameterization helps in finding acceptable solutions more easily.
 Theoretical bounds are hard to apply; instead, empirical techniques and heuristics
are used to guide optimization.

Mrs. Pooja R Rao, CSE(DS), RNSIT 20