BAI701, DL & RL

MODULE 1: Introduction to Deep Learning

1.1 Introduction

Machine Learning (ML) is a branch of Artificial Intelligence (AI) that enables systems to
learn from data and make decisions or predictions without being explicitly programmed. It
allows machines to improve their performance with experience. The fundamental goal of ML
is to allow a machine to learn useful information from raw data and improve its prediction or
classification capabilities over time, much like a human.

Types of Machine Learning

Machine learning techniques are broadly categorized into four types:
a) Supervised Learning
 Definition: In supervised learning, the algorithm is trained on a labeled dataset, which
means each training example is paired with the correct output.
 Data Format: Input-output pairs (x, y), where 'x' is the input data and 'y' is the label.
 Objective: Learn a mapping from input to output so that the model can predict the
output for new inputs.
 Applications: Email spam detection, image classification, fraud detection.
 Example: Classifying images of cats and dogs using labeled photos.
b) Unsupervised Learning
 Definition: In unsupervised learning, the model is trained on data that has no labels.
 Objective: Discover hidden patterns or structures in the input data.
 Tasks:
o Clustering: Grouping similar data points.
o Density Estimation: Estimating the probability distribution of data.
o Visualization: Projecting high-dimensional data to lower dimensions for
interpretation.
 Applications: Customer segmentation, anomaly detection, topic modeling.
 Example: Grouping news articles based on content similarity.
c) Semi-supervised Learning
 Definition: Combines both labeled and unlabeled data during training.
 Objective: Use a small amount of labeled data to guide the learning process for a
larger set of unlabeled data.

DR. SHRUTHI U, AI&ML, RNSIT 1

 BAI701, DL & RL

 Advantage: Reduces the cost of labeling large datasets while achieving high
performance.
 Example: Identifying emotions in facial images when only some are labeled.

d) Reinforcement Learning

 Definition: The model learns to make decisions by interacting with an environment

and receiving feedback in the form of rewards or penalties.
 Objective: Maximize cumulative reward over time by learning the best sequence of
actions.
 Applications: Game playing, robotics, navigation, self-driving cars.
 Example: A robot learns to walk through trial and error.

1.2 Shallow Learning

Shallow architectures, which typically consist of one or two layers, are well studied and work
effectively for many basic machine learning tasks. They are still widely used in most machine
learning applications today because of their simplicity and efficiency. However, for more
complex real-world problems like image analysis and natural language processing, shallow
models often fail to capture the deeper patterns in the data. This has led to a growing interest
in deep architectures, which are better suited for handling such complex tasks due to their
ability to learn multiple levels of representation.
1.3 Deep Learning

Deep learning is a recent and rapidly growing area within machine learning that focuses on
using architectures with multiple hidden layers, known as deep networks. These networks are
designed to automatically learn useful features and patterns from data at various levels of
abstraction. Unlike traditional methods, deep learning aims to uncover the hidden structures
within the input data and build hierarchical representations, where complex features at higher
levels are formed from simpler features learned at lower levels.
Conventional machine learning methods rely heavily on manual feature extraction,
requiring domain expertise to convert raw data into suitable feature vectors. In traditional
systems, engineers design specific algorithms or extractors to transform raw data (like image
pixels) into usable representations for classifiers.

DR. SHRUTHI U, AI&ML, RNSIT 2

 BAI701, DL & RL

Deep learning eliminates the need for manual feature extraction by allowing raw data
to be directly fed into the model. Deep learning models automatically learn the right set of
features during training, often more effectively than manually coded ones. Deep learning
builds a hierarchy of features, where simple concepts learned in lower layers are combined
into more complex concepts in higher layers. When processing an image, lower layers detect
edges, intermediate layers detect corners and contours, and higher layers recognize object
parts using these basic shapes. All layers and features are learned automatically from data,
not handcrafted by human engineers. The output layer uses all the learned features from
previous layers to classify the input and produce the final output label. Each layer builds on
the outputs of previous layers, forming a structured, layered learning process. Figure 1.1
visually contrasts traditional ML (with hand-engineered features) versus deep learning (with
learned hierarchical representations).

Fig. 1.1 a Conventional machine learning using hand-designed feature extraction

algorithms. b deep learning approach using hierarchy of representations that are learnt
automatically.

DR. SHRUTHI U, AI&ML, RNSIT 3

BAI701, DL & RL

Difference from Conventional Machine Learning

Aspect Conventional ML Deep Learning

Feature Extraction Manual, domain-specific Automatic, learned from data
Input Requires feature vectors Raw data (e.g., pixels, audio, text)
Layer Depth One or two layers (shallow) Multiple layers (deep)
Learning Process Handcrafted feature design Hierarchical feature learning

Connection with Artificial Intelligence

In deep learning, meaningful representations of input data are learned by constructing
complex mappings through a sequence of simpler ones. The term "deep" signifies the use of
multiple successive layers, each learning increasingly abstract representations. The depth of
the model is determined by the number of these layers. Unlike traditional (shallow) machine
learning approaches that learn only one or two layers of features, deep learning can involve
tens or even hundreds of layers. Both deep learning and machine learning are subfields of
Artificial Intelligence (AI), and their relationship is visually represented in Figure 1.2.

Fig. 1.2 Relationship between AI, machine learning, and deep learning
In deep learning, successive layers of representations are often learned using submodels,
which are structured as layers stacked on top of each other. Deep learning networks typically
have more layers and parameters, which gives them the potential to represent more complex
inputs effectively.

DR. SHRUTHI U, AI&ML, RNSIT 4

 BAI701, DL & RL

Evolution of Deep Learning

Although deep learning has existed since the 1980s, it remained unpopular for many years
due to inadequate computational infrastructure (both hardware and software) and the limited
size of available datasets. As traditional neural networks lost popularity, deep networks re-
emerged recently after showing spectacular results in applications like speech recognition and
computer vision. Two major factors that contributed to the success and evolution of deep
learning are:

o Improved computational resources that allow processing of massive datasets and

training of larger models.
o Automatic feature extraction, which reduces the need for manual effort in designing
feature extractors.

The term artificial neural networks has its roots in neuroscience, but deep learning
networks are not exact models of the human brain. Deep learning models are only inspired by
the understanding of biological brain functions and are not direct imitations of it. Not all
components of deep learning models are neuroscience-inspired; many are based on empirical
research, mathematical theory, and practical intuition. The neural activity in the human brain
is significantly more complex than what is modeled using artificial neurons.

Learning Mechanism in Deep Networks

The working of a deep learning model begins with feeding the raw input data—such
as pixel values of an image—into the input layer. The data then passes through multiple
hidden layers, each performing mathematical transformations to extract increasingly complex
features. Each layer uses the output of the previous layer to construct higher-level concepts.
Finally, the transformed data reaches the output layer, which provides the final prediction or
classification. For example, if the task is to recognize handwritten digits, the output layer will
assign the image to one of the digit classes from 0 to 9. Figure 1.3 illustrates a deep learning
network used for digit classification, and Figure 1.4 shows how different layers learn
progressively abstract representations of the input digit.

DR. SHRUTHI U, AI&ML, RNSIT 5

 BAI701, DL & RL

Fig. 1.3 A deep learning network for digit classification

Fig. 1.4 Representations learnt by a deep network for digit classification during the first
pass. Network structural changes can be incorporated that result in desired representations
at various layers

While deep learning draws inspiration from the human brain and its neural structure,
it is not an exact biological model. The artificial neural networks used in deep learning are
mathematical constructs that mimic certain aspects of brain function, such as learning from
experience through the adjustment of weights and biases. The learning mechanism in these
models, though effective, is far simpler than the intricate activity of neurons in a biological
brain.

DR. SHRUTHI U, AI&ML, RNSIT 6

 BAI701, DL & RL

1.4 Why to Use Deep Learning

The choice of features used to represent a dataset significantly impacts the performance of
any machine learning system. In traditional machine learning, domain experts and data
scientists must spend extensive time and effort to manually identify and extract useful
features from raw data. This process is not only time-consuming but also heavily reliant on
human expertise. Moreover, conventional algorithms struggle when dealing with high-
dimensional data or complex tasks, as they lack the flexibility to automatically determine
which aspects of the input are most relevant.

 Deep learning eliminates the need for manual feature extraction by learning directly
from raw data.
 It can efficiently process complex inputs such as images, audio, and natural language
that traditional machine learning struggles with.
 Deep networks often outperform traditional models in accuracy and can generalize
well to unseen data.
 Unlike traditional models that rely heavily on domain expertise, deep learning reduces
human effort by learning features automatically.
 Deep learning has led to major advances in tasks like speech recognition, image
classification, and machine translation.
 Its success has been driven by the availability of large datasets and advanced
computing infrastructure.
 Companies like Google, Amazon, Facebook, and Microsoft have successfully
deployed deep learning for real-world applications.
 Deep learning powers chatbots, autonomous vehicles, digital assistants,
recommendation systems, and more.
 It has achieved results comparable to or better than humans in areas like Go, Shogi,
and speech recognition.
 Deep learning models can be applied across various domains with minimal
customization, increasing their versatility and value.
 Deep learning gained prominence after 2007 due to its success in perceptual tasks like
vision and speech, where traditional methods failed.
 It has successfully addressed complex problems that shallow models and earlier ML
methods could not solve.

DR. SHRUTHI U, AI&ML, RNSIT 7

 BAI701, DL & RL

 Deep learning is now used in medicine, finance, and manufacturing to guide

important decisions based on data.
 Techniques like regularization and improved optimizers allow deep models with
millions of parameters to be trained effectively.
 Deep learning is expected to revolutionize fields such as scientific discovery, software
automation, and intelligent systems development when combined with other AI tools.

In the future, deep learning is expected to support advances in many fields including software
development, scientific discovery, and automation. When combined with other AI
technologies, deep learning may lead to revolutionary breakthroughs, transforming industries
and reshaping the way humans interact with intelligent systems.

1.5 How Deep Learning Works

 Deep learning networks map input data to output targets through a sequence of
layered transformations.
 Each layer applies transformations that are controlled by weights, which are
numerical parameters learned during training.
 Learning in deep networks involves finding the optimal values of these weights so
that the network can correctly map inputs to their corresponding outputs.
 A deep network contains thousands of parameters, and finding the right combination
is challenging because changes in one weight can affect others.
 The network evaluates its performance using a loss function (or objective function),
which measures the difference between the predicted output and the actual target.
 The goal of training is to minimize the loss function by adjusting the weights so that
predictions become more accurate.
 The error or loss value acts as a feedback signal used to update the weights in the
right direction.
 This weight update process is handled by the backpropagation algorithm, which is the
core optimization method in deep learning.
 Initially, weights are assigned random values, and the network’s output is usually far
from correct, resulting in a high loss.

DR. SHRUTHI U, AI&ML, RNSIT 8

 BAI701, DL & RL

 As more training examples are fed into the network, weights are iteratively adjusted to
reduce the loss, and the network is considered to have "learned" once the outputs
closely match the expected targets.
1.6 Deep Learning Challenges

 Initially, deep learning was limited by slow training speed and insufficient
computational resources, making it impractical for general use.
 Traditional methods like kernel-based approaches outperformed deep networks due to
these limitations.
 The revival of deep learning was enabled by the growth of GPU computing and the
availability of large labeled datasets.
 Training deep networks remains a complex and ill-posed optimization problem due to
the huge number of parameters involved.
 Researchers have developed solutions like better optimizers, smart initialization
techniques, activation functions with local competition, and skip connections to
improve training.
 Despite these advances, the stacking of many nonlinear transformations still causes
training difficulties and inefficiencies.
 Deep learning models often require large volumes of data, which may not be available
in all application areas.
 There's a need for more flexible models that can learn well even with limited or small
datasets.
 Deep learning networks are generally designed to solve one specific problem, and
adapting them to similar tasks often requires retraining and architectural changes.
 More research is needed to develop models capable of multitasking without the need
for complete reconfiguration or retraining.

1.7 How Learning Differs from Pure Optimization

Optimization plays a crucial role in many aspects of deep learning, particularly in training
neural networks. Among various optimization tasks, neural network training is the most
challenging, often requiring days to months of computation on hundreds of machines for a
single model. Due to its complexity and cost, specialized optimization techniques have been
developed specifically for this purpose.

DR. SHRUTHI U, AI&ML, RNSIT 9

 BAI701, DL & RL

Pure Optimization focuses on gradient-based optimization used to minimize a cost

function J(θ), where θ represents the parameters of the neural network. The cost function
typically includes both a performance metric and regularization terms.

Optimization algorithms used for training deep learning models differ significantly from
traditional optimization approaches. In deep learning, we do not directly optimize the actual
performance measure P, which is defined over the test set and is often intractable. Instead, we
minimize a different, surrogate cost function J(θ), hoping that doing so will indirectly
improve P. This differs from conventional optimization, where the goal is to minimize a
known and directly relevant cost function.

Deep learning cost functions are usually tailored to reflect the structure of machine learning
tasks. A common formulation of the cost function is as an average over the training set,
particularly in supervised learning. This is represented by the empirical distribution p^data ,
and the per-example loss function L evaluates the difference between predicted and true
outputs.

The typical objective function in supervised learning is:

…………..(Eq 1)

Where:

 f(x;θ): the model’s prediction for input x using parameters θ

 y: the true label
 L: the loss function (e.g., mean squared error, cross-entropy)

Ideally, we would prefer to minimize the expected cost over the true data distribution pdata ,
which leads to the theoretical objective:

………………(Eq 2)

However, since the true distribution is unknown, we work with the empirical version based
on available training data. This framework can also be extended to include regularization or

DR. SHRUTHI U, AI&ML, RNSIT 10

 BAI701, DL & RL

to accommodate unsupervised learning scenarios by modifying the arguments of the loss

function L.

1.7.1 Empirical Risk Minimization

The primary objective of a machine learning algorithm is to minimize the expected

generalization error, also referred to as risk, which is represented by the expected loss over
the true data distribution pdata(x,y). This is described by Equation 8.2:

………(Eq 2)

If the true distribution pdata were known, minimizing risk would be a standard optimization
problem. However, in practice, we do not have access to the true distribution and only
observe a finite training dataset, turning the task into a machine learning problem.

To address this, the common strategy is to minimize the expected loss over the empirical
distribution defined by the training data, which leads to empirical risk minimization (ERM).
The empirical risk is the average loss over the training samples:

……..(Eq 3)

Here:

 m is the number of training examples.

 (xi,yi)are the training samples.

ERM transforms the learning task into an optimization problem based on available data.
However, ERM comes with two major limitations:

1. Overfitting: High-capacity models may memorize training data instead of learning

general patterns.
2. Non-differentiability: Many useful loss functions (e.g., 0-1 loss) lack useful gradients,
making gradient-based optimization difficult.

DR. SHRUTHI U, AI&ML, RNSIT 11

 BAI701, DL & RL

As a result, in deep learning, pure empirical risk minimization is rarely used. Instead, we
optimize alternative, differentiable loss functions that approximate the original objective
while being compatible with optimization algorithms like gradient descent.

1.7.2 Surrogate Loss Functions and Early Stopping

In many machine learning tasks, the true loss function of interest—such as 0-1 classification
error—is difficult or impossible to optimize directly due to its computational intractability,
even for simple models like linear classifiers. To address this, practitioners use a surrogate
loss function, which serves as a proxy that is easier to optimize. A common surrogate is the
negative log-likelihood, which enables the model to estimate class probabilities. By
accurately modeling these probabilities, the system can minimize expected classification
errors effectively.

Interestingly, using a surrogate loss can sometimes lead to better learning outcomes. For
instance, when using the log-likelihood surrogate, the model may continue improving its test
set performance even after achieving zero training error. This happens because the model
becomes more robust and confident, further separating class boundaries and extracting more
useful information from the data.

Another key difference between general optimization and optimization in training algorithms
is the stopping condition. In machine learning, training often uses early stopping, where
training halts based on validation performance (e.g., 0-1 loss), not necessarily when the
surrogate loss gradient is near zero. This prevents overfitting and ensures better
generalization. Therefore, training may stop while the surrogate loss still has large gradients,
which contrasts with pure optimization, where convergence typically requires very small
gradients.

1.7.3 Batch and Minibatch Algorithms

In machine learning, the objective function often decomposes over individual training
examples, making it feasible to estimate gradients using only a subset of the data—known as
a minibatch—rather than evaluating the entire dataset. This is especially relevant in
optimization problems like maximum likelihood estimation, which in log-space can be
expressed as:

DR. SHRUTHI U, AI&ML, RNSIT 12

 BAI701, DL & RL

…….(Eq 4)

This is equivalent to maximizing the expected log-likelihood under the empirical distribution:

……(Eq 5)

The gradient of the cost function J(θ) is also an expectation over the training data:

……..(Eq 6)

Computing this gradient exactly requires evaluating the model on every training example,
which is computationally expensive for large datasets. Hence, in practice, we estimate it
using minibatch sampling, where only a small group of examples is used to compute the
average gradient. This strategy is more efficient and is justified statistically: the standard
error of the mean reduces only as σ/√𝑛, meaning that even a small sample provides a
reasonably accurate gradient estimate.

 Types of Gradient Descent in Optimization

1. Batch Gradient Descent:

o Uses the entire training set for each update.
o Also called deterministic gradient methods.
o Processes all examples simultaneously.
o The term “batch” here refers to full dataset, not a subset.

2. Minibatch Gradient Descent:

o Uses a small subset of training examples (e.g., 32, 64, 128).
o Most deep learning algorithms use this method.
o Originally called minibatch stochastic gradient descent.
o Now commonly referred to as stochastic methods.

3. Stochastic Gradient Descent (SGD):

o Updates the model using only one training example at a time.

DR. SHRUTHI U, AI&ML, RNSIT 13

 BAI701, DL & RL

o Also called online learning, especially when examples come from a

continuous data stream.
o Suitable when data is received sequentially and not from a fixed dataset.
4. Terminology Clarification:
o "Batch size" usually refers to the number of examples in a minibatch, not the
full dataset.
o So, batch gradient descent ≠ batch size in minibatch SGD.

 Factors Influencing Minibatch Size in Deep Learning

1. Gradient Estimation Accuracy

o Larger batches give more accurate gradient estimates.
o However, the improvement is less than linear—doubling the batch size does
not halve the error.
2. Hardware Utilization
o Very small batches underutilize multicore CPUs and GPUs.
o There is a minimum practical batch size below which processing time does not
decrease.

3. Memory Constraints
o When processing batches in parallel, memory usage increases with batch size.
o Hardware memory limits often restrict how large the batch size can be.

4. Hardware Optimization
o Some hardware (especially GPUs) works more efficiently with power-of-2
batch sizes.
o Common efficient sizes: 32, 64, 128, 256.
o For large models, batch size 16 is sometimes used.
5. Regularization Effect of Small Batches
o Smaller batches introduce noise into the training process.
o This can act as a form of regularization, helping reduce overfitting.
o Batch size 1 often gives the lowest generalization error, but:
 Needs a smaller learning rate for stable training.
 Takes more time to train due to more steps required.

DR. SHRUTHI U, AI&ML, RNSIT 14

 BAI701, DL & RL

 Gradient Estimation, Minibatch Use, and Generalization in Deep Learning

1. Algorithm Sensitivity to Batch Size

o Some algorithms are more sensitive to sampling errors due to their use of complex
or higher-order information.
o First-order methods (use gradient g only) are more robust and work well with
smaller batches (~100 samples).
o Second-order methods (use Hessian H and compute updates like H-1g) require
much larger batches (~10,000) to reduce estimation noise.
2. Error Amplification in Second-Order Methods
o If H is poorly conditioned, even small errors in g get amplified, leading to
inaccurate updates.
o Since H is also approximated, the final update H-1g may contain significant errors.
3. Importance of Random Sampling
o For unbiased gradient estimates, samples in a minibatch must be independent.
o Highly ordered datasets (e.g., patient-wise medical records) may result in biased
minibatches.
o Shuffling the dataset before training avoids such bias. For very large datasets, one-
time shuffling is usually sufficient.
4. Parallel & Asynchronous Updates
o Many ML objectives decompose well enough to allow parallel updates using
separate minibatches.
o This forms the basis for asynchronous distributed optimization.
5. SGD and Generalization Error
o Minibatch SGD can minimize the true generalization error as long as training
examples are not reused.
o On the first epoch, minibatches give unbiased gradient estimates.
o From the second epoch onward, estimates become biased, but still help reduce
training error.
6. Generalization Error and Gradient (Discrete Case)
o Generalization error is:

………….( Eq 7)

DR. SHRUTHI U, AI&ML, RNSIT 15

 BAI701, DL & RL

o Exact gradient:

……….(Eq. 8)

7. Unbiased Gradient Estimate from Minibatch

o Given minibatch {x(1),...,x(m)}, gradient estimate:

………(Eq. 9)

o Updating θ using g performs SGD on generalization error.

8. When to Use Multiple Epochs

o Although only the first epoch follows an unbiased gradient, further epochs reduce
training error.
o The trade-off is a potential increase in the gap between training and test error
(overfitting risk).

9. Large Datasets: New Strategy

o In very large datasets, reusing data is inefficient and overfitting is rare.
o It is now common to do only one pass or partial passes through such datasets for
computational efficiency.
o The main challenge becomes underfitting and efficient use of resources.

1.8 Challenges in Neural Network Optimization

Neural network optimization is challenging due to the non-convex nature of the objective
functions. Unlike traditional machine learning where convexity ensures convergence to
global minima, deep learning involves navigating complex landscapes with saddle points,
plateaus, and ill-conditioned regions.

i. Ill-Conditioning

 A key issue even in convex optimization is ill-conditioning of the Hessian matrix

H.

DR. SHRUTHI U, AI&ML, RNSIT 16

 BAI701, DL & RL

 In neural networks, this can lead to slow learning, as SGD steps may increase the
cost function.
 The Taylor series expansion for cost function:

 Ill-conditioning becomes problematic when:

>

 Figure 1.5 below Shows gradient norm increasing over training time (left), yet
validation error decreases (right), indicating learning continues despite gradient
growth.

Figure 1.5: Gradient descent often does not arrive at a critical point of any
kind.

ii. Local Minima

 In non-convex functions, many local minima may exist due to weight space symmetry
and non-identifiability.
 Example: Swapping hidden units or scaling input/output weights doesn’t change the
output, leading to many equivalent minima.
 Most local minima in practice are not problematic, as they have low cost.
 A gradient norm test can help identify if the model is stuck in a true minimum.
DR. SHRUTHI U, AI&ML, RNSIT 17
 BAI701, DL & RL

iii. Saddle Points & Flat Regions

 Saddle points are more common than local minima in high-dimensional space.
 At saddle points, the Hessian matrix has both positive and negative eigenvalues.
 In higher dimensions, probability of encountering saddle points grows exponentially.
 Figure 1.6 Shows cost surface near a saddle point and the SGD trajectory curving
around it, avoiding the trap.
 Second-order methods like Newton’s method are more likely to get stuck at saddle
points without modification (e.g., Saddle-Free Newton).

Figure 1.6: A visualization of the cost function of a neural network.

iv. Cliffs & Exploding Gradients

 Sharp cliffs in the cost function arise due to large weight multiplications.
 Gradient descent can jump off these cliffs, undoing previous learning.
 Solution: Use Gradient Clipping—limit the size of update steps to stay within safe
descent regions.
 Figure 1.7 Illustrates how steep nonlinearities cause instability in deep/recurrent
networks.

DR. SHRUTHI U, AI&ML, RNSIT 18

 BAI701, DL & RL

Figure 1.7: The objective function for highly nonlinear deep neural networks

v. Long-Term Dependencies

 In very deep networks or RNNs, repeated matrix multiplications cause:

 Exploding gradients: when eigenvalues ∣λ∣>1
 Vanishing gradients: when ∣λ∣<1
 Equation:

This affects the learning signal across time steps or layers.

 Feedforward networks are less affected; RNNs are highly susceptible.

vi. Inexact Gradients

 Gradients are often approximated using minibatches, leading to noise.

 Some loss functions are intractable (e.g., Boltzmann machine’s log-likelihood).
 Contrastive Divergence is used to approximate intractable gradients.
 Surrogate loss functions simplify training by approximating the true objective.

vii. Poor Local-Global Correspondence

 The locally optimal direction may not lead to a globally optimal solution.

DR. SHRUTHI U, AI&ML, RNSIT 19

 BAI701, DL & RL

 Much of training time is spent circumnavigating obstacles (e.g., plateaus, mountains).

 Figure 1.8, shows an example of being stuck on the wrong side of a “mountain”; no
minima or saddle points, but difficult to cross.
 Emphasizes the need for good initialization to start in regions from where local descent
is effective.

Figure 1.8: Optimization based on local downhill moves can fail if the local
surface does not point toward the global solution.

viii. Theoretical Limits of Optimization

 Some optimization problems are provably intractable (NP-hard).

 However, these limits are often not practical constraints:
o Neural networks use smooth outputs, allowing local search.
o Overparameterization helps in finding acceptable solutions more easily.
 Theoretical bounds are hard to apply; instead, empirical techniques and heuristics
are used to guide optimization.

DR. SHRUTHI U, AI&ML, RNSIT 20