lOMoARcPSD|56847386

DL unit 1 - DL unit 1

computer science (Anurag University)

Scan to open on Studocu

Studocu is not sponsored or endorsed by any college or university

Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

SIDDHARTHA INSTITUTE OF ENGINEERING & TECHNOLOGY

DEPARTMENT OF AI & ML
DEEP LEARNING
IV - YEAR I SEM

UNIT - I

LEARNING ALGORITHMS IN MACHINE LEARNING

Machine learning algorithms are the core of artificial intelligence, allowing computers
to learn from data, make predictions, and adapt without being explicitly programmed.
These algorithms fall into different categories, each with its own strengths and
weaknesses depending on the type of problem and data available.

1. Supervised learning

Supervised learning is a branch of machine learning that focuses on training models

using labelled datasets, where each input data point is associated with a known output
label. This process allows algorithms to learn relationships between inputs and outputs,
enabling them to make predictions or classify new, unseen data. Supervised learning is
commonly categorized into two main types: regression and classification.:

• Regression: Predicting continuous values, like house prices based on features

such as square footage and location.

• Classification: Categorizing data into predefined classes, for example, classifying

emails as spam or not spam based on content and sender information.

Deep Learning 1
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

Common supervised learning algorithms include:

Regression Algorithms (for predicting continuous values):

• Linear Regression:
A foundational algorithm that models the relationship between a dependent
variable and one or more independent variables by fitting a linear equation to the
observed data.
• Polynomial Regression:
Extends linear regression by allowing for a non-linear relationship between
variables, fitting a polynomial equation.
• Ridge, Lasso, and Elastic Net Regression:
Regularized versions of linear regression that help prevent overfitting by adding
penalty terms to the cost function.
• Support Vector Regression (SVR):
An extension of Support Vector Machines for regression tasks, aiming to find a
hyperplane that best fits the data within a specified margin.
• Decision Tree Regression:
Uses a tree-like model of decisions and their possible consequences to predict
continuous outcomes.
• Random Forest Regression:
An ensemble method that combines multiple decision trees to improve predictive
accuracy and reduce overfitting.
• Gradient Boosting Regression (e.g., XGBoost, LightGBM, CatBoost):
Powerful ensemble methods that build a strong predictive model by sequentially
adding weak learners.

Deep Learning 2
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

Classification Algorithms (for predicting categorical values):

• Logistic Regression:
A widely used algorithm for binary classification, modeling the probability of a
binary outcome.
• Support Vector Machines (SVM):
Creates a hyperplane in a high-dimensional space to separate data points into
different classes.
• Decision Tree Classification:
Similar to decision tree regression, but used for classification tasks, where leaf
nodes represent class labels.
• Random Forest Classification:
An ensemble method for classification, combining multiple decision trees.
• K-Nearest Neighbors (KNN):
A non-parametric algorithm that classifies a data point based on the majority class
among its 'k' nearest neighbors in the feature space.
• Naive Bayes:
A probabilistic classifier based on Bayes' theorem with the assumption of
independence between features.
• Neural Networks (e.g., Multilayer Perceptron):
Inspired by the human brain, these algorithms learn complex patterns through
interconnected layers of nodes.
• Gradient Boosting Machines (GBM):
Ensemble methods like AdaBoost, XGBoost, LightGBM, and CatBoost are also
highly effective for classification.

2. Unsupervised learning:

Unsupervised learning algorithms analyze unlabeled data to find patterns without

predefined outputs. This is helpful when the desired outcome is unknown, allowing for
exploration of the data's structure. Applications include clustering and dimensionality
reduction. Common algorithms include K-Means Clustering and Principal Component
Analysis (PCA) and DBSCAN clustering.

Clustering :

• K-Means Clustering: Groups data into 'k' clusters based on how close the points
are to each other. The algorithm iteratively assigns data points to the closest
cluster centroid and recalculates the centroid's position until the clusters are
stable.

• Hierarchical Clustering: Creates a tree-like structure (dendrogram) by either

merging small clusters or splitting larger clusters based on their similarities. It
can be either agglomerative (bottom-up) or divisive (top-down).

• DBSCAN (Density-Based Spatial Clustering of Applications with

Noise): Forms clusters in dense areas of the data, treating scattered points as
noise or outliers.

Deep Learning 3
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

• Mean-Shift Clustering: Discovers clusters by moving data points towards the

most crowded areas, effectively finding modes (peaks) in the data density.

• Spectral Clustering: Groups data by analyzing connections between points

using graphs, which can uncover complex, non-linear relationships.

• Gaussian Mixture Models (GMMs): Models clusters as overlapping Gaussian

distributions, assigning probabilities to data points for their cluster membership.

• Affinity Propagation: Takes the input data points as a measure of similarity

between two data points, then messages are exchanged between data points to
determine the exemplars (representative points) that best summarize the dataset.

• BIRCH (Balanced Iterative Reducing and Clustering using

Hierarchies): Constructs a clustering feature (CF) tree to summarize the data
and then performs clustering on the summarized data.

• OPTICS (Ordering Points To Identify the Clustering Structure): An

algorithm derived from DBSCAN, that helps to find density-based clusters in
spatial data.

3. Reinforcement learning :

Reinforcement learning trains agents to make decisions by using rewards and penalties.
The goal is to maximize long-term rewards through trial and error. Examples include
robotics and game AI. Algorithms like Q-Learning, R-Learning and Policy Gradient
Methods are used.

4. Semi-supervised learning:

This approach combines supervised and unsupervised learning, using both labeled and
unlabeled data, which is useful when labeled data is limited. Applications include
improving text and image classification models.

Choosing the right algorithm :

Selecting an algorithm depends on the problem type, data characteristics, computational

resources, and desired accuracy. Experimenting with different algorithms is
recommended.

Deep Learning 4
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

CAPACITY, OVERFITTING, AND UNDERFITTING IN MACHINE

LEARNING

• Capacity: A model's capacity refers to its ability to fit a wide range of functions.
High-capacity models can fit more complex relationships, while low-capacity
models are limited to simpler ones.

In machine learning, capacity is a way to measure a model's ability to learn and

find patterns from data. You can think of it as the model's complexity or
flexibility.

An Easy Example: The Student and Exam

Imagine two students studying for a math exam:

• Student A (Low Capacity): This student can only understand and solve very
simple, linear equations like y=2x+1. Their brain is like a model with very low
capacity. They can only handle basic patterns, and they would fail if the test
included more complex problems. This is similar to an underfitting model.

• Student B (High Capacity): This student has a deep understanding of math.

They can handle simple problems, but they can also solve very complex
equations like y=5x4+3x3−2x2+x+7. Their brain is like a model with very high
capacity.

The ideal student is one with enough capacity to learn the right amount—not so
little that they underfit, and not so much that they overfit by memorizing every
single practice problem.

Capacity in Machine Learning

In a machine learning model, capacity is controlled by its structure and
complexity.

Example: Linear vs. Polynomial Regression

• A simple straight line (y=ax+b) has very low capacity. It can only learn linear
relationships in the data. If the data is curved, this model will underfit.

• A complex, curvy line (e.g., y=ax5+bx4+...) has very high capacity. It can learn
very complex, "curved" patterns, even if those patterns are just noise in the data.
This model is more likely to overfit.

So, a model with higher capacity can fit more complex functions, while a model
with lower capacity can only fit simpler functions. The key is to choose a model
with the right capacity for your specific problem to achieve the best fitting.

Deep Learning 5
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

• Underfitting Definition: This occurs when a model has insufficient capacity to

capture the underlying patterns in the training data, leading to high error on both
the training set and unseen test data. The model is too simple and fails to learn
effectively.

• Underfitting occurs when a model is too simple (low capacity) to capture the
underlying patterns in the training data, resulting in poor performance on both
training and test sets.

• Characteristics: High bias, low variance. The model makes strong, incorrect
assumptions about the data.

• Overfitting Definition: This happens when a model has too much capacity and
learns the training data too well, including noise and specific details that are not
representative of the true data distribution. While performing well on the training
data, it generalizes poorly to new, unseen data, resulting in a large gap between
training and test error.

• Overfitting occurs when a model is excessively complex (high capacity) and

learns the training data too well, including its noise and outliers. This leads to
excellent performance on the training data but poor generalization to new,
unseen data.

• Characteristics: Low bias, high variance. The model is overly sensitive to

fluctuations in the training data.

• Balancing Capacity: The goal is to find a model with appropriate capacity – one
that is complex enough to learn the significant patterns in the data (avoiding
underfitting) but not so complex that it memorizes noise and fails to generalize
(avoiding overfitting).

Deep Learning 6
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

• Finding the Right Balance: The aim is to achieve a "good fit" where the model
captures the essential patterns in the data without memorizing noise, striking a
balance between bias and variance. This results in good performance on both
training and test [Link]: For the curved data points example, a well-fitted
model would be a smooth curve that accurately represents the general trend of
the data, ignoring minor fluctuations caused by noise.

• Bias-Variance Tradeoff : Underfitting is associated with high bias (the model

makes strong assumptions and oversimplifies), while overfitting is associated
with high variance (the model is too sensitive to fluctuations in the training data).

Example : Assume that Student study is Training and Exam is a Testing. That
means Bias is Training Error and Variance is a Testing Error.

• Underfitting (High Bias)

o Student Analogy: This is a student who doesn't study enough. They have
a very basic and oversimplified understanding of the material. When
faced with an exam, they can't answer the questions well, because their
knowledge is too general.
o Machine Learning: The model is too simple and has a high bias. It fails
to capture the underlying patterns in the training data, resulting in poor
performance on both the training data and new data. This is often caused
by a model with low complexity.

• Overfitting (High Variance)

o Student Analogy: This is a student who studies by memorizing every
single detail of the textbook, including minor facts and footnotes, but fails
to understand the core concepts. They can answer questions that are
identical to those in their study materials but fail on any new or slightly
rephrased questions. Their performance is highly dependent on the
specific questions they were trained on.
o Machine Learning: The model is too complex and has a high variance. It
learns the noise and random fluctuations in the training data rather than
the true underlying relationship. This leads to excellent performance on
the training data but poor performance on new, unseen data because the
model cannot generalize.

• Best Fitting (Low Bias, Low Variance)

o Student Analogy: This is the ideal student who studies just enough to
grasp the core concepts and can apply them to solve a variety of
problems, including those they haven't seen before. They perform well on
both their practice questions and the actual exam.
o Machine Learning: This is the goal of model training. The model is
complex enough to capture the true underlying pattern of the data (low
bias) but simple enough to not be overly influenced by the specific
training examples (low variance). This results in a model that performs

Deep Learning 7
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

well on both the training data and new data, meaning it generalizes
effectively.

Hyperparameters and Validation Sets

Most machine learning algorithms have several settings that we can use to control the
behavior of the learning algorithm. These settings are called hyperparameters. The
values of hyperparameters are not adapted by the learning algorithm itself (though we
can design a nested learning procedure where one learning algorithm learns the best
hyperparameters for another learning algorithm).

Hyperparameters
• Definition: These are settings that control the learning algorithm's behavior.
Unlike model parameters, they are not learned during training.
• Examples: The degree of a polynomial in polynomial regression or the λ value
in weight decay are examples of hyperparameters.
• Purpose: They influence the model's capacity and ability to generalize.

Example:

Think of it like this: If a model is a baker, the hyperparameters are the recipe's
instructions (e.g., "bake at 350 degrees for 30 minutes"). The model's parameters (the
weights and biases) are the baker's adjustments and techniques they learn while baking
to make the cake perfect (e.g., how to mix the ingredients to get a smooth batter). The
baker doesn't learn what temperature to use during baking; that's set beforehand.

Validation Sets
• Definition: A separate dataset used to evaluate a model's performance during
hyperparameter tuning.
• Purpose: To select the best hyperparameters for a model by finding the
combination that yields the lowest generalization error.
• Process:
1. The model is trained on the training data.
2. The model is evaluated on the validation set for a given set of
hyperparameters.
3. This process is repeated with different hyperparameter values.
4. The hyperparameter values that result in the best performance on the
validation set are chosen.
5. Training data is split into two disjoint parts. First to learn the parameters
and other is the validation set to estimate generalization error during or
after training.
6. Allowing for the hyperparameters to be updated – Typically 80% of
training data for training and 20% for validation

Deep Learning 8
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

Relationship Between Hyperparameters and Validation Sets

• Hyperparameter Tuning: The process of using a validation set to find the

optimal hyperparameters for a model.

• U-Shaped Curve: The generalization error of a model typically forms a U-

shaped curve when plotted against a hyperparameter.

o Underfitting: With a low-capacity model (e.g., a small number of hidden

units), both training and generalization error are high.
o Overfitting: With a high-capacity model, the training error is low, but the
generalization error is high due to a large gap between training and test
error.
o Optimal: The ideal hyperparameter setting corresponds to the point of
lowest generalization error, which balances underfitting and overfitting.

Cross-Validation:

This is a powerful technique used in machine learning to assess the performance of a

model and ensure it can generalize well to new, unseen data. It helps to mitigate
the problems of overfitting and selection bias that can occur with a simple train-
test split.

Key Points of Cross-Validation

• Purpose: The main goal is to get a more reliable and less biased estimate of a
model's performance on unseen data. It does this by repeatedly training and
testing the model on different subsets of the data.
• Process: The dataset is partitioned into multiple subsets (or "folds"). The model
is trained on a combination of these folds and then tested on the remaining fold.
This process is repeated until every fold has been used as the test set.
• Result: The performance scores from each iteration are averaged to produce a
final, more robust performance estimate. This score is a better indicator of how
the model will perform in a real-world scenario.

1. K-Fold Cross-Validation

• Description: The most common form of cross-validation. The dataset is

randomly divided into k equal-sized folds.
• Process: For each of the k iterations, one fold is used as the test set, and the
remaining k-1 folds are used for training.
• Example: If you have 1000 data points and choose k=10, the data is split into 10
folds of 100 points each. In the first iteration, fold 1 is the test set, and folds 2-10
are used for training. In the second iteration, fold 2 is the test set, and folds 1, 3-
10 are used for training, and so on. This is repeated 10 times.

Deep Learning 9
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

Best example in Cross-Validation:

Imagine you have a new recipe for a cake and you want to know if it's really good.
A simple way to test it would be to bake one cake and have one person taste it. But
what if that person doesn't like cake? Or what if you made a mistake on that one
cake? Your single test might not give you a good idea of how the recipe really is.
Cross-validation is like taking that recipe and making it multiple times, but with a slight
twist each time, and having a group of people try them all.
Here's how it works in simple terms:
1. Split the Data: You take all your ingredients (your data) and divide them into
several smaller, equal groups. Let's say you divide them into five groups. These
are your "folds."
2. Iterate and Test:
o First test: You use four of the groups of ingredients to bake the cake.
Then, you use the fifth group of ingredients to bake a test cake. You have
a judge taste that test cake and give it a score.
o Second test: Now you swap! You use a different four groups of
ingredients to bake the cake, and you use another one of the groups as
your test cake. The same judge tastes it and gives a score.
o Repeat: You keep doing this until every single group of ingredients has
had a chance to be the test cake.
3. Average the Scores: In the end, you have a score from each of your test cakes.
You add up all these scores and find the average. This average score is a much
more reliable and fair measure of how good your recipe (your model) really is.

Why is this better?

• It makes sure your model isn't just lucky with one particular part of the data.
• It gives you a much better idea of how well your model will perform on new data
it has never seen before.
• It's a more trustworthy way to judge your model's performance than just a single
test.

Deep Learning 10
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

Estimators, Bias and Variance:

In machine learning, an estimator is an object that learns a model from data, typically
using a "fit" method, and then can predict properties of new data using a "predict"
method. Essentially, it's the algorithm or function that attempts to find the best
representation of the underlying patterns in the data.

An estimator measures a given quantity based on observed data and produces an

estimate of that quantity

• Types of Estimators:
Estimators can be used for various machine learning tasks, including classification
(predicting categories) and regression (predicting continuous values).

Example:

Imagine you're building a model to predict house prices. You might use a linear
regression estimator. The estimator would:
1. Fit:
Analyze a dataset of house prices and their features (size, location, etc.) to learn the
relationship between these features and the price.
2. Predict:
Once fitted, you could input the features of a new house into the trained estimator, and
it would predict the corresponding price.

Bias:

Bias is one type of error that occurs due to wrong assumptions about data such as
assuming data is linear when in reality, data follows a complex function. On the other
hand, variance gets introduced with high sensitivity to variations in training data. This
also is one type of error since we want to make our model robust against noise. There
are two types of error in machine learning. Reducible error and Irreducible error. Bias
and Variance come under reducible error.

Bias is simply defined as the inability of the model because of that there is some
difference or error occurring between the model's predicted value and the actual value.
These differences between actual or expected values and the predicted values are
known as error or bias error or error due to bias. Bias is a systematic error that occurs
due to wrong assumptions in the machine learning process.

Let Y be the true value of a parameter, and let hat Y be an estimator of Y based on a
sample of data. Then, the bias of the estimator hatY is given by:

Bias (Y^) = E(Y^) – Y

where E(Y^) is the expected value of the estimator Y^. It is the measurement of the
model that how well it fits the data.

Deep Learning 11
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

• Low Bias: Low bias value means fewer assumptions are taken to build the target
function. In this case, the model will closely match the training dataset.
• High Bias: High bias value means more assumptions are taken to build the target
function. In this case, the model will not match the training dataset closely.

The high-bias model will not be able to capture the dataset trend. It is considered as
the underfitting model which has a high error rate. It is due to a very simplified
algorithm.

For example, a linear regression model may have a high bias if the data has a non-linear
relationship.

Ways to reduce high bias in Machine Learning:

• Use a more complex model: One of the main reasons for high bias is the very
simplified model. it will not be able to capture the complexity of the data. In
such cases, we can make our mode more complex by increasing the number of
hidden layers in the case of a deep neural network. Or we can use a more
complex model like Polynomial regression for non-linear
datasets, CNN for image processing, and RNN for sequence learning.

• Increase the number of features: By adding more features to train the dataset
will increase the complexity of the model. And improve its ability to capture the
underlying patterns in the data.

• Reduce Regularization of the model: Regularization techniques such as L1 or

L2 regularization can help to prevent overfitting and improve the generalization
ability of the model. if the model has a high bias, reducing the strength of
regularization or removing it altogether can help to improve its performance.

• Increase the size of the training data: Increasing the size of the training data
can help to reduce bias by providing the model with more examples to learn
from the dataset.

Variance :

Variance is the measure of spread in data from its mean position. In machine learning
variance is the amount by which the performance of a predictive model changes when it
is trained on different subsets of the training data. More specifically, variance is the
variability of the model that how much it is sensitive to another subset of the training
dataset. i.e. how much it can adjust on the new subset of the training dataset.

Let Y be the actual values of the target variable, and Y^ be the predicted values of
the target variable. Then the variance of a model can be measured as the expected value
of the square of the difference between predicted values and the expected value of the
predicted values.

Variance = E[(Y^ − E[Y^])2]

Deep Learning 12
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

where E[Yˉ] is the expected value of the predicted values. Here expected value is
averaged over all the training data.

Variance errors are either low or high-variance errors.

• Low variance: Low variance means that the model is less sensitive to changes in
the training data and can produce consistent estimates of the target function with
different subsets of data from the same distribution. However, low variance can
also indicate underfitting if the model is too simple and fails to capture the
underlying patterns in the data. This is when the model performs poorly on both
the training data and testing data.
• High variance: High variance means that the model is very sensitive to changes
in the training data and can result in significant changes in the estimate of the
target function when trained on different subsets of data from the same
distribution. This is the case of overfitting when the model performs well on the
training data but poorly on new, unseen test data. It fits the training data too
closely that it fails on the new training dataset.

Ways to Reduce the reduce Variance in Machine Learning:

• Cross-validation: By splitting the data into training and testing sets multiple
times, cross-validation can help identify if a model is overfitting or underfitting
and can be used to tune hyperparameters to reduce variance.

• Feature selection: By choosing the only relevant feature will decrease the
model's complexity. and it can reduce the variance error.
• Regularization: We can use L1 (Lasso) or L2 (Ridge) regularization to reduce
variance in machine learning models
• Ensemble methods: It will combine multiple models to improve generalization
performance. Bagging, boosting, and stacking are common ensemble methods
that can help reduce variance and improve generalization performance.
• Simplifying the model: Reducing the complexity of the model, such as
decreasing the number of parameters or layers in a neural network, can also help
reduce variance and improve generalization performance.
• Early stopping: Early stopping is a technique used to prevent overfitting by
stopping the training of the deep learning model when the performance on the
validation set stops improving.

Different Combinations of Bias-Variance

There can be four combinations between bias and variance.

• High Bias, Low Variance: A model with high bias and low variance is said to
be underfitting.

• High Variance, Low Bias: A model with high variance and low bias is said to
be overfitting.

Deep Learning 13
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

• High-Bias, High-Variance: A model has both high bias and high variance,
which means that the model is not able to capture the underlying patterns in the
data (high bias) and is also too sensitive to changes in the training data (high
variance). As a result, the model will produce inconsistent and inaccurate
predictions on average.

• Low Bias, Low Variance: A model that has low bias and low variance means
that the model is able to capture the underlying patterns in the data (low bias)
and is not too sensitive to changes in the training data (low variance). This is the
ideal scenario for a machine learning model, as it is able to generalize well to
new, unseen data and produce consistent and accurate predictions. But in
practice, it's not possible.

Top-Left (Low Bias, High Variance): The data points (green dots) are spread out but
centered around the bullseye (black dot). This indicates that the model is, on average,
correct (low bias), but its predictions are inconsistent and widely distributed (high
variance). This often happens when a model is overly complex and has learned the
training data too well, including the noise. This is known as overfitting.

Top-Right (High Bias, High Variance): The data points are both spread out and off-
center. This is the worst-case scenario. The model's predictions are inaccurate on
average (high bias) and also inconsistent (high variance). This indicates a poorly
performing model that is neither accurate nor reliable.

Bottom-Left (Low Bias, Low Variance): The data points are tightly clustered and
centered around the bullseye. This is the ideal scenario. The model is consistently
accurate (low variance) and its predictions are correct on average (low bias). This

Deep Learning 14
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

represents a well-performing model that has successfully generalized from the training
data.

Bottom-Right (High Bias, Low Variance): The data points are tightly clustered but
off-center. This indicates that the model is consistently wrong. Its predictions are
reproducible (low variance), but they are systematically incorrect (high bias). This is
often a sign of an overly simplistic model that fails to capture the underlying patterns in
the data, a phenomenon known as underfitting.

Now we know that the ideal case will be Low Bias and Low variance, but in practice,
it is not possible. So, we trade off between Bias and variance to achieve a balanced bias
and variance.
A model with balanced bias and variance is said to have optimal generalization
performance. This means that the model is able to capture the underlying patterns in the
data without overfitting or underfitting. The model is likely to be just complex enough
to capture the complexity of the data, but not too complex to overfit the training data.
This can happen when the model has been carefully tuned to achieve a good balance
between bias and variance, by adjusting the hyperparameters and selecting an
appropriate model architecture.

Machine Learning Algorithm Bias Variance

Linear Regression High Bias Less Variance

Decision Tree Low Bias High Variance

Random Forest Low Bias High Variance

Bagging Low Bias High Variance

Bias Variance Tradeoff

If the algorithm is too simple (hypothesis with linear equation) then it may be on high
bias and low variance condition and thus is error-prone. If algorithms fit too complex
(hypothesis with high degree equation) then it may be on high variance and low bias.
In the latter condition, the new entries will not perform well. Well, there is something
between both of these conditions, known as a Trade-off or Bias Variance Trade-off.
This tradeoff in complexity is why there is a tradeoff between bias and variance. An
algorithm can’t be more complex and less complex at the same time. For the graph, the
perfect tradeoff will be like this.

Deep Learning 15
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

The technique by which we analyze the performance of the machine learning model is
known as Bias Variance Decomposition. Now we give 1-1 example of Bias Variance
Decomposition for classification and regression.

BAYESIAN STATISTICS :

Bayesian statistics is a branch of statistics that interprets probability as a measure of

believability or confidence that an individual may possess about the occurrence of a
particular event. It is named after Thomas Bayes, an 18th-century Presbyterian minister
and mathematician. Bayes' most notable contribution, Bayes' Theorem, is the
cornerstone of Bayesian statistics and provides a mathematical formula for updating
probabilities based on new evidence.

Bayes' Theorem

Bayes' Theorem describes the probability of an event, based on prior knowledge of

conditions that might be related to the event. The theorem is mathematically stated as:

where:
• P(A|B) is the probability of event A occurring given that B is true.
• P(B|A) is the probability of event B occurring given that A is true.
• P(A) is the probability of event A.
• P(B) is the probability of event B.

This formula allows statisticians to update the probability estimate for a hypothesis as
more evidence or information becomes available.

Three key concepts in Bayesian statistics are the prior, the likelihood, and the posterior:

• Prior (P(A)): This is the initial judgment or belief about the probability of an
event before new data is introduced.

Deep Learning 16
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

• Likelihood (P(B|A)): This is the probability of observing the new data under the
assumption that the prior and the model are true.
• Posterior (P(A|B)): The updated probability of the event after considering the
new data. The posterior becomes the new prior if more data becomes available,
allowing for continuous updating of our belief.

Deep Learning 17
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

BUILDING A MACHINE LEARNING ALGORITHM / MODEL :

How to build any machine learning Algorithm / model from scratch

These steps form the backbone to any machine learning process and knowing them will
make your life much easier when trying to build ML models.

1. Data Collection
Machine learning requires training data, a lot of it. This data can either be labelled
meaning Supervised Learning or not labelled meaning Unsupervised Learning.
Accuracy of the model depends on the quality and quantity of the data. The
outcome of this step is generally a representation of data which will be used for
training. Using pre-collected data, by way of datasets from sites like Kaggle, UCI, etc.
forms the basis of you Machine learning project. You may also collect data through
user-surveys, analysis reports, trends, usage metrics, etc.

2. Data Preparation
We cannot work on raw data. Data needs to be processed by normalization,
removing duplicates, errors and biases.

Visualising data can be helpful in searching for patterns and outliers to check if the
data collected is right or if it contains missing values. This can be done using libraries
like seaborn, matplotlib, etc. Visualize data to help detect relevant relationships
between variables or class imbalances, or perform other exploratory analysis.

After performing data wrangling, we need to prepare the data for training. Cleaning of
data is done that involves steps like removing duplicates, dealing with missing values,
type conversions, correcting errors, normalizing the data, etc.
Not all the above steps are needed to be performed as it depends entirely on the data
collected. Some datasets may not require data preparation at all while for some
data preparation step takes majority of their ML model build time.

Deep Learning 18
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

We can also Randomize data, which erases the effects of the particular order in
which we collected and/or otherwise prepared our data. Later we can split the data
into training, testing and evaluation sets
3. Choose a Model / Algorithm
The third step consists of selecting the right model. There are many models which can
be used for many different purposes. Once the model is selected, it needs to meet the
business goal.
We need to have an idea about the preparation the model requires along with its
accuracy and scalability. Having a complex model does not mean a better model.
Common machine learning algorithms include Decision Trees, Random Forest, Linear
Regression, Support Vector Machines (SVM), Logistic Regression, K-means, Principal
Component Analysis (PCA), Naïve Bayes, and Neural Networks. Different algorithms
need to be applied to different tasks, you need to choose the correct one for your use
case.
4. Training the Model
Training a model forms the basis of machine learning. The goal is to use our training
data and improve the predictions of our model.

Every cycle in training a model involves updating the weights and biases in each
training step. We can use labelled sample data in case supervised machine learning and
unlabelled sample data for unsupervised learning.
The goal of training is to evaluate and further improve our model accuracy and
performance. Training happens in the form of iterations which is called a training step.

5. Evaluate the Model

After training the model comes evaluating the model. The larger the number of
variables in the real world, the bigger the training and test data should be.
Performance metrics are used to measure the performance of the model. These include
precision, recall, accuracy, specificity, etc.
The model is then tested against previously unseen data. The unseen data is meant to act
as representative of model performance in the real world, but still helps tune the model
(as opposed to test data, which does not).

Deep Learning 19
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

A 70/30 split, or similar, is considered a good train/eval split, which depends on things
like data availability, dataset features, domain, etc.
6. Parameter Tuning
The original model parameters need to be tested after evaluating your model. By
increasing the training, it can lead to better results.
Parameter tuning is an experimental process and hence we need to define when to
stop parameter tuning otherwise it will continue to tweak the model.
Hyperparameter tuning is an art and one that requires patience & experience. Once
the model parameters are tuned it can give us better results. Some common
hyperparameters include: number of training steps, learning rate, initialization
values and distribution, etc.
7. Make Predictions
After the processes of collecting data, preparing the data, selecting a machine learning
algorithm, training the model and evaluating the model & tuning the parameters, we
need to make predictions.

Our machine learning model can make predictions ranging from image recognition to
predictive analytics to natural language processing.
After building the model needs to be tested on a testing set to check how the model
performs on unseen data. It helps to further evaluate the model and provides better
approximation.

Deep Learning 20
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

STOCHASTIC GRADIENT DESCENT :

Stochastic Gradient Descent (SGD) is an optimization algorithm that uses a randomly

selected subset of data, rather than the entire dataset, to calculate the gradient and
update the model's parameters. This makes it significantly more computationally
efficient for large datasets compared to standard Gradient Descent, where a single
update can be slow and memory-intensive.

The core idea of SGD:

In many machine learning problems, the objective is to minimize a loss function, J(θ),
which is an average of the losses for each training example.

• Gradient Descent (GD): The update for standard gradient descent uses the gradient
of the entire loss function.

• Stochastic Gradient Descent (SGD): Instead of averaging the gradient over all n,
samples, SGD computes the gradient for a single randomly chosen sample, ik, at
each step.

The random selection of a single data point makes the gradient an unbiased
estimate of the true gradient.

Mini-batch SGD :

The most common variant of SGD used in practice is mini-batch SGD, which strikes a
balance between the computational efficiency of SGD and the update stability of
standard gradient descent.

• It updates parameters using a small, randomly selected subset (mini-batch) of the

data, with a batch size b < < n. Popular choices for b are 32 or 64.

• The update rule is given by

Deep Learning 21
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

Where Ik is the set of indices for the random mini-batch at iteration k.

Key differences and characteristics

Feature Gradient Descent (GD) Stochastic Gradient Descent

(SGD)

Computational High. Processes all training Low per iteration. Computes the
cost examples to compute a single gradient for only one (or a small
gradient update. batch of) example(s).

Update Infrequent. Updates parameters Frequent. Updates parameters after

frequency once per epoch. every sample (or mini-batch).

Path to Smooth. The consistent Noisy. The random selection of

minimum gradient direction leads directly samples causes the path to oscillate
to the minimum. around the minimum.

Escaping local Can get stuck in local minima, The noisy updates can help the
minima especially in non-convex algorithm "jump out" of local
optimization problems. minima and saddle points.

Online learning Not suitable, as it requires the Well-suited for online learning, as
entire dataset for updates. it can update parameters with new
data points as they arrive.

Deep Learning 22
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

CHALLENGES MOTIVATING DEEP LEARNING:

Challenges motivating deep learning include the need to handle vast amounts of data,
overcome the "black box" problem of interpretability, and the high computational cost
of training models. Overfitting, scalability issues, the vulnerability of models to
adversarial attacks, and potential biases in data also pose significant hurdles that must
be addressed for deep learning to be applied effectively and ethically in real-world
scenarios.

Data-Related Challenges

• Data Quantity and Quality:

Deep learning models require massive datasets to learn effectively, which can be
expensive and time-consuming to acquire.

• Bias:
Training data can contain inherent biases, leading to unfair or inaccurate
predictions and requiring careful steps for mitigation.

• Adversarial Attacks:
Malicious actors can manipulate input data to trick models, causing
misclassifications and unreliable outputs.

Model-Related Challenges :

• Interpretability (The "Black Box" Problem):

Deep learning models are complex, making it difficult to understand how they
arrive at their conclusions, which is crucial for critical applications.

• Overfitting:
Models can become too specialized to the training data, learning noise rather than
underlying patterns, leading to poor performance on new, unseen data.

• Scalability:
Effectively handling massive datasets and increasingly complex tasks in real-
world applications is a significant challenge.

Computational and Deployment Challenges :

• Computational Resources:
Training deep learning models is computationally expensive, requiring powerful
hardware like GPUs and TPUs.

• Deployment Cost and Power:

Deploying these models to various devices, such as smart devices, can be costly
and power-intensive.

Deep Learning 23
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

• Hyperparameter Tuning:
Finding the optimal settings for hyperparameters can be a complex and time-
consuming process.

Ethical and Practical Concerns :

• Generalization:
Traditional algorithms have failed to generalize well on complex AI tasks like speech
and object recognition, motivating the development of deep learning to address these
failures.
• Lack of Flexibility:
Deep learning models are often task-specific; they are ineffective at solving problems
other than the one they were trained for.

DEEP FEEDFORWARD NETWORK:

A deep feedforward network is a fundamental type of artificial neural network in deep

learning that processes information in a Single, Unidirectional "Forward" Direction,
from the input layer, through one or more "hidden" layers, to the output layer. There are
no cycles or feedback loops in the network. These networks learn by approximating a
function, using back propagation to adjust their internal parameters (weights and biases)
to map input data to desired outputs. Deep feedforward networks are used for a wide
range of applications, including object recognition and natural language processing.

Deep Learning 24
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

1. Input Layer:
This layer receives the raw input data, such as pixels of an image or words in a
sentence.

2. Hidden Layers:
Multiple hidden layers are stacked between the input and output layers. Each
neuron in a hidden layer receives inputs from the previous layer, computes a
weighted sum, adds a bias, and then applies a nonlinear activation function. This
process introduces complex non-linear transformations, enabling the network to
learn intricate patterns.

3. Output Layer:
This final layer produces the network's output. The specific form of the output
function depends on the task, such as classifying an image or predicting a
numerical value.

4. Unidirectional Flow:
Data travels only from the input layer to the hidden layers and then to the output
layer, without any backward flow or loops.

Learning XOR:

Learning the XOR function is a classic problem in deep learning used to illustrate the
capabilities of multi-layer neural networks and the limitations of single-layer
perceptrons.

The XOR Problem:

The XOR (exclusive OR) function returns a true (1) output if exactly one of its two
binary inputs is true (1), and a false (0) output otherwise. Its truth table is:

Input 1 Input 2 Output (XOR)

0 0 0

0 1 1

1 0 1

1 1 0

The challenge arises because the XOR function is not linearly separable. This means a
single straight line (or hyperplane in higher dimensions) cannot separate the input

Deep Learning 25
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

combinations that result in a 0 output from those that result in a 1 output. A simple,
single-layer perceptron, which can only learn linearly separable functions, therefore
cannot solve the XOR problem.

HIDDEN UNITS :

In deep learning, a hidden unit refers to an individual artificial neuron or node located
within a hidden layer of a neural network. Unlike input units, which directly receive
raw data, or output units, which produce the final predictions, hidden units are not
directly exposed to the external environment. Their primary function is to perform
intermediate computations and transform the input data into more abstract and
meaningful representations that can be used by subsequent layers.

ARCHITECTURE DESIGN IN DEEP LEARNING:

Architecture design in deep learning refers to the process of structuring a neural

network, defining its components, and determining how they interact to achieve a
specific task. This involves decisions about the network's depth, width, and the types of
layers used.

Key Elements of Deep Learning Architecture Design:

• Layers
• Neurons/Units
• Connections and Weights
• Activation Functions
• Data Flow
• Optimization

Common Deep Learning Architectures:

• Multi-layer Perceptrons (MLPs)
• Convolutional Neural Networks (CNNs)
• Recurrent Neural Networks (RNNs)
• Transformers
• Hybrid Models

Considerations in Architecture Design:

• Problem Type
• Data Characteristics
• Computational Resources
• Performance Metrics

Deep Learning 26
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

THE BACKPROPAGATION ALGORITHM :

The backpropagation algorithm is a fundamental algorithm in deep learning used to

train artificial neural networks. It enables the network to learn by adjusting its internal
parameters (weights and biases) to minimize the difference between its predictions and
the actual target values.

Process:

• Forward Pass:
Input data is fed through the network, layer by layer, with each neuron computing its
output based on the weighted sum of its inputs and an activation function. This process
continues until an output is generated by the final layer.

• Loss Calculation:
The network's output is compared to the true target values, and a loss function
calculates the error or discrepancy between them.

• Backward Pass (Backpropagation of Error):

The core of the algorithm begins here. The calculated error is propagated backward
through the network, starting from the output layer and moving towards the input
layer. This involves:

• Calculating Gradients: Using the chain rule of calculus, the algorithm

computes the gradient of the loss function with respect to each weight and
bias in the network. These gradients indicate how much each parameter
contributes to the overall error.

• Propagating Deltas: The error signals (often referred to as "deltas") are

calculated for each neuron, representing the contribution of that neuron to
the total error. These deltas are propagated backward to subsequent
layers.

• Weight and Bias Update:

With the gradients computed, an optimization algorithm (like gradient
descent) uses these gradients to adjust the weights and biases of the
network. The adjustments are made in a direction that reduces the loss,
effectively teaching the network to make more accurate predictions in
future iterations.

Deep Learning 27
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])
 lOMoARcPSD|56847386

Deep Learning 28
Downloaded by Tilak ram Sahu (tilakramsahu7618@[Link])