Deep Learning is transforming the way machines understand, learn and interact with complex

data.
Deep learning mimics neural networks of the human brain, it enables computers to
autonomously uncover patterns and make informed decisions from vast amounts of unstructured
data.

Machine Learning Deep Learning

Applies statistical algorithms to learn hidden Uses artificial neural network architectures to
patterns and relationships in the dataset. learn hidden patterns and relationships in the
dataset.
Can work on a smaller amount of data. Requires a larger volume of data compared to
machine learning.
Better for low-label tasks. Better for complex tasks like

Takes less time to train the model. Takes more time to train the model.
Model created by manually Automatically extracts features from data
extracted features (e.g., detecting objects in end-to-end learning process.
images).
Less complex and easier to More complex, often acts as a black box
interpret results. (difficult to interpret).
Can work on CPU or with low computing Requires high-performance computing,
power. typically GPU.
Learning Algorithms
The Task, T
1. Classification
2. Classification with missing inputs
3. Regression
4. Transcription
5. Machine translation
6. Structured output
7. Anomaly detection
8. Synthesis and sampling
9. Imputation of missing values
10. Denoising
11. Density estimation or probability mass function estimation
“A computer program is said to learn from experience (E) with respect to some class of
tasks (T) and performance measure (P), if its performance at tasks in T, as measured
by P, improves with experience E.”

Component Meaning Example

The data or interactions from which A spam filter learns from a set
the program learns. of labeled emails (spam / not
E (Experience)
spam).
The specific problem or Classifying incoming emails as
T (Task) activity the program is designed to spam or not spam.
perform.
P(Performance The metric used to evaluate how The accuracy or F1 score of
Measure) well the program performs the task. the spam predictions.

The Task (T) in Machine Learning

i. A task (T) is the goal-oriented activity that a machine learning algorithm is
designed to perform.
ii. It is not the act of learning itself — learning is merely the means to accomplish
that task.
Example: If we want a robot to walk, then walking is the task.
The learning process helps the robot develop the ability to walk.
Importance of Defining the Task
i. Machine learning enables us to handle complex tasks that are difficult or
impossible to program explicitly by humans.
 ii. Understanding these tasks helps us explore the principles of intelligence
and how machines can simulate human-like decision- making.

Representation of a Task
A machine learning task is usually described in terms of how a system processes an
example. An example: is a single observation or data point from the dataset.
is represented as a vector of features: x=[x1,x2,x3,…,xn]∈Rn
where each xi represents one measured feature of the object or event.

Term Description
Task (T) Classify an image as “cat” or “dog”.
Example (x) One image represented as pixel values.
Feature Vector (x = [x_1, x_2, x_3, \dots, x_n]), where each (x_i) is a pixel intensity.
Learning Goal Learn a function ( f(x) \to \text{label} ) that correctly predicts the
image category.

[Link] Task
Classification is a type of supervised learning task in which a computer program learns to
assign input data to one of several predefined categories or classes. Mathematically, the task is
to learn a function:

Intuitive Explanation
The algorithm learns from labeled examples (inputs with known categories) to predict
the correct class for new, unseen inputs.
The program learns a mapping from input features to discrete output labels.
 Term Description
Task (T) Identify the object in an image.
Input (x) Image represented as pixel brightness values.
Output (y) Category (e.g., “bottle”, “glass”, “cup”).
Goal Correctly recognize which kind of drink is in the image.

Real-World Example — PR2 Robot (Goodfellow et al., 2010)

The Willow Garage PR2 robot acts as a waiter:
• It captures an image of a drink.
• Uses an image recognition model to classify the drink type (e.g., water, soda,
coffee).
• Then delivers the correct drink to the person on command. This is a classification
task where:
• Input → image features (pixels)
• Output → drink category
• Algorithm → deep learning (CNNs)

Probabilistic Variant of Classification

Instead of outputting just one label, some models output a probability distribution across
all possible classes:

• 80% chance it’s a cat

• 15% chance it’s a dog
• 5% chance it’s a rabbit
Domain Example
Computer Vision Object recognition, facial recognition, handwriting recognition
Healthcare Disease diagnosis (healthy vs. diseased)
Finance Loan approval (approve/reject)
Cybersecurity Spam detection, intrusion classification
Speech Processing Speaker identification, emotion recognition
 Concept Description
Definition Task of assigning inputs to one of ( k ) discrete categories.
Function Form f : R^n {1, 2, ..., k}
Input Feature vector x
Output Class label ( y )
Learning Goal Minimize misclassification error; improve accuracy
Modern Approach Deep learning using Convolutional Neural Networks (CNNs)
Example PR2 robot recognizing drinks; facial recognition systems

2. Classification with Missing Inputs

In classification with missing inputs, the learning system must still correctly assign a
category to an input even when some of its features are unavailable.
Unlike standard classification (which assumes all input features are present), this variant
deals with incomplete data vectors. x=[x1,x2,…,xn]
where one or more feature values xi may be missing.
The Challenge
• A normal classifier learns one function:

When some inputs are missing, the algorithm must conceptually handle many possible
input subsets.
Each missing-input combination corresponds to a different function.
If there are n input variables, there can be up to: different functions (one for every possible
subset of missing inputs). Clearly, learning so many separate functions is impractical.

Real-World Example — Medical Diagnosis

• This situation frequently arises in healthcare and diagnostics:
Scenario Description
Input Features Results of multiple medical tests (blood sugar, MRI, ECG, etc.)
Missing Inputs Some tests are not done (expensive, invasive, or unavailable)
Output (Task) Classify the patient’s condition (e.g., “healthy”, “diabetic”, “anemic”)
• A robust learning system must still make accurate predictions despite these
missing measurements.
Efficient Solution — Probabilistic Modeling
i. Instead of training a separate classifier for each missing-input case, we can train
one probabilistic model that captures the joint distribution over all input and
output variables.
P(x1,x2,…,xn,y)
ii. When an input feature (say xi) is missing, we can marginalize it out
(integrate or sum over all its possible values):

iii. This approach allows the classifier to handle any missing subset of inputs
without retraining.

Marginalization
Marginalization = summing or integrating over the missing features to estimate their
effect. For example:
Here x2 is missing, so the model integrates over its possible values.
Deep Probabilistic Models
Modern solutions use deep learning architectures that model complex joint distributions
efficiently. Examples include:

• Deep Belief Networks (DBNs)

• Variational Autoencoders (VAEs)
• Deep Generative Models (Goodfellow et al., 2013b)
These can implicitly learn P(x,y) and perform classification even when part of x is
unobserved.
Concept Explanation
Problem Inputs may be incomplete or missing.
Naïve Solution Train many classifiers (2ⁿ functions).
Efficient Solution Learn one joint probability distribution and marginalize missing
inputs.
Mathematical Tool Marginalization over missing variables.
Applications Medical diagnosis, sensor failure, incomplete survey data.
Modern Approach Deep probabilistic models like VAEs or DBNs.
3. Regression
In a regression task, the computer program predicts a continuous numerical value from an
input vector of features.

Mathematically:
Domain Example
Insurance
Predict expected claim amount made by an insured person to set premiums.
Finance Predict future stock prices or market trends for algorithmic trading.
Real Estate Predict house prices based on features (location, area, rooms).
Health Estimate blood sugar levels or tumor size from medical data.
• Output Type: Continuous numeric value.
• Goal: Minimize prediction error (e.g., Mean Squared Error).
• Relation to Classification: Similar structure, but output is real-valued rather than discrete.

4. Transcription
Transcription involves converting unstructured input data (images, audio, etc.) into discrete
textual sequences. Mathematically

Domain Example
Convert images of text into machine-readable
Optical Character Recognition (OCR) characters.

Speech Recognition Convert audio signals (waveforms) into text.

Street View Analysis Identify house numbers from images (Google
Street View).
 Modern transcription tasks use deep neural networks (especially CNNs + RNNs
or Transformers) to extract hierarchical features from raw data.
Google Street View (Goodfellow et al., 2014d): Used deep CNNs to read address numbers
from photos. Speech Recognition (Hinton et al., 2012b): Deep Neural Networks (DNNs)
replaced traditional models like GMM-HMMs, greatly improving accuracy.

5. Machine Translation
In machine translation, the input is a sequence of symbols (words) in one language,
and the system must generate a sequence in another language.

f:Sequence in Language A→Sequence in Language B

Input Output
“How are you?” (English) “Comment ça va ?” (French)
“I love AI.” “J’adore l’intelligence artificielle.”

Example — Neural Machine Translation (NMT)

Modern translation models use deep sequence-to-sequence (Seq2Seq) architectures:
Encoder–Decoder Models (Sutskever et al., 2014):
Encode source sentence → generate translated target sentence.
Attention Mechanism (Bahdanau et al., 2015):
Improves translation by focusing on relevant parts of the source sentence.
Concept Description
Input Type Sequence of words or tokens (source language).
Output Type Sequence of words or tokens (target language).
Goal Preserve meaning and fluency between languages.
Modern Models Transformers (BERT, GPT, T5).
Impact Powering systems like Google Translate, DeepL, Meta NLLB.

6. Structured Output Learning

Structured output tasks involve predicting multiple interrelated outputs rather than a
single scalar or categorical label.
Instead of predicting a single number or class, the goal is to output a complex structure such
as a sequence, tree, or image segmentation map.
Ex: Parsing (Natural Language Processing): Converts a sentence into a syntactic tree showing
the grammatical relationships between words. Example: Input: "The cat sat on the mat.“
Output: Parse tree with nodes tagged as noun, verb, preposition, etc.
• Image Segmentation (Computer Vision):
Each pixel is classified into a category such as road, building, or vegetation.
Example: Detecting roads in aerial imagery (Mnih & Hinton, 2010).
• Image Captioning: The system generates a sentence describing an image.
Input: Image of a dog playing with a ball. Output: "A brown dog is playing fetch in the park."

7. Anomaly Detection
Anomaly detection involves identifying unusual or abnormal data points that deviate from
expected patterns.
Applications: Credit Card Fraud Detection: A model learns the normal pattern of a user’s
transactions.
If a purchase occurs in an unusual location or amount, it flags it as fraudulent.
• Medical Diagnosis: Detecting rare symptoms that deviate from typical patient data.
• Network Security: Detecting irregular network traffic indicating cyberattacks.
Approach: Machine learning models estimate a probability distribution over
normal data. When a new observation has low probability, it is marked as
anomalous. (Chandola et al., 2009) provides a comprehensive survey.

8. Synthesis and Sampling

In synthesis or generative tasks, the algorithm creates new data samples resembling
the training set.
Examples Texture Generation (Graphics): Automatically generating textures for games or
landscapes to reduce manual work.
Speech Synthesis: Converts text into spoken audio — generating a waveform that sounds
human-like.
Image Generation (GANs): Produces realistic images of faces, scenes, or artworks.
Task Type Description Example Key Reference
Structured Output Predicts interrelated Parsing, Collobert (2011),
values (trees, sequences) Captioning Vinyals et al. (2015)
Anomaly Detection Flags unusual events Credit card Chandola et al. (2009)
fraud
Synthesis / Sampling Generates realistic data Texture, Speech Luo et al. (2013)

9. Imputation, Denoising, and Density Estimation

These three tasks—imputation, denoising, and density estimation—are fundamental to
probabilistic and generative learning. Each involves learning or inferring missing or
hidden aspects of the data.

9.1 Imputation of Missing Values

Definition: In missing value imputation, the learning algorithm is given a feature vector
with one or more missing components xi.

The task is to predict the missing entries based on the observed ones. Example
Medical Data: A patient’s record may lack blood test results. The model can infer
the missing hemoglobin level based on age, weight, and other known medical parameters.
Sensor Networks: Some sensors may fail, so the system estimates the missing
readings from nearby sensors.

Mathematical Basis
If we know the joint probability distribution p(x), we can compute:

where x−i denotes all variables except xi.

The conditional expectation or mode of this distribution gives the imputed value.
Applications
• Healthcare record completion
• Recommendation systems (predicting missing ratings in user–item matrices)
• Sensor fusion and time-series reconstruction
10. Denoising
In denoising, the algorithm receives a corrupted input:
where the corruption process is unknown.

The task is to recover the clean signal x or estimate the conditional probability:
Example: Image Denoising: Removing Gaussian noise or motion blur from an image.
Audio Denoising: Filtering noise from voice recordings.
Approach: Denoising Autoencoders (DAE): Learn a mapping

The model learns robust features that help reconstruct clean data from noise.
Significance: Denoising improves model robustness and helps uncover the
underlying structure of data even when partially corrupted.

Density Estimation (Probability Distribution Learning)

Definition: In density estimation, the goal is to learn a model

that approximates the true data distribution.

For continuous data, this is a probability density function; for discrete data, a probability
mass function.
Intuition: The model should learn where data points are likely to occur (high probability
regions) and where they are rare (low probability regions).
Examples: Estimating the distribution of handwritten digits (MNIST dataset).
Modeling customer purchase behaviors.
Challenge
Computing or marginalizing p(x) exactly is often computationally intractable, especially in
high dimensions.
Approximation methods include:
• Variational Inference (VI)
• Monte Carlo Sampling
• Generative Models (e.g., GANs, VAEs, Normalizing Flows)
Task Goal Input Output Example
Predict Incomplete Predict
Imputation missing values data x-i Estimated xi missing
medical test
Denoising Remove corruption Noisy data x Clean data x Image noise
removal
Density Learn probability Raw Modeling
Estimation distribution samples (x) Pmodel(x) handwritten
digits

The Performance Measure, P

In order to evaluate the abilities of a machine learning algorithm, we must design a
quantitative measure of its performance.
Usually this performance measure P is specific to the task T being carried out by the system.
For tasks such as classification, classification with missing inputs, and
transcription, we often measure the accuracy of the model.
Accuracy is just the proportion of examples for which the model produces the correct output.
We can also obtain equivalent information by measuring the error rate, the proportion of
examples for which the model produces an incorrect output.
We often refer to the error rate as the expected 0-1 loss. The 0-1 loss on a particular
example is 0 if it is correctly classified and 1 if it is not.
For tasks such as density estimation, it does not make sense to measure accuracy, error rate,
or any other kind of 0-1 loss.
Instead, we must use a different performance metric that gives the model a continuous
valued score for each example.
The most common approach is to report the average log-probability the model assigns to
some examples
The choice of performance measure may seem straightforward and objective, but it is often
difficult to choose a performance measure that corresponds well to the desired behavior of
the system.
In some cases, this is because it is difficult to decide what should be measured.

True Word Model Probability

“cat” 0.8
“sat” 0.5
“mat” 0.1
(Using natural log)
log(0.8) = −0.223
log(0.5) = −0.693
log(0.1) = −2.302
Step 2: Compute the average log-probability
−0.223−0.693−2.302
Avg log−prob =
3
= −1.072
Interpretation
• The value −1.072 is the model’s average performance per word.
• Higher (less negative) = better model
• If another model produced −0.7 average log-probability, it would be better.

For example, when performing a transcription task, should we measure the accuracy of the
system at transcribing entire sequences, or should we use a more fine-grained performance
measure that gives partial credit for getting some elements of the sequence correct?
True Statement the cat sat“ System Output "the cat sat“ Acc : 100%

When performing a regression task, should we correct the system more if it frequently
makes medium-sized mistakes or if it rarely makes very large mistakes?
True Statement "the cat sat on the mat“ System Output "the cat sat in the mat"

These kinds of design choices depend on the application.

The Experience, E
Machine learning algorithms can be broadly categorized as unsupervised or supervised by
what kind of experience they are allowed to have during the learning process.
Most of the learning algorithms can be understood as being allowed to experience
an entire dataset.
One of the oldest datasets studied by statisticians and machine learning researchers is the
Iris dataset (Fisher, 1936).
It is a collection of measurements of different parts of 150 iris plants. Each individual plant
corresponds to one example.
The features within each example are the measurements of each of the parts of the plant:
the sepal length, sepal width, petal length and petal width.
The dataset also records which species each plant belonged to. Three
different species are represented in the dataset.
Unsupervised learning algorithms experience a dataset containing many features, then
learn useful properties of the structure of this dataset.
In the context of deep learning, we usually want to learn the entire probability distribution
that generated a dataset, whether explicitly as in density estimation or implicitly for tasks
like synthesis or denoising.
Some other unsupervised learning algorithms perform other roles, like clustering, which
consists of dividing the dataset into clusters of similar examples.
Supervised learning algorithms experience a dataset containing features, but each
example is also associated with a label or target.
Learning Type What Deep Learning Learns IRIS Example
Explicit distribution The full probability Model learns typical petal lengths
learning (density distribution P(x) of each species and their variances
estimation)
Implicit distribution How to generate
Model synthesizes new realistic
learning (synthesis) new samples from P(x)
IRIS flower measurements
Implicit distribution How to map noisy samples Model corrects noisy
learning (denoising) back to the high-probability measurements to biologically
region of P(x) likely IRIS values
For example, the Iris dataset is annotated with the species of each iris plant.
A supervised learning algorithm can study the Iris dataset and learn to classify iris plants
into three different species based on their measurements.
Some machine learning algorithms do not just experience a fixed dataset. For example,
reinforcement learning algorithms interact with an environment, so there is a feedback loop
between the learning system and its experiences.
Most machine learning algorithms simply experience a dataset. A dataset can be described
in many ways. In all cases, a dataset is a collection of examples, which are in turn
collections of features.
There is no formal definition of supervised and unsupervised learning, there is no rigid
taxonomy of datasets or experiences.
The structures described here cover most cases, but it is always possible to design new ones
for new applications.

Capacity, Overfitting and Underfitting

The central challenge in machine learning is that we must perform well on new, previously
unseen inputs—not just those on which our model was trained.
The ability to perform well on previously unobserved inputs is called
generalization.
Typically, when training a machine learning model, we have access to a training set, we
can compute some error measure on the training set called the training error, and we reduce
this training error.
So far, what we have described is simply an optimization problem.
What separates machine learning from optimization is that we want the generalization
error, also called the test error, to be low as well.
The generalization error is defined as the expected value of the error on a new input.
Here the expectation is taken across different possible inputs, drawn from the distribution
of inputs we expect the system to encounter in practice.
We typically estimate the generalization error of a machine learning model by
measuring its performance on a test set of examples that were collected separately from
the training set.
If the training and the test set are collected randomly.
If we are allowed to make some assumptions about how the training and test set are
collected, then we can make some progress.
The train and test data are generated by a probability distribution over datasets called
the data generating process.
Suppose you train a neural network on the IRIS flower dataset:
Training error: 0% (the model memorized all 150 flowers)
Test error: 18% (fails on unseen flowers)
Even though training error is perfect, the model is bad because it cannot identify new plants
correctly. This is why test error matters more.
We typically make a set of assumptions known collectively as the these assumptions i.i.d.
assumptions (Independent and identically distributed random variables) are that the
examples in each dataset are independent from each other, and that the train set and test set
are identically distributed, drawn from the same probability distribution as each other.
This assumption allows us to describe the data generating process with a probability
distribution over a single example.
The same distribution is then used to generate every train example and every test example.
We call that shared underlying distribution the data generating distribution, denoted pdata.
This probabilistic framework and the i.i.d. assumptions (Independent and identically
distributed random variables) allow us to mathematically study the relationship between
training error and test error.
One immediate connection we can observe between the training and test error is that the
expected training error of a randomly selected model is equal to the expected test error of
that model.
Suppose we have a probability distribution p( x, y) and we sample from it repeatedly to
generate the train set and the test set.
For some fixed value w, the expected training set error is exactly the same as the expected
test set error, because both expectations are formed using the same dataset sampling
process.
Simple Example (IRIS Dataset)
Assume each IRIS flower is an i.i.d. sample from the plant population. Train a classifier on
100 flowers and test on 50.
Training error = 4% Model misclassified 4 out of 100.
Test error = 6% Model misclassified 3 out of 50 unseen flowers.
Why are they close?
Because both sets (training and test) are i.i.d. samples from the same population distribution.
If the data were not i.i.d. (e.g., one class appears only in the test set), the errors would not
align, and mathematical guarantees would break.
The only difference between the two conditions is the name we assign to the dataset we
sample
We sample the training set, then use it to choose the parameters to reduce training set error,
then sample the test set.
Under this process, the expected test error is greater than or equal to the expected value of
training error.
The factors determining how well a machine learning algorithm will perform are its ability
to:
1. Make the training error small.
2. Make the gap between training and test error small.
These two factors correspond to the two central challenges in machine learning: underfitting
and overfitting.
Underfitting occurs when the model is not able to obtain a sufficiently low error value on
the training set.
Overfitting occurs when the gap between the training error and test error is too large.
We can control whether a model is more likely to overfit or underfit by altering its capacity.
Informally, a model’s capacity is its ability to fit a wide variety of functions.
 Fit Type Test Error Explanation
Training Error
Underfitting High High Model too simple, fails to capture
patterns
Overfitting Low High Model too complex, memorizes
training data, poor generalization
Good Fit Low Low Model captures patterns well,
generalizes

Models with low capacity may struggle to fit the training set.
Models with high capacity can overfit by memorizing properties of the training set that do
not serve them well on the test set.

One way to control the capacity of a learning algorithm is by choosing its hypothesis
space, the set of functions that the learning algorithm is allowed to select as being the
solution.

For example, the linear regression algorithm has the set of all linear functions of its input
as its hypothesis space.

We can generalize linear regression to include polynomials, rather than just linear
functions, in its hypothesis space.

Doing so increases the model’s capacity.

A polynomial of degree one gives us the linear regression model with which we are
already familiar, with prediction yˆ = b + wx.

By introducing x2 as another feature provided to the linear regression model, we can

learn a model that is quadratic as a function of x: yˆ = b + w1x1 + w2x2
Step 1: Recall the concept
Hypothesis space (H): The set of all functions/models that the learning algorithm can
choose from.
Model capacity: How complex a function the model can represent.
Control: By restricting H, we control underfitting and overfitting.
x y
1 2
2 4
3 6
4 8
 Step 3: Choose hypothesis spaces
Option 1: Very simple hypothesis space (low capacity)
Hypothesis: y=b (a constant)
H1 = { all constant functions }
Training: Model can only choose a single constant to fit all points.
Result: Underfits the data (high training and test error)
Option 2: Linear hypothesis space (moderate capacity)
Hypothesis: y=mx+b
H2 = { all linear functions }
Training: Model chooses slope m=2 and intercept b=0
Result: Perfectly fits this dataset (low training and test error)

• Interpretation: Good choice of hypothesis space

Option 3: High-capacity hypothesis space
Hypothesis: y=a0+a1x+a2x^2+⋯+anx^n (degree n polynomial, n = 10)
H3 = { all polynomials of degree ≤ 10 }
Training: Model can fit exactly through all points and even noise if present
Result: Could overfit if there were noise (training error very low, test error high)

Step 4: Conclusion By choosing H, we control the model’s capacity:

 Too small H → underfitting
 Too large H → overfitting
 Right-sized H → good generalization
Hypothesis Space Model Complexity Training Error Test Error Fit Type
y=b Low High High Underfit
y = mx + b Moderate Low Low Good fit
y = polynomial High Very Low High (if noise) Overfit
Though this model implements a quadratic function of its input, the output is still a linear
function of the parameters, so we can still use the normal equations to train the model in
closed form.
We can continue to add more powers of x as additional features, for example to obtain a
polynomial of degree 9:
Machine learning algorithms will generally perform best when their capacity is appropriate
in regard to the true complexity of the task they need to perform and the amount of training
data they are provided with.

Models with insufficient capacity are unable to solve complex tasks. Models with high
capacity can solve complex tasks, but when their capacity is higher than needed to solve the
present task they may overfit.

Fig. 5.2 shows this principle in action. We compare a linear, quadratic and degree-9
predictor attempting to fit a problem where the true underlying function is quadratic.
The linear function is unable to capture the curvature in the true underlying problem, so it
underfits.
The degree-9 predictor is capable of representing the correct function, but it is also capable
of representing infinitely many other functions that pass exactly through the training
points, because we have more parameters than training examples.
We have little chance of choosing a solution that generalizes well when so many wildly
different solutions exist.
In this example, the quadratic model is perfectly matched to the true structure of the task so
it generalizes well to new data. Figure 5.2: We fit three models to this example training set.

The training data was generated synthetically, by randomly sampling x values and
choosing y deterministically by evaluating a quadratic function.

(Left) A linear function fit to the data suffers from underfitting—it cannot capture the
curvature that is present in the data.

(Center) A quadratic function fit to the data generalizes well to unseen points. It does not
suffer from a significant amount of overfitting or underfitting.
(Right) A polynomial of degree 9 fit to the data suffers from overfitting. Here we used the
Moore-Penrose pseudoinverse to solve the underdetermined normal equations.
The solution passes through all of the training points exactly, but we have not been lucky
enough for it to extract the correct structure.

underlying function.

this area.
ng a model’s capacity by changing the number of
input features it has (and simultaneously adding new parameters associated with those
features).

determined only by the choice of model.

when varying the parameters in order to reduce a training objective.

function within this family is a very difficult optimization problem.

one that significantly reduces the training error.

imperfection of the optimization algorithm,
mean that the learning algorithm’s effective capacity may be less than the representational
capacity of the model family

behave differently. At the left end of the graph, training error and generalization error are
both high. This is the underfitting rule.

generalization error increases. Eventually, the size of this gap outweighs the decrease in
training error, and we enter the overfitting rule, where capacity is too large, above the
optimal capacity.






Regularization
The no free lunch theorem implies that we must design our machine learning algorithms to
perform well on a specific task.
We do so by building a set of preferences into the learning algorithm. When these
preferences are aligned with the learning problems we ask the algorithm to solve, it
performs better.
Regularization is a technique used in machine learning and deep learning to reduce
overfitting by discouraging the model from becoming too complex.
It does this by adding a penalty to the loss function, which limits how much the model can
rely on large weights or complicated structures.
In simple terms, regularization helps the model focus on the most important patterns in
the training data and ignore noise or random fluctuations, leading to better performance on
new, unseen data.
There are different types of regularization:

L1 regularization (Lasso): Makes some weights exactly zero, encouraging sparsity.

L2 regularization (Ridge or weight decay): Keeps weights small, encouraging smoother

Regularization Feature
Name Effect on Weights Initial Weights Example
Type Selection
Result

Some weights
L1 Lasso Yes ([4.9,-0.4,0.3]) ([4.9, 0,
become exactly 0 0.3])
All weights shrink but ([3.7, -0.5,
L2 Ridge No ([5.2,-0.8,0.46])
stay nonzero 0.25])
 models.

L1 Regularization (Lasso) — Choosing only the L2 Regularization (Ridge) — Balanced

strongest factors decision
L1 makes some weights exactly zero, meaning the L2 keeps all weights small, none become
model ignores certain features. zero.
IT Job Example: You want to choose an IT IT Job Example:
company but you decide: Only 2 main factors You consider every factor but in a
matter: Salary & Work-Life Balance balanced way:

Completely ignore: Location, Tech stack, Company Salary, Tech stack, Work-life balance,
size, Leave policy, Onsite opportunities, Manager, Company reputation, Career growth,
Job title Hybrid/remote, Leave policy
* All are important
This is sparse decision making → EXACTLY what L2 = “Consider everything, but keep it
L1 does. balanced.”
You end up with TCS because IBM or Infosys because: They provide a
➢ Salary is high & Work-life balance good mix of salary, learning, stability,
culture, hybrid work, benefits
is excellent

Hyperparameters and Validation Sets

Most machine learning algorithms have several settings that we can use to control the
behavior of the learning algorithm. These settings are called hyperparameters.
The values of hyperparameters are not adapted by the learning algorithm itself (though we
can design a nested learning procedure where one learning algorithm learns the best
hyperparameters for another learning algorithm).
Hyperparameters are parameters whose values control the learning process and determine
the values of model parameters that a learning algorithm ends up learning.

The prefix 'hyper_' suggests that they are 'top-level' parameters that control the learning
process and the model parameters that result from it.
Term Meaning Vehicles Dataset Example
Weights (W) Importance given to features High weight to wheels, low weight to
background
Biases (b) Adjustment that helps Helps detect vehicle even if image
activation contrast low
Layers Stages of feature extraction Layer 1 = edges,
Layer 2 = wheels/windows
Hidden Neurons inside layers Each unit learns parts: wheel, door,
Units light
 Learning Rate Speed of weight updates High LR → learns too fast;
Low LR → too slow
Batch Size Images processed per update Batch = 32 vehicle images at a time
Epochs Number of times dataset is seen 10 epochs = sees all vehicle images 10
times

What is the difference between parameters and hyperparameters?

Parameters are the internal values (like weights and biases in a neural network) that are
automatically learned from the training data and directly determine the model’s output.
Hyperparameters are the external settings (like learning rate, batch size,number of
layers) manually defined before training that guide the learning process but are not updated
during it.
Sometimes a setting is chosen to be a hyperparameter that the learning algorithm
does not learn because it is difficult to optimize
More frequently, we do not learn the hyperparameter because it is not appropriate to learn
that hyperparameter on the training set.
This applies to all hyperparameters that control model capacity.
If learned on the training set, such hyperparameters would always choose the maximum
possible model capacity, resulting in overfitting (refer to Fig. ).
For example, we can always fit the training set better with a higher degree polynomial
and a weight decay setting of λ = 0 than we could with a lower degree polynomial and a
positive weight decay setting.

🔵 Parameters (Learned automatically by the model)

Parameters are internal values learned automatically by the model during training.
Examples: weights and biases inside a neural network.
Bank Example: The bank trains a model using past customer data:
Salary, Credit Score, Existing Loans, Age, EMI History, Bank Balance
During training, the model learns how important each factor is.

👉 These learned importance values are parameters.

 Ex: Weight for Salary = 0.8 Weight for Credit Score = 0.6
Weight for EMI Missed Payments = -1.2

These values are automatically learned. The bank does not set them manually.

Hyperparameters (Set manually before training)

Hyperparameters are external settings that control how the model learns, not what it
learns. These are fixed by humans before training begins.

🎯 Bank Example:
Before building a loan model, the bank decides:
i. Learning rate → How fast the model learns patterns
ii. Batch size → How many customer records processed at once
Ex: 32 customers per batch
iii. Number of layers → How many decision layers the model use
iv. Number of training epochs → How many times the model sees all data
Ex: 50 epochs

Category Examples

Training Control Learning rate, batch size, number of epochs

Model Architecture Number of layers, number of neurons, kernel size

Regularization Dropout rate, L1/L2 penalty

Optimization Choice of optimizer (Adam, SGD)

These values are not [Link] are chosen by the data scientist before training.
To solve this problem, we need a validation set of examples that the training algorithm does
not observe.
Earlier we discussed how a held-out test set, composed of examples coming from the same
distribution as the training set, can be used to estimate the generalization error of a learner,
after the learning process has completed.
It is important that the test examples are not used in any way to make choices
about the model, including its hyperparameters.
For this reason, no example from the test set can be used in the validation set. Therefore,
we always construct the validation set from the training data.
Specifically, we split the training data into two disjoint subsets.
One of these subsets is used to learn the parameters.
The other subset is our validation set, used to estimate the generalization error during or
after training, allowing for the hyperparameters to be updated accordingly.
The subset of data used to learn the parameters is still typically called the training
set, even though this may be confused with the larger pool of data used for the entire
training process.
The subset of data used to guide the selection of hyperparameters is called the validation
set.
Typically, one uses about 80% of the training data for training and 20% for validation.
Since the validation set is used to “train” the hyperparameters, the validation set error will
underestimate the generalization error, though typically by a smaller amount than the
training error.
After all hyperparameter optimization is complete, the generalization error may be
estimated using the test set.

Cross-Validation in ML

Cross validation is a technique used in machine learning to evaluate the performance of a

model on unseen data.
It involves dividing the available data into multiple folds or subsets, using one of these folds
as a validation set, and training the model on the remaining folds.

This process is repeated multiple times, each time using a different fold as the validation
set.
Finally, the results from each validation step are averaged to produce a more robust estimate
of the model’s performance.
Cross validation is an important step in the machine learning process and helps to ensure that
the model selected for deployment is robust and generalizes well to new data.
Cross-validation is a technique for evaluating ML models by training several ML models on
subsets of the available input data and evaluating them on the complementary subset of the
data. Use cross-validation to detect overfitting, ie, failing to generalize a pattern.
Why do we need cross-validation?
Cross Validation is commonly used in Machine Learning to compare different models and
select the most appropriate one for a specific problem. It is both easy to understand, easy to
implement, and less biased than other methods.
Purpose of cross validation in machine learning
The purpose of cross–validation is to test the ability of a machine learning model to predict
new data.
It is also used to flag problems like overfitting or selection bias and gives insights on how
the model will generalize to an independent dataset.

Bias and Variance in Machine Learning

Machine learning is a branch of Artificial Intelligence, which allows machines to perform
data analysis and make predictions.
However, if the machine learning model is not accurate, it can make predictions errors,
and these prediction errors are usually known as Bias and Variance.
In machine learning, these errors will always be present as there is always a slight
difference between the model predictions and actual predictions.

The main aim of ML/data science analysts is to reduce these errors in order to get ore
accurate results.
Aspect Bias Variance
Definition
Error due to overly Error due to sensitivity to training
simple assumptions data
Model Behavior Underfits the data Overfits the data
Model Complexity Too low (simple model) Too high (complex model)
Training Error High Very Low
 Test Error High High
Reason Model cannot capture real Model memorizes noise in training
pattern data
Example Model Linear regression with 1 feature Deep neural network or 15-degree
polynomial
Outcome Misses patterns Fails to generalize
Goal in ML Reduce bias without increasing Reduce variance without increasing
variance too much bias too much

Errors in Machine Learning?

In machine learning, an error is a measure of how accurately an algorithm can make
predictions for the previously unknown dataset.
On the basis of these errors, the machine learning model is selected that can perform best on
the particular dataset.
There are mainly two types of errors in machine learning, which are:
Reducible errors: These errors can be reduced to improve the model accuracy. Such errors
can further be classified into bias and Variance.
Irreducible errors: These errors will always be present in the model regardless of which
algorithm has been used.
The cause of these errors is unknown variables whose value can't be
reduced.

What is Bias?

In general, a machine learning model analyses the data, find patterns in it and make

predictions.
While training, the model learns these patterns in the dataset and applies them to test data
for prediction.
While making predictions, a difference occurs between prediction values made by the
model and actual values and expected values, and this difference is known as bias errors
or Errors due to bias.
It can be defined as an inability of machine learning algorithms such as
Linear Regression to capture the true relationship between the data points.
Each algorithm begins with some amount of bias because bias occurs from
assumptions in the model, which makes the target function simple to learn.
A model has either:

Low Bias: A low bias model will make fewer assumptions about the form of the target
function.
High Bias: A model with a high bias makes more assumptions, and the model becomes
unable to capture the important features of our dataset.
A high bias model also cannot perform well on new data.
Generally, a linear algorithm has a high bias, as it makes them learn fast.
The simpler the algorithm, the higher the bias it has likely to be introduced. Whereas a
nonlinear algorithm often has low bias.
Some examples of machine learning algorithms with low bias are Decision Trees,
kNearest Neighbours and Support Vector Machines.

At the same time, an algorithm with high bias is Linear Regression, Linear Discriminant
Analysis and Logistic Regression.

Algorithm Bias Level Variance Level Simple Example

Decision Tree Low Bias High Variance Memorizes training patterns of pass/fail
students exactly
k-Nearest Slight noise in data changes predictions
Neighbours Low Bias High Variance drastically(Difficulty level of exam, Quality
(kNN) of teaching / coaching, Health issues during
exam, Time management during the exam)
Can create very detailed boundaries for
SVM (RBF Low Bias Moderate / pass/fail classification(Study hours,
Kernel) High Variance Attendance, Internal marks The relationship
is non-linear)
Linear High Bias Low Variance Models exam marks as straight-line relation
Regression with study hours
Logistic High Bias Low Variance Predicts pass/fail with linear threshold,
Regression misses curved relations
LDA (Linear Fails when classes are non-linear or overlap
Discriminant High Bias Low Variance in complex shapes
Analysis)

Ways to reduce High Bias:

 High bias mainly occurs due to a much simple model. Below are some ways to reduce the
high bias
 Identify potential sources of bias.
 Identify accurate representative data
 Increase the input features as the model is underfitted.
 Decrease the regularization term.
 Use more complex models, such as including some polynomial
features.
Regularization is a technique used in machine learning and deep learning to prevent
overfitting and improve the generalization performance of a model. It involves adding a
penalty term to the loss function during training.
What is a Variance Error?
The variance would specify the amount of variation in the prediction if the different
training data was used.

In simple words, variance tells that how much a random variable is different from its
expected value.
Ideally, a model should not vary too much from one training dataset to another, which
means the algorithm should be good in understanding the hidden mapping between inputs
and output variables.
Case Model Example Training Test Real-World
Behavior Assumption Error Error Result
“EV adoption Wrong predictions;
High Bias Too simple depends only on High High misses important
(underfits) petrol price.” factors

Too “EV adoption Fails for new people;

High complex depends on 40 Very Low High memorizes training
Variance (overfits) tiny personal data
habits.”
 “EV adoption
depends on Accurate predictions;
Balanced Just right income, Low Low good generalization
Model charging,
subsidies, fuel
cost, range.”

Variance errors are either of low variance or high variance.

How much the predicted target values from models trained using different samples vary
from each other;
Noise: How much the sample target values differ from the true model.
Low variance means there is a small variation in the prediction of the target function with
changes in the training data set.
At the same time, High variance shows a large variation in the prediction of the target
function with changes in the training dataset.
A model that shows high variance learns a lot and perform well with the training dataset, and
does not generalize well with the unseen dataset.
As a result, such a model gives good results with the training dataset but shows high error rates
on the test dataset.
Since, with high variance, the model learns too much from the dataset, it leads to overfitting
of the model. A model with high variance has the below problems:

✓ A high variance model leads to overfitting.

✓ Increase model complexities.
✓ Usually, nonlinear algorithms have a lot of flexibility to fit the model, have high
variance.
Some examples of machine learning algorithms with low variance are, Linear Regression,
Logistic Regression, and Linear discriminant analysis.
At the same time, algorithms with high variance are decision tree, Support Vector
Machine, and K-nearest neighbours.

Ways to Reduce High Variance:

Reduce the input features or number of parameters as a model is overfitted.
✓ Do not use a much complex model.
✓ Increase the training data.
✓ Increase the Regularization term.
➢ Different Combinations of Bias-Variance
➢ Low-Bias, Low-Variance:
✓ The combination of low bias and low variance shows an ideal machine
learning model. However, it is not possible practically.
➢ Low-Bias, High-Variance:
✓ With low bias and high variance, model predictions are inconsistent and
accurate on average.
✓ This case occurs when the model learns with a large number of parameters and hence
leads to an overfitting

High-Bias, Low-Variance:
With High bias and low variance, predictions are consistent but inaccurate on average.
This case occurs when a model does not learn well with the training dataset or uses few
numbers of the parameter. It leads to underfitting problems in the model.

High-Bias, High-Variance:

With high bias and high variance, predictions are inconsistent and also inaccurate on
average.
Case Meaning Real-World Example Explanation
1. Low Bias, Low Accurate The archer understands wind,
Variance and A skilled archer hitting the distance, angle (model captures
consistent bullseye repeatedly real patterns) and performs
consistently (low variance).
2. High Bias, Low A thermostat always showing It is consistently off-targetasdue
room temperature 5°Ctohigh
Variance Consistently a systematic error (high bias), but
wrong very stable (low variance).
3. Low Bias, High She understands how to frame the
Variance Accurate on A talented photographer photo (low bias) but camera
average but using an unstable tripod shakes create large fluctuations
inconsistent (high variance).
4. High Bias, Inaccurate and A new learner throwing darts Misses target due to wrong
High Variance inconsistent randomly, far from the technique (high bias) and
bullseye produces scattered throws (high
variance).

There are four possible combinations of bias and variances, which are represented by the

below diagram
 Maximum Likelihood Estimation

The EM algorithm is considered a latent variable model to find the local maximum
likelihood parameters of a statistical model, proposed by Arthur Dempster, Nan Laird, and
Donald Rubin in 1977.
The EM (Expectation-Maximization) algorithm is one of the most commonly
used terms in machine learning to obtain maximum likelihood estimates of variables that are
sometimes observable and sometimes not observable .
However, it is also applicable to unobserved data or sometimes called latent (hidden).
It has various real-world applications in statistics, including obtaining the mode of the
posterior marginal distribution of parameters in machine learning and data mining
applications.
Note: Posterior: Conditional probability distribution representing what parameters are likely
after observing the data object.
Likelihood: The probability of falling under a specific category or class. In most real-life
time applications of machine learning, it is found that several relevant learning features are
available, but very few of them are observable, and the rest are unobservable.
Example used: Estimating the number of customers from two types (regular and tourists)
in a coffee shop where we cannot directly see who is who.

Real-World Scenario
A coffee shop wants to estimate:
How many regular customers visit daily
How many tourists visit daily
But customers do not tell who they are → hidden/unobserved variable. They
only record purchase amount, which is observable

Step Meaning (Simple) Coffee Shop Example What EM Does

E-Step Estimate the Estimate the probability Uses current model to

(Expectation) hidden/missing that each customer is a “guess” hidden data
information based regular or tourist based
on what we know on purchase amount
M-Step Recalculate model Update estimates of: — Improves model using
(Maximiz parameters from the Average spend of updated guesses
ation) estimated values regulars — Average
spend of tourists —
Percentage of regular vs
tourist customers
 Re-estimate customer Converges to best
Repeat E Keep types (maximum likelihood)
and M imp → update averages estimates
roving until stable → re-estimate again…
Find best model Find: ✔ true proportion Maximum
Goal explaining data with of regular customers ✔ Likelihood
hidden variables true proportion of Estimation
tourists

If the variables are observable, then it can predict the value using instances.

On the other hand, the variables which are latent (hidden) or directly not observable, for such
variables Expectation-Maximization (EM) algorithm plays a vital role to predict the value
with the condition that the general form of probability distribution governing those latent
variables is known to us.

Application:

A medical researcher is studying the relationship between a patient's age and whether
they have a particular disease (yes/no).
Using a dataset of patients, the researcher uses MLE to estimate the coefficients in a
logistic regression model that predicts the probability of disease based on age.

What is an EM algorithm?

The Expectation-Maximization (EM) algorithm is defined as the combination of various

unsupervised machine learning algorithms, which is used to determine the local
maximum likelihood estimates (MLE) or maximum a posteriori estimates (MAP) for
unobservable variables in statistical models.
Further, it is a technique to find maximum likelihood estimation when the latent
variables are present. It is also referred to as the latent variable model.

A latent variable model consists of both observable and unobservable variables where
observable can be predicted while unobserved are inferred from the observed variable.
These unobservable variables are known as latent variables.

EM Algorithm
The EM algorithm is the combination of various unsupervised
ML algorithms, such as the k-means clustering algorithm
Being an iterative approach, it consists of two modes.
In the first mode, we estimate the missing or latent variables. Hence it is
referred to as the Expectation/estimation step (E-step).
Further, the other mode is used to optimize the parameters of the models so that it can
explain the data more clearly.
The second mode is known as the maximization-step or M-step

Expectation step (E - step): It involves the estimation (guess) of all missing values in the
dataset so that after completing this step, there should not be any missing value.
Maximization step (M - step): This step involves the use of estimated data in the E-step

and updating the parameters.

Repeat E-step and M-step until the convergence of the values occurs.
The primary goal of the EM algorithm is to use the available observed data of the dataset to
estimate the missing data of the latent variables and then use that data to update the values of
the parameters in the M-step.

What is Convergence in the EM algorithm?

Convergence is defined as the specific situation in probability based on intuition,

e.g., if there are two random variables that have very less difference in their probability,
then they are known as converged.
In other words, whenever the values of given variables are matched with each other, it is
called convergence.

Convergence in the Expectation-Maximization (EM) algorithm refers to the point at

which the algorithm has reached a stable set of parameter estimates, meaning that further
iterations will not significantly change these estimates.

Steps in EM Algorithm

The EM algorithm is completed mainly in 4 steps, which include Initialization Step,

Expectation Step, Maximization Step, and Convergence Step. These steps are explained
as follows:

Initialization Step :1st Step: The very first step is to initialize the parameter

values. Further, the system is provided with incomplete observed data with the
assumption that data is obtained from a specific model.
Expectation Step 2nd Step: This step is known as Expectation or E-Step, which is used to
estimate or guess the values of the missing or incomplete data using the observed data.
Further, E-step primarily updates the variables.
Maximization Step 3rd Step: This step is known as Maximization or M-step, where we use
complete data obtained from the 2nd step to update the parameter values.
Further, Mstep primarily updates the hypothesis.

Convergence Step 4th step: The last step is to check if the values of latent variables are
converging or not. If it gets "yes", then stop the process; else, repeat the process from step 2
until the convergence occurs.

Applications of EM algorithm

The primary aim of the EM algorithm is to estimate the missing data in the latent
variables through observed data in datasets.
The EM algorithm or Latent Variable Model has a broad range of real-life applications
in machine learning.
These are as follows:
 The EM algorithm is applicable in data clustering in machine learning.
  It is often used in computer vision and NLP (Natural language processing).
 It is used to estimate the value of the parameter in mixed models such as the Gaussian
Mixture Model and quantitative genetics.
 It is also used in psychometrics for estimating item parameters and latent
abilities of item response theory models.
 It is also applicable in the medical and healthcare industry, such as in image
reconstruction and structural engineering.
 It is used to determine the Gaussian density of a function.

Explore how MLE is used in classification tasks, such as object recognition or sentiment
analysis, by deriving and implementing the MLE-based classifier

Maximum Likelihood Estimation (MLE) is a statistical method used to find the parameters of
a model that maximize the probability of observing the given data.
In the context of classification tasks, MLE is used to determine the optimal model parameters
that best fit the training data, allowing for accurate predictions on new, unseen data.
The MLE Process in Classification Model Selection: Choose a suitable probability
distribution for the data.
This choice often depends on the nature of the classification problem (e.g., Bernoulli
distribution for binary classification, multinomial distribution for multi-class classification).
Parameter Estimation: Define the model parameters. These parameters represent the
characteristics of the distribution that influence the classification decision.
Likelihood Function: Construct the likelihood function, which expresses the
probability of observing the given training data under the chosen model and parameters.
Maximization: Find the values of the model parameters that maximize the likelihood
function. This is typically done using optimization techniques like gradient ascent or
Newton-Raphson.

Classification: Once the optimal parameters are determined, the model can be used to
classify new data points by calculating their likelihood under the learned model and
assigning them to the class with the highest probability.

MLE in Object Recognition (Deep Learning)

In more complex tasks like object recognition, MLE is used in models like
convolutional neural networks (CNNs).
The process is conceptually similar: A CNN outputs a probability distribution over different
classes (e.g., cat, dog, etc.) for a given input image.
The probability distribution is based on the model's parameters (weights and biases).

-likelihood is maximized during training using backpropagation and stochastic

gradient descent to adjust the weights in the network.

Implementing the MLE-based classifier.

The EM algorithm is the combination of various unsupervised ML algorithms, such as the k-
means clustering algorithm.
Being an iterative approach, it consists of two modes.
In the first mode, we estimate the missing or latent variables.
Hence it is referred to as the Expectation/estimation step (E-step).
Further, the other mode is used to optimize the parameters of the models so that it can
explain the data more clearly.
The second mode is known as the maximization-step or M-step.

Expectation step (E - step): It involves the estimation (guess) of all missing values in the
dataset so that after completing this step, there should not be any missing value.
➢ Maximization step (M - step): This step involves the use of estimated data in
the E-step and updating the parameters
➢ Repeat E-step and M-step
until the convergence of the
values occurs.
➢ The primary goal of the EM
algorithm is to use the
available observed data of the
dataset to estimate the
missing data of the latent
variables and then use that
data to update the values of
the parameters in the M-step.

Steps in EM Algorithm

The EM algorithm is completed mainly in 4 steps, which include

Initialization Step, Expectation Step, Maximization Step, and Convergence Step.

These steps are explained as follows:

Initialization Step :1st Step: The very first step is to initialize the parameter values. Further,
the system is provided with incomplete observed data with the assumption that data is
obtained from a specific model
Expectation Step 2nd Step: This step is known as Expectation or E-Step, which is used to
estimate or guess the values of the missing or incomplete data using the observed data.
Further, E-step primarily updates the variables.
Maximization Step 3rd Step: This step is known as Maximization or M- step, where we
use complete data obtained from the 2nd step to update the parameter values. Further, Mstep
primarily updates the hypothesis.
Convergence Step 4th step: The last step is to check if the values of latent variables
are converging or not.

If it gets "yes", then stop the process; else, repeat the process from step 2 until the
convergence occurs.

In classification tasks, MLE is used to estimate model parameters that

maximize the probability of correctly classifying the data.
We construct a likelihood function based on the probability of the class labels
given the data and model parameters.
MLE maximizes the log-likelihood of observing the data by adjusting the
parameters through optimization methods like gradient descent.

The Basics of Bayesian Statistics

 Bayesian statistics is a branch of statistics that deals with uncertainty by
incorporating prior knowledge or beliefs into the analysis of data.
 It is named after Thomas Bayes, an 18th-century mathematician and theologian. The
fundamental concept in Bayesian statistics is the Bayes' theorem, which describes
how to update our beliefs in light of new evidence. Here are the basics of Bayesian
 statistics:
 Bayesian statistics is a powerful approach for making statistical inferences,
especially when dealing with small or complex data sets and when incorporating
prior information is valuable or necessary.
 However, the choice of prior can be subjective and influence results, so it's essential
to carefully consider and justify your prior assumptions in Bayesian analysis.
 Bayesian statistics is a statistical theory and approach to data analysis that
uses Bayes' theorem to describe the probability of an event based on previous
knowledge and observed and unobserved parameters.

Bayes' Theorem
Bayes' Theorem, named after 18th-century British mathematician Thomas Bayes, is a
mathematical formula for determining conditional probability.
Conditional probability is the likelihood of an outcome occurring based on a previous
outcome in similar circumstances.
The Bayes theorem is frequently referred to as the Bayes rule or Bayes Law.
One of the most well-known theories in machine learning, the Bayes theorem helps
determine the likelihood that one event will occur with unclear information while another
has already happened.

Bayes' theorem describes the probability of occurrence of an event related to any condition.
It is also considered for the case of conditional probability.
Bayes theorem is also known as the formula for the probability of “causes” Bayes'

Theorem: At its core, Bayesian statistics is built upon Bayes' theorem, which relates
conditional probabilities.
The theorem can be expressed as follows:

P(A | B) = [P(B | A) * P(A)] / P(B) P(A | B)

represents the probability of event A occurring given that event B has occurred. P(B | A)
is the probability of event B occurring given that event A has occurred.
P(A) and P(B) are the marginal probabilities of events A and B, respectively.

In the context of Bayesian statistics:

  P(A | B) is the posterior probability, which represents our updated belief in A
after observing B.
 P(B | A) is the likelihood, describing the probability of observing B if A is true.
 P(A) is the prior probability, representing our initial belief in A before
considering B.
 P(B) is the marginal likelihood or evidence, the probability of observing B
without considering A.
Prior Probability: In Bayesian statistics, prior knowledge or beliefs about the
probability of an event play a crucial role. This is known as the prior probability (P(A)),
which represents your initial beliefs about the likelihood of an event occurring before
observing any data. The choice of the prior can significantly influence the final results.
Likelihood: The likelihood (P(B | A)) represents the probability of observing the data (B)
given a specific hypothesis or model (A). It quantifies how well the hypothesis or model
explains the observed data. It's essential to choose an appropriate likelihood function that
accurately reflects the relationship between your data and the parameters you want to
estimate.
Posterior Probability: The posterior probability (P(A | B)) is the updated belief in the
hypothesis or model A after observing the data B.
It is calculated using Bayes' theorem and combines the prior knowledge and the likelihood
of the data.
The posterior probability provides a more informed and updated estimate of the event's
probability based on the observed data.

Bayesian Inference:
Bayesian inference involves using Bayes' theorem to estimate or update the parameters of a
statistical model based on observed data.
This process typically includes:
Choosing a prior distribution that reflects your initial beliefs.
Defining a likelihood function that describes the data-generation process.

Applying Bayes' theorem to calculate the posterior distribution, which

represents updated parameter estimates based on the data.