UNIT - 1

Q1: Vapnik-Chervonenkis (VC) Dimension in Machine Learning:

In Machine Learning, understanding the capacity and performance of a model is critical.

One important concept that helps in this understanding is the Vapnik-Chervonenkis (VC)
dimension. The VC dimension measures the ability of a hypothesis space (the set of all possible
models) to fit different patterns in a dataset.

Introduced by Vladimir Vapnik and Alexey Chervonenkis, this concept plays a vital role
in assessing the trade-off between model complexity and generalization. In simple terms, it
helps us understand how well a model can balance learning from the training data and
performing well on unseen data.
What is Shattering?
A hypothesis class is said to “shatter” a set of data points if, no matter how you label
those points (e.g., assign them as positive or negative), the hypothesis class has a function that
can correctly classify them.
Example of Shattering:
Imagine you have two points on a 2D plane.
 A straight line (linear hypothesis) can divide these two points in all possible ways based
on their labels (e.g., positive-negative or negative-positive). Hence, the hypothesis class
of straight lines shatters these two points.
 However, for three points that form a triangle, a straight line cannot shatter them if their
labels are mixed in a specific way.
This simple idea of shattering helps us measure the capacity of a model.
What is VC Dimension?
The VC dimension of a hypothesis class is the maximum number of points that the
hypothesis class can shatter.
 If a model can shatter three points but not four, its VC dimension is 3.
VC dimension gives a way to quantify the “complexity” of a model. A higher VC dimension
means the model is more complex and can handle more complicated data patterns.
Why is VC Dimension Important?
 Generalization: Models with too high a VC dimension may overfit (perform well on
training data but poorly on unseen data).
 Simplicity: A model with a lower VC dimension may underfit (fail to capture the patterns
in data).
Mathematical Foundations
The mathematical basis of the VC dimension allows us to analyze and understand the
relationship between a model’s complexity and its ability to generalize.
Formal Definition of VC Dimension
The VC dimension of a hypothesis class $H$ is the largest number of data points that can be
shattered by $H$.
In other words, for a dataset of size :
 If can shatter points, but not points, the VC dimension of $H$ is $n$.
Example:
 A straight line in 2D space has a VC dimension of 3. It can shatter any arrangement of 3
points, but it cannot shatter 4 points if one lies outside the plane formed by the others.
Bounds of VC Dimension
The VC dimension plays a crucial role in providing theoretical guarantees about a model’s
performance. It helps in estimating two important aspects of machine learning: generalization
error and sample complexity.
Generalization Error and VC Dimension
The generalization error measures how well a model performs on unseen data. The VC
dimension helps to bound this error using the following principle:
 A lower VC dimension indicates a simpler model, reducing the risk of overfitting but
increasing the risk of underfitting.
 A higher VC dimension allows the model to fit complex data patterns but may lead to
overfitting if not managed correctly.
VC Dimension and Error Bound Formula:
For a hypothesis class $H$ with VC dimension $d$, and a dataset of size $N$, the
generalization error can be bounded as:
This formula indicates that the larger the dataset $N$, the smaller the error, even for models
with higher VC dimensions.
Calculating VC Dimension
Calculating the VC dimension helps quantify the complexity of different hypothesis classes.
The process involves understanding how many data points a model can perfectly classify (or
“shatter”). Let’s explore how to calculate VC dimension step by step.
Step-by-Step Method to Calculate VC Dimension
1. Identify the Hypothesis Class
 The first step is to define the set of functions (or models) under consideration, such
as lines, circles, or decision trees.
2. Test for Shattering
 Determine the maximum number of points that can be classified in every possible
way using the hypothesis class.
 If the hypothesis can separate all possible label combinations of n points but fails
for n+1, then the VC dimension is n.
3. Formal Verification
 Ensure the hypothesis class satisfies the conditions for shattering up to $n$ points,
using mathematical or visual proofs.
Q2: Probably Approximately Correct (PAC) Learning:

What is PAC Learning?

 PAC learning is a theoretical framework that addresses the question of how much data is
necessary for a learning algorithm to perform well on new, unseen data. The core idea is that a
learning algorithm can be considered PAC if, given a sufficient number of training samples, it
can produce a hypothesis that is likely (with high probability) to be approximately correct
(within a specified error margin).

Key Components of PAC Learning

1. Hypothesis Space: This is the set of all possible hypotheses that a learning algorithm can
choose from. The complexity of the hypothesis space significantly impacts the sample
complexity required for learning.
2. Sample Complexity: This refers to the number of training examples needed to ensure that
the learned hypothesis will generalize well to new data. In PAC learning, it is crucial to
determine how many samples are required to achieve a desired level of accuracy and
confidence.
3. Generalization: This is the ability of a learning algorithm to perform well on unseen data. In
the PAC framework, generalization is quantified by the probability that the chosen hypothesis
will have an error rate within an acceptable range on new samples.
4. Error Rate: The error rate is defined as the probability that the hypothesis will misclassify
an example drawn from the underlying distribution. PAC learning aims to minimize this error
rate while ensuring that the hypothesis is consistent with the training data.

The PAC Learning Theorem

The PAC learning theorem provides formal guarantees about the performance of
learning algorithms. It states that for a given accuracy (ε) and confidence (δ), there exists a
sample size (m) such that any learning algorithm that returns a hypothesis consistent with the
training samples will, with probability at least 1−δ1−δ, have an error rate less than ε on unseen
data. Mathematically, this can be expressed as:

m≥1ϵ(ln ∣H∣+ln 1δ)

Where:
 mm is the sample size,
 ϵϵ is the maximum acceptable error,
 ∣H∣∣H∣ is the size of the hypothesis space,
 δδ is the acceptable failure probability.

Importance of PAC Learning

Understanding PAC learning is essential for several reasons:

 Theoretical Foundation: It provides a rigorous foundation for analyzing the behavior and
performance of learning algorithms, helping researchers and practitioners design better
models.
 Generalization Guarantees: PAC learning offers theoretical guarantees regarding the
generalization ability of algorithms, which is crucial for assessing their reliability.
 Guidance for Sample Size: By quantifying the sample complexity, PAC learning helps
determine how much data is necessary for effective learning, which is particularly important
in real-world applications.

Challenges in PAC Learning

Despite its advantages, PAC learning faces several challenges:

 Computational Complexity: Finding the optimal hypothesis can be computationally
expensive, especially as the hypothesis space grows.
 Model Assumptions: PAC learning relies on certain assumptions about the underlying
distribution of the data, which may not always hold in practice.
 Overfitting: As the complexity of the hypothesis space increases, there is a risk of overfitting,
where the model performs well on training data but poorly on unseen data.

Practical Example of PAC Learning

To illustrate PAC learning, let’s consider a simple example using Python. We will implement
a basic PAC learning scenario using a linear classifier.
importnumpyas np
[Link]
from sklearn.linear_modelimportLogisticRegression
fromsklearn.datasetsimportmake_classification
fromsklearn.model_selectionimporttrain_test_split

# Generate a synthetic dataset

X,y =
make_classification(n_samples=100,n_features=2,n_informative=2,n_redundant=0,random_s
tate=42)

# Split the dataset into training and testing sets

X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.3,random_state=42)

# Create and fit the logistic regression model

model = LogisticRegression()
[Link](X_train,y_train)

# Calculate the accuracy on the test set

accuracy = [Link](X_test,y_test)
print(f"Model Accuracy: {accuracy:.2f}")

# Visualize the decision boundary

xx,yy = [Link]([Link](X[:,0].min()-1,X[:,0].max()+1,100),
[Link](X[:,1].min()-1,X[:,1].max()+1,100))
Z = [Link](np.c_[[Link](),[Link]()])
Z = [Link]([Link])
 [Link](xx,yy,Z,alpha=0.8)
[Link](X[:,0],X[:,1],c=y,edgecolors='k',marker='o')
[Link]('PAC Learning Example with Logistic Regression')
[Link]('Feature 1')
[Link]('Feature 2')
[Link]()
In this example, we generate a synthetic dataset and train a logistic regression model.
The accuracy of the model on the test set provides an empirical measure of its generalization
performance, illustrating the principles of PAC learning.
Applications of PAC Learning
PAC learning has broad applications across various domains:
 Classification: It serves as a foundation for designing classifiers that can generalize well
from limited training data.
 Active Learning: PAC learning principles guide the selection of the most informative
samples to label, minimizing the sample complexity.
 Reinforcement Learning: The framework helps in understanding the trade-offs between
exploration and exploitation in learning environments.
 UNIT – 2
Supervised Learning

INTRODUCTION:
Supervised learning is a category of machine learning that uses labeled datasets to train
algorithms to predict outcomes and recognize patterns. Unlike unsupervised learning,
supervised learning algorithms are given labeled training to learn the relationship between the
input and the outputs.

Supervised machine learning algorithms make it easier for organizations to create

complex models that can make accurate predictions. As a result, they are widely used across
various industries and fields, including healthcare, marketing, financial services, and more.

Supervised machine learning is a fundamental approach for machine learning and

artificial intelligence. It involves training a model using labeled data, where each input comes
with a corresponding correct output. The process is like a teacher guiding a student—hence
the term “supervised” learning.

Types of Supervised Learning in Machine Learning

Now, Supervised learning can be applied to two main types of problems:
 Classification: Where the output is a categorical variable (e.g., spam vs. non-spam
emails, yes vs. no).
 Regression: Where the output is a continuous variable (e.g., predicting house prices,
stock prices).
Q: Linear Regression Models:

The term regression is used when you try to find the relationship between [Link]
Machine Learning, and in statistical modeling, that relationship is used to predict the outcome
of future events.

Linear Regression
Linear regression uses the relationship between the data-points to draw a straight line through
all [Link] line can be used to predict future values.

In Machine Learning, predicting the future is very important.

How Does it Work?

Python has methods for finding a relationship between data-points and to draw a line of
linear regression. We will show you how to use these methods instead of going through the
mathematic formula.

In the example below, the x-axis represents age, and the y-axis represents speed. We
have registered the age and speed of 13 cars as they were passing a tollbooth. Let us see if the
data we collected could be used in a linear regression:

Example:
Start by drawing a scatter plot:

import [Link] as plt

x = [5,7,8,7,2,17,2,9,4,11,12,9,6]
y = [99,86,87,88,111,86,103,87,94,78,77,85,86]
[Link](x, y)
[Link]()
Result:
Conclusion:
Linear regression is a statistical method used to model the relationship between a
dependent variable and one or more independent variables. It provides valuable insights for
prediction and data analysis. This article will explore its types, assumptions, implementation,
advantages, and evaluation metrics.
Understanding Linear Regression
Linear regression is also a type of supervised machine-learning algorithm that
learns from the labelled datasets and maps the data points with most optimized linear
functions which can be used for prediction on new datasets. It computes the linear
relationship between the dependent variable and one or more independent features by fitting
a linear equation with observed data. It predicts the continuous output variables based on the
independent input variable.
For example if we want to predict house price we consider various factor such as house
age, distance from the main road, location, area and number of room, linear regression uses
all these parameter to predict house price as it consider a linear relation between all these
features and price of house.

Q: Single Variables and Multiple Variables (or) Simple Linear Regression and Multiple
Linear Regression:
Introduction to Linear Regression:
Linear regression in machine learning is defined as a statistical model that analyzes the
linear relationship between a dependent variable and a given set of independent variables.
The linear relationship between variables means that when the value of one or more
independent variables will change (increase or decrease), the value of the dependent variable
will also change accordingly (increase or decrease).
In machine learning, linear regression is used for predicting continuous numeric values
based on learned linear relation for new and unseen data. It is used in predictive modeling,
financial forecasting, risk assessment, etc.
In Other Words, Linear regression is a statistical technique that estimates the linear
relationship between a dependent and one or more independent variables. In machine
learning, linear regression is implemented as a supervised learning approach. In machine
learning, labeled datasets contain input data (features) and output labels (target values). For
linear regression in machine learning, we represent features as independent variables and
target values as the dependent variable.
Line of Regression
A straight line that shows a relation between the dependent variable and independent
variables is known as the line of regression or regression line.

Types of Linear Regression

Linear regression is of the following two types −
 Simple Linear Regression

 Multiple Linear Regression

1. Simple Linear Regression (or) Single Variable:
Simple linear regression is a type of regression analysis in which a single independent
variable (also known as a predictor variable) is used to predict the dependent variable. In
other words, it models the linear relationship between the dependent variable and a single
independent variable.

In the above image, the straight line represents the simple linear regression line where
Ŷ is the predicted value, and X is the input [Link], the relationship can be
modeled as a linear equation −
Y=w0+w1X+ϵY=w0+w1X+ϵ
Where
 Y is the dependent variable (target).
 X is the independent variable (feature).
 w0 is the y-intercept of the line.
 w1 is the slope of the line, representing the effect of X on Y.
 ε is the error term, capturing the variability in Y not explained by X.

2. Multiple Linear Regression

Multiple linear regression is basically the extension of simple linear regression that predicts
a response using two or more [Link] dealing with more than one independent
variable, we extend simple linear regression to multiple linear regression. The model is
expressed as:

Multiple linear regression extends the concept of simple linear regression to multiple
independent variables. The model is expressed as:

Y=w0+w1X1+w2X2+⋯+wpXp+ϵY=w0+w1X1+w2X2+⋯+wpXp+ϵ

Where

 X1, X2, ..., Xp are the independent variables (features).

 w0, w1, ..., wp are the coefficients for these variables.
 ε is the error term.
Q: Bayesian Linear Regression:

Bayesian regression is a type of linear regression that uses Bayesian statistics to

estimate the unknown parameters of a model. It uses Bayes’ theorem to estimate the
likelihood of a set of parameters given observed data. The goal of Bayesian regression is to
find the best estimate of the parameters of a linear model that describes the relationship
between the independent and the dependent variables.
Bayesian Regression can be very useful when we have insufficient data in the dataset or the
data is poorly distributed. The output of a Bayesian Regression model is obtained from a
probability distribution, as compared to regular regression techniques where the output is just
obtained from a single value of each attribute.
Some Dependent Concepts for Bayesian Regression
The important concepts in Bayesian Regression are as follows:
Bayes Theorem
Bayes Theorem gives the relationship between an event’s prior probability and its posterior
probability after evidence is taken into account. It states that the conditional probability of an
event is equal to the probability of the event given certain conditions multiplied by the prior
probability of the event, divided by the probability of the conditions.
i.e P(A∣B)=P(B∣A)⋅P(A)P(B) P(A∣B)=P(B)P(B∣A)⋅P(A) .
Where P(A|B) is the probability of event A occurring given that event B has already occurred,
P(B|A) is the probability of event B occurring given that event A has already occurred, P(A)
is the probability of event A occurring and P(B) is the probability of event B occurring.
Maximum Likelihood Estimation (MLE)
MLE is a method used to estimate the parameters of a statistical model by maximizing the
likelihood function. it seeks to find the parameter values that make the observed data most
probable under the assumed model. MLE does not incorporate any prior information or
assumptions about the parameters, and it provides point estimates of the parameters
Maximum A Posteriori (MAP) Estimation
MAP estimation is a Bayesian approach that combines prior information with the likelihood
function to estimate the parameters. It involves finding the parameter values that maximize
the posterior distribution, which is obtained by applying Bayes’ theorem. In MAP estimation,
a prior distribution is specified for the parameters, representing prior beliefs or knowledge
about their values. The likelihood function is then multiplied by the prior distribution to
obtain the joint distribution, and the parameter values that maximize this joint distribution are
selected as the MAP estimates. MAP estimation provides point estimates of the parameters,
similar to MLE, but incorporates prior information.
Need for Bayesian Regression
There are several reasons why Bayesian regression is useful over other regression techniques.
Some of them are as follows:
1. Bayesian regression also uses the prior belief about the parameters in the analysis. which
makes it useful when there is limited data available and the prior knowledge are
relevant.
3. Bayesian regression provides a natural way to measure the uncertainty in the estimation
of regression parameters by generating the posterior distribution, which captures the
uncertainty in the parameter values, as opposed to the single point estimate that is
produced by standard regression techniques.
4. In order to incorporate complicated correlations and non-linearities, Bayesian regression
provides flexibility by offering a framework for integrating various prior distributions,
which makes it capable to handle situations where the basic assumptions of standard
regression techniques, like linearity or homoscedasticity, may not be true. It enables the
modeling of more realistic and nuanced relationships between the predictors and the
response variable.
4. Bayesian regression facilitates model selection and comparison by calculating the
posterior probabilities of different models.
Implementation of Bayesian Regression
Let’s independent features for linear regression is X={x1,x2,…,xP} X={x1,x2,…,xP
} , where xᵢ represents the ith independent features and target variables will be Y. Assume
we have n samples of (X, y).The linear relationship between the dependent variable Y and
the independent features X can be represented as:
y=w₀+w₁x₁+w₂x₂+…+wₚxₚ+ϵ y=w₀+w₁x₁+w₂x₂+…+wₚxₚ+ϵ
or
y=f(x,w)+ϵy=f(x,w)+ϵ
Here, w={w₀,w₁,w₂,…,wₚ} w={w₀,w₁,w₂,…,wₚ} are the regression coefficients, representing
the relationship between the independent variables and the dependent variable, and ε is the
error term.
We assume that the errors (ε) follow a normal distribution with mean 0 and constant
variance σ2 σ2 i.e. (ϵ∼N(0,σ2)) (ϵ∼N(0,σ2)) . This assumption allows us to model the
distribution of the target variable around the predicted values.

Implementation of Bayesian Regression Using Python:

Method 1:
import numpy as np
import pymc3 as pm
import [Link] as plt

# Generate some sample data

[Link](0)
X = [Link](0, 10, 100)
true_slope = 2
true_intercept = 1
Y = true_intercept + true_slope * X + [Link](0, 1, size=100)

# Create a PyMC3 model

with [Link]() as model:
# Priors for the parameters
slope = [Link]('slope', mu=0, sd=10)
intercept = [Link]('intercept', mu=0, sd=10)
sigma = [Link]('sigma', sd=1)

# Expected value of the outcome

mu = intercept + slope * X

# Likelihood (sampling distribution) of the observations

Y_obs = [Link]('Y_obs', mu=mu, sd=sigma, observed=Y)

# Run the MCMC sampling

trace = [Link](2000, tune=1000)

# Plot the posterior distributions

pm.plot_posterior(trace, var_names=['slope', 'intercept', 'sigma'])
[Link]()
Output:

100.00% [6000/6000 00:02<00:00 Sampling 2 chains, 0 divergences]

Q: Gradient Descent:
Gradient descent allows us to iteratively change these parameters by moving slightly
in the direction of the steepest fall, minimizing the error over time, and bringing about
convergence. Gradient descent is an essential technique in machine learning since it allows
us to efficiently optimize the performance of linear regression models. In this post, we will
look closely at Gradient Descent in Linear Regression.
Understanding Linear Regression
An important statistical method for simulating the connection between a dependent variable
and one or more independent variables is linear regression. Finding the appropriate line to
depict the connection between the variables entails fitting a linear equation to a pre−existing
dataset. A simple linear regression equation is stated as:

Where

 The dependent variable is y.

 The independent variable is x.
 y−intercept (the value of y when x is 0) is β0β0
 Slope (the change in y for a one−unit increase in x) is β1β1
 The random error is represented by εε

The goal of linear regression is to minimize the difference between the predicted values ()
and the actual values (y), often known as cost, loss, or error. The mean squared error (MSE),
which is outlined as follows, is the objective function that is most frequently utilized.

Where,

 The total number of observations is n.

  y is the dependent variable's actual value.
 ŷ is the dependent variable's anticipated value.
Understanding Gradient Descent

Finding the best−fit line in the context of linear regression that minimizes the gap
between the predicted and actual values depends critically on gradient descent. The mean
squared error (MSE) is frequently used as the cost function in linear regression. Gradient
descent allows the model to modify its predictions, lowering the overall error and increasing
the precision of the regression line. This is done by iteratively updating the parameters (slope
and intercept) using the gradient of the MSE.

Implementing Gradient Descent in Linear Regression

An example of functional Python code that uses gradient descent to do linear regression is
shown below:

Open Compiler
import numpy as np
import [Link] as plt

# Generate sample data for demonstration

[Link](42)
X = 2 * [Link](100, 1)
y = 4 + 3 * X + [Link](100, 1)

# Add bias term to X

X_b = np.c_[[Link]((100, 1)), X]

# Set hyperparameters
learning_rate = 0.01
num_iterations = 1000

# Initialize parameters randomly

theta = [Link](2, 1)

# Perform gradient descent

for iteration in range(num_iterations):
gradients = 2 / 100 * X_b.[Link](X_b.dot(theta) - y)
theta = theta - learning_rate * gradients

# Print the final parameter values

print("Intercept:", theta[0][0])
print("Slope:", theta[1][0])
# Plot the data points and the regression line
[Link](X, y)
[Link](X, X_b.dot(theta), color='red')
[Link]('X')
[Link]('y')
[Link]('Linear Regression with Gradient Descent')
[Link]()

Output

Intercept: 4.158093763822134
Slope: 2.8204434017416244

Q: Linear Classification Models

Discriminant Function: A discriminant function in machine learning is a function that
classifies data into groups or categories. It's used in pattern recognition, image retrieval, and
other applications.
Applications:
Face recognition: It's used to reduce the high-dimensional feature space of pixel values in
face recognition applications.
 Medical diagnosis: It classifies disease severity based on patient parameters.
 Customer identification: It can help identify customer segments most likely to purchase a
specific product.
Other uses
 Biometric identification systems, such as fingerprint recognition and iris recognition
 Quality control and manufacturing
 Document classification
 Marketing and customer segmentation
 Remote sensing and image analysis
 (Or)
In machine learning, a discriminant function is used to classify data points by finding a
decision boundary that separates different classes, and in Python, this is often implemented
using techniques like Linear Discriminant Analysis (LDA) or Quadratic Discriminant Analysis
(QDA).
What is a Discriminant Function?
 A discriminant function is a function that takes a data point (feature vector) as input and
outputs a value that helps determine which class the data point belongs to.
 It essentially creates a decision boundary in the feature space, separating different
classes.
 The goal is to find a function that maximizes the separation between classes while
minimizing the variance within each class.
Linear Discriminant Analysis (LDA)
 LDA is a supervised learning technique that aims to find a linear combination of features
that best separates different classes.
 It's often used for dimensionality reduction and as a pre-processing step for classification
tasks.
 LDA assumes that the data within each class follows a Gaussian distribution with the
same covariance matrix.
 How it works:
o LDA finds a projection of the data onto a lower-dimensional space that maximizes the
between-class variance and minimizes the within-class variance.
o This projection is achieved by finding a linear combination of the original features.
o The resulting discriminant function is a linear equation that can be used to classify new
data points.
 Implementation in Python: Scikit-learn provides a LinearDiscriminantAnalysis class
for implementing LDA.
Quadratic Discriminant Analysis (QDA):
 QDA is similar to LDA but relaxes the assumption that all classes share the same
covariance matrix.
 Each class is allowed to have its own covariance matrix.
 How it works:
o QDA finds a quadratic decision boundary that separates the classes.
o This allows for more complex decision boundaries than LDA.
 Implementation in Python: Scikit-learn provides
a QuadraticDiscriminantAnalysis class for implementing QDA.
Q: Perception Algorithm:
 The Perceptron is one of the simplest artificial neural network architectures,
introduced by Frank Rosenblatt in 1957. It is primarily used for binary classification.
 At that time, traditional methods like Statistical Machine Learning and Conventional
Programming were commonly used for predictions. Despite being one of the simplest
forms of artificial neural networks, the Perceptron model proved to be highly effective
in solving specific classification problems, laying the groundwork for advancements
in AI and machine learning.
Types of Perceptron:
1. Single-Layer Perceptron is a type of perceptron is limited to learning linearly separable
patterns. It is effective for tasks where the data can be divided into distinct categories
through a straight line. While powerful in its simplicity, it struggles with more complex
problems where the relationship between inputs and outputs is non-linear.
2. Multi-Layer Perceptron possess enhanced processing capabilities as they consist of two
or more layers, adept at handling more complex patterns and relationships within the data.
How does Perceptron work?
A weight is assigned to each input node of a perceptron, indicating the importance of that
input in determining the output. The Perceptron’s output is calculated as a weighted sum of
the inputs, which is then passed through an activation function to decide whether the
Perceptron will fire.
The weighted sum is computed as:
z=w1x1+w2x2+…+wnxn=XTWz=w1x1+w2x2+…+wnxn=XTW
The step function compares this weighted sum to a threshold. If the input is larger than the
threshold value, the output is 1; otherwise, it’s 0. This is the most common activation function
used in Perceptrons are represented by the Heaviside step function:
h(z)={0if z<Threshold1if z≥Thresholdh(z)={01if z<Thresholdif z≥Threshold
A perceptron consists of a single layer of Threshold Logic Units (TLU), with each TLU fully
connected to all input nodes.

In a fully connected layer, also known as a dense layer, all neurons in one layer are connected
to every neuron in the previous layer.
The output of the fully connected layer is computed as:
fW,b(X)=h(XW+b)fW,b(X)=h(XW+b)
where XX is the input WW is the weight for each inputs neurons and bb is the bias and hh is
the step function.
During training, the Perceptron’s weights are adjusted to minimize the difference between
the predicted output and the actual output. This is achieved using supervised learning
algorithms like the delta rule or the Perceptron learning rule.
The weight update formula is:
wi,j=wi,j+η(yj−y^j)xiwi,j=wi,j+η(yj−y^j)xi
Where:
 wi,jwi,j is the weight between the ithith input and jthjth output neuron,
 xixi is the ithith input value,
 yjyj is the actual value, and y^jy^j is the predicted value,
 ηη is the learning rate, controlling how much the weights are adjusted.
This process enables the perceptron to learn from data and improve its prediction accuracy
over time.
Q: Probabilistic Discriminative Model:
Probabilistic models are an essential component of machine learning, which aims to learn
patterns from data and make predictions on new, unseen data. They are statistical models that
capture the inherent uncertainty in data and incorporate it into their predictions. Probabilistic
models are used in various applications such as image and speech recognition, natural
language processing, and recommendation systems. In recent years, significant progress has
been made in developing probabilistic models that can handle large datasets efficiently.
Categories Of Probabilistic Models
These models can be classified into the following categories:
 Generative models
 Discriminative models.
 Graphical models
Generative models:
Generative models aim to model the joint distribution of the input and output variables. These
models generate new data based on the probability distribution of the original dataset.
Generative models are powerful because they can generate new data that resembles the
training data. They can be used for tasks such as image and speech synthesis, language
translation, and text generation.
Discriminative models
The discriminative model aims to model the conditional distribution of the output variable
given the input variable. They learn a decision boundary that separates the different classes
of the output variable. Discriminative models are useful when the focus is on making accurate
predictions rather than generating new data. They can be used for tasks such as image
recognition, speech recognition, and sentiment analysis.
Graphical models
These models use graphical representations to show the conditional dependence between
variables. They are commonly used for tasks such as image recognition, natural language
processing, and causal inference.
Q: Naive Bayes Algorithm in Probabilistic Models
The Naive Bayes algorithm is a widely used approach in probabilistic models, demonstrating
remarkable efficiency and effectiveness in solving classification problems. By leveraging the
power of the Bayes theorem and making simplifying assumptions about feature
independence, the algorithm calculates the probability of the target class given the feature
set. This method has found diverse applications across various industries, ranging from spam
filtering to medical diagnosis. Despite its simplicity, the Naive Bayes algorithm has proven
to be highly robust, providing rapid results in a multitude of real-world problems.
Naive Bayes is a probabilistic algorithm that is used for classification problems. It is based
on the Bayes theorem of probability and assumes that the features are conditionally
independent of each other given the class. The Naive Bayes Algorithm is used to calculate
the probability of a given sample belonging to a particular class. This is done by calculating
the posterior probability of each class given the sample and then selecting the class with the
highest posterior probability as the predicted class.
The algorithm works as follows:
1. Collect a labeled dataset of samples, where each sample has a set of features and a class
label.
2. For each feature in the dataset, calculate the conditional probability of the feature given
the class.
3. This is done by counting the number of times the feature occurs in samples of the class
and dividing by the total number of samples in the class.
4. Calculate the prior probability of each class by counting the number of samples in each
class and dividing by the total number of samples in the dataset.
5. Given a new sample with a set of features, calculate the posterior probability of each
class using the Bayes theorem and the conditional probabilities and prior probabilities
calculated in steps 2 and 3.
6. Select the class with the highest posterior probability as the predicted class for the new
sample.

Advantages Of Probabilistic Models

 Probabilistic models are an increasingly popular method in many fields, including
artificial intelligence, finance, and healthcare.
 The main advantage of these models is their ability to take into account uncertainty and
variability in data. This allows for more accurate predictions and decision-making,
particularly in complex and unpredictable situations.
Disadvantages Of Probabilistic Models
There are also some disadvantages to using probabilistic models.
 One of the disadvantages is the potential for overfitting, where the model is too specific
to the training data and doesn’t perform well on new data.
 Not all data fits well into a probabilistic framework, which can limit the usefulness of
these models in certain applications.
Q: What is Logistic Regression?
Logistic regression is a supervised machine learning algorithm used for classification
tasks where the goal is to predict the probability that an instance belongs to a given class or
not. Logistic regression is a statistical algorithm which analyze the relationship between two
data factors. The article explores the fundamentals of logistic regression, it’s types and
implementations.
Logistic regression is used for binary classification where we use sigmoid function, that takes
input as independent variables and produces a probability value between 0 and 1.
For example, we have two classes Class 0 and Class 1 if the value of the logistic function for
an input is greater than 0.5 (threshold value) then it belongs to Class 1 otherwise it belongs to
Class 0. It’s referred to as regression because it is the extension of linear regression but is
mainly used for classification problems.
Key Points:
 Logistic regression predicts the output of a categorical dependent variable. Therefore, the
outcome must be a categorical or discrete value.
 It can be either Yes or No, 0 or 1, true or False, etc. but instead of giving the exact value
as 0 and 1, it gives the probabilistic values which lie between 0 and 1.
 In Logistic regression, instead of fitting a regression line, we fit an “S” shaped logistic
function, which predicts two maximum values (0 or 1).
Types of Logistic Regression
On the basis of the categories, Logistic Regression can be classified into three types:
1. Binomial: In binomial Logistic regression, there can be only two possible types of the
dependent variables, such as 0 or 1, Pass or Fail, etc.
2. Multinomial: In multinomial Logistic regression, there can be 3 or more possible
unordered types of the dependent variable, such as “cat”, “dogs”, or “sheep”
3. Ordinal: In ordinal Logistic regression, there can be 3 or more possible ordered types of
dependent variables, such as “low”, “Medium”, or “High”.
Assumptions of Logistic Regression
We will explore the assumptions of logistic regression as understanding these assumptions is
important to ensure that we are using appropriate application of the model. The assumption
include:
1. Independent observations: Each observation is independent of the other. meaning there is
no correlation between any input variables.
2. Binary dependent variables: It takes the assumption that the dependent variable must be
binary or dichotomous, meaning it can take only two values. For more than two categories
SoftMax functions are used.
3. Linearity relationship between independent variables and log odds: The relationship
between the independent variables and the log odds of the dependent variable should be
linear.
4. No outliers: There should be no outliers in the dataset.
5. Large sample size: The sample size is sufficiently large
How does Logistic Regression work?
The logistic regression model transforms the linear regression function continuous value
output into categorical value output using a sigmoid function, which maps any real-valued set
of independent variables input into a value between 0 and 1. This function is known as the
logistic function.
Let the independent input features be:
X=[x11 …x1mx21 …x2m ⋮⋱ ⋮ xn1 …xnm]X=x11 x21 ⋮xn1 ……⋱ …x1mx2m⋮ xnm
and the dependent variable is Y having only binary value i.e. 0 or 1.
Y={0 if Class11 if Class2Y={01 if Class1 if Class2
then, apply the multi-linear function to the input variables X.
z=(∑i=1nwixi)+bz=(∑i=1nwixi)+b
Here xixi is the ith observation of X, wi=[w1,w2,w3,⋯,wm]wi=[w1,w2,w3,⋯,wm] is the
weights or Coefficient, and b is the bias term also known as intercept. simply this can be
represented as the dot product of weight and bias.
z=w⋅X+bz=w⋅X+b
whatever we discussed above is the linear regression.
Sigmoid Function
Now we use the sigmoid function where the input will be z and we find the probability
between 0 and 1. i.e. predicted y.
σ(z)=11+e−zσ(z)=1+e−z1

Sigmoid function

As shown above, the figure sigmoid function converts the continuous variable data into
the probability i.e. between 0 and 1.
 σ(z) σ(z) tends towards 1 as z→∞z→∞
 σ(z) σ(z) tends towards 0 as z→−∞z→−∞
 σ(z) σ(z) is always bounded between 0 and 1
where the probability of being a class can be measured as:
P(y=1)=σ(z)P(y=0)=1−σ(z)P(y=1)=σ(z)P(y=0)=1−σ(z)
Probalistic Generative Model:
• Generative models are a class of statistical models that generate new data instances. These
models are used in unsupervised machine learning to perform tasks such as probability and
likelihood estimation, modelling data points, and distinguishing between classes using these
probabilities.
• Generative models rely on the Bayes theorem to find the joint probability. Generative models
describe how data is generated using probabilistic models. They predict P(y | x), the probability
of y given x, calculating the P(x,y), the probability of x and y.
Q: Explain about Naive Bayes:
• Naive Bayes classifiers are a family of simple probabilistic classifiers based on applying
Bayes' theorem with strong independence assumptions between the features. It is highly
scalable, requiring a number of parameters linear in the number of variables
(features/predictors) in a learning problem.
• A Naive Bayes Classifier is a program which predicts a class value given a set of attributes.
• For each known class value,
1. Calculate probabilities for each attribute, conditional on the class value.
2. Use the product rule to obtain a joint conditional probability for the attributes.
3. Use Bayes rule to derive conditional probabilities for the class variable.
• Once this has been done for all class values, output the class with the highest probability.
• Naive bayes simplifies the calculation of probabilities by assuming that the probability of
each attribute belonging to a given class value is independent of all other attributes. This is a
strong assumption but results in a fast and effective method.
• The probability of a class value given a value of an attribute is called the conditional
probability. By multiplying the conditional probabilities together for each attribute for a given
class value, we have a probability of a data instance belonging to that class.
Conditional Probability
• Let A and B be two events such that P(A) > 0. We denote P(BIA) the probability of B given
that A has occurred. Since A is known to have occurred, it becomes the new sample space
replacing the original S. From this, the definition is,
P(B/A) = P(A∩B)/P(A)
OR
P(A ∩ B) = P(A) P(B/A)
• The notation P(B | A) is read "the probability of event B given event A". It is the probability
of an event B given the occurrence of the event A.
• We say that, the probability that both A and B occur is equal to the probability that A occurs
times the probability that B occurs given that A has occurred. We call P(B | A) the conditional
probability of B given A, i.e., the probability that B will occur given that A has occurred.
• Similarly, the conditional probability of an event A, given B by,
P(A/B) = P(A∩B)/P(B)
• Another way to look at the conditional probability formula is :
P(Second/First) = P(First choice and second choice)/P(First choice)
• Conditional probability is a defined quantity and cannot be proven.
• The key to solving conditional probability problems is to:
1. Define the events.
2. Express the given information and question in probability notation.
3. Apply the formula.
Joint Probability
• A joint probability is a probability that measures the likelihood that two or more events will
happen concurrently.
• If there are two independent events A and B, the probability that A and B will occur is found
by multiplying the two probabilities. Thus for two events A and B, the special rule of
multiplication shown symbolically is :
P(A and B) = P(A) P(B).
• The general rule of multiplication is used to find the joint probability that two events will
occur. Symbolically, the general rule of multiplication is,
P(A and B) = P(A) P(B | A).
• The probability P(A ∩ B) is called the joint probability for two events A and B which intersect
in the sample space. Venn diagram will readily shows that
P(A ∩ B) = P(A) + P(B) - P (AUB)
Decision Tree:Decision Tree is a supervised machine learning algorithm where all the
decisions were made based on some conditions. The decision tree has a root node and leaf
nodes extended from the root node. These nodes were decided based on some parameters
like Gini index, entropy, information gain.
A Decision tree is a tree-like structure that represents a set of decisions and their possible
consequences. Each node in the tree represents a decision, and each branch represents an
outcome of that decision. The leaves of the tree represent the final decisions or predictions.
Decision trees are created by recursively partitioning the data into smaller and smaller
subsets. At each partition, the data is split based on a specific feature, and the split is made
in a way that maximizes the information gain.

Decision Tree

In the above figure, decision tree is a flowchart-like tree structure that is used to make
decisions. It consists of Root Node(WINDY), Internal nodes(OUTLOOK,
TEMPERATURE), which represent tests on attributes, and leaf nodes, which represent the
final decisions. The branches of the tree represent the possible outcomes of the tests.

Working with Dataset:

Before creating and training our model, first, we have to preprocess our [Link] us start
by Importing some important basic libraries.
import pandas as pd
import numpy as np
import seaborn as sns
import [Link] as plt
import warnings
[Link]("ignore")
To just ignore warnings that we come across during model creation, just import
warnings and set it to ignore.
Q: Explain about Support Vector Machine (SVM) Algorithm?
Support Vector Machine (SVM) is a supervised machine learning algorithm used
for classification and regression tasks. It tries to find the best boundary known as
hyperplane that separates different classes in the data. It is useful when you want to do
binary classification like spam vs. not spam or cat vs. dog. (OR)
Support vector machines (SVMs) are powerful yet flexible supervised machine
learning algorithm which is used for both classification and regression. But generally, they
are used in classification problems. In 1960s, SVMs were first introduced but later they got
refined in 1990 also. SVMs have their unique way of implementation as compared to other
machine learning algorithms. Now a days, they are extremely popular because of their
ability to handle multiple continuous and categorical variables.
Working of SVM:
The goal of SVM is to find a hyperplane that separates the data points into different classes.
A hyperplane is a line in 2D space, a plane in 3D space, or a higher-dimensional surface in
n-dimensional space. The hyperplane is chosen in such a way that it maximizes the margin,
which is the distance between the hyperplane and the closest data points of each class. The
closest data points are called the support vectors.
The distance between the hyperplane and a data point "x" can be calculated using the
formula −
distance = (w . x + b) / ||w||
where "w" is the weight vector, "b" is the bias term, and "||w||" is the Euclidean norm of the
weight vector. The weight vector "w" is perpendicular to the hyperplane and determines its
orientation, while the bias term "b" determines its position.
Let's understand it in detail with the help of following diagram –
 Given below are the important concepts in SVM −
 Support Vectors − Datapoints that are closest to the hyperplane is called support
vectors. Separating line will be defined with the help of these data points.
 Hyperplane − As we can see in the above diagram it is a decision plane or space
which is divided between a set of objects having different classes.
 Margin − It may be defined as the gap between two lines on the closet data points of
different classes. It can be calculated as the perpendicular distance from the line to
the support vectors. Large margin is considered as a good margin and small margin is
considered as a bad margin.

Implementing SVM Using Python:

For implementing SVM in Python we will start with the standard libraries import as follows
import numpy as np
import [Link] as plt
from scipy import stats
import seaborn as sns; [Link]()
Advantages of SVMs:
SVMs are powerful machine learning algorithms that have the following advantages:
 Effective in high-dimensional spaces. High-dimensional data refers to data in which the
number of features is larger than the number of observations, i.e., data points. SVMs
perform well even when the number of features is larger than the number of samples. They
can handle high-dimensional data efficiently, making them suitable for applications with a
large number of features.
 Resistant to overfitting. SVMs are less prone to overfitting compared to other algorithms,
such as decision trees -- overfitting is where a model performs extremely well on the
training data but becomes too specific to that data and can't generalize to new data. SVMs'
use of the margin maximization principle helps in generalizing well to unseen data.
 Versatile. SVMs can be applied to both classification and regression problems. They
support different kernel functions, enabling flexibility in capturing complex relationships
in the data. This versatility makes SVMs applicable to a wide range of tasks.
 Effective in cases of limited data. SVMs can work well even when the training data set is
small. The use of support vectors ensures that only a subset of data points influences the
decision boundary, which can be beneficial when data is limited.
 Able to handle nonlinear data. SVMs can implicitly handle non-linearly separable data
by using kernel functions. The kernel trick enables SVMs to transform the input space into
a higher-dimensional feature space, making it possible to find linear decision boundaries.
 Memory-efficient. SVMs are memory-efficient as they rely on a subset of training points,
known as support vectors, in their decision function. This helps reduce the computational
load, especially when dealing with large data sets.
 Less sensitive to noise. Compared to other classifiers, SVMs are less sensitive to outliers.
Their focus on support vectors minimizes the influence of noisy data points, leading to
more stable models.
Q: Describe Decision Tree with an example?