a) What is Machine learning ? when we use machine learning?

Write about
some applications of machine learning.
Ans: Machine from Learning is the field of study that enables a system
experience without being explicitly programmed.
a. Problems for which existing solutions require a lot of hand-tuning or
long lists of rules: a Machine Learning algorithm can often simplify
code and perform better.
b. Complex problems for which there is no good solution at all using a
traditional approach: the best Machine Learning techniques can find a
solution.
Fluctuating environments: a Machine Learning system can adapt to new data.
• Getting insights about complex problems and large amounts of
data
Applications of Machine Learning
1. Spam Detection – Identifying spam emails
2. Recommendation Systems – Netflix, YouTube, Amazon
3. Image & Face Recognition – Security systems
4. Speech Recognition – Google Assistant, Siri
5. Medical Diagnosis – Disease prediction
6. Fraud Detection – Banking and credit card systems
7. Autonomous Vehicles – Self-driving cars

Variable (attribute) transformation is the process of applying a mathematical

function to an attribute to convert its original values into new values, often to
reduce range, normalize distribution, or improve data analysis.

b) Distinguish between rule-based programming and machine learning approach.

 c) c what is the goal of machine learning? show the workflow of machine learning
?
d) classify the different type of machine learning?

Exploratory Data Analysis (EDA)

1. An approach to analyze and investigate data sets to summarize their main
characteristics, often using statistical graphics and other data
visualization methods.
2. EDA is primarily used to see what data can reveal beyond the formal
modeling or hypothesis testing task.
3. It can help to identify obvious errors, as well as better understand the
patterns within the data, detect outliers or anomalous events, find
interesting relations among the variables.

Filter Approach (Filter Method)

Definition:
The filter approach is a feature selection method where features are selected
before model training using statistical measures, without involving any
machine learning algorithm.

Wrapper Method
Definition:
The wrapper method is a feature selection technique that selects features by
training and evaluating a machine learning model on different subsets of
features and choosing the subset that gives the best performance.

Embedded Method
Definition:
The embedded method is a feature selection approach where feature selection
is performed during the model training process, and the model itself decides
which features are important.

How to Construct a Decision Tree (Short & Clear Explanation)

A Decision Tree is constructed recursively by dividing the training data into smaller
and purer subsets.
Steps to construct a Decision Tree:
1. Select the best attribute to split the data using an Attribute Selection
Measure
(e.g., Information Gain, Gini Impurity).
2. Create a decision node using the selected attribute.
3. Split the dataset into smaller subsets based on attribute values.
 Repeat the process recursively for each subset until one of the following
conditions is met:
 All the records belong to the same class label (make it as
a leaf node)
 Maximum tree depth or minimum records per leaf is
reached.
 No more attributes to split on.
 No more records.
4. Assign a class label to each leaf node using the majority class.
5. Prune the tree to reduce overfitting and improve generalization.
1. Select the best attribute using Information Gain / Gini Index
2. Split the dataset and create a decision node
3. Recursively repeat for each subset
4. Stop when:
a. All data in a node belong to same class
b. No attributes left or limit reached
5. Assign majority class to leaf nodes
6. Apply pruning to avoid overfitting
Attribute Selection Measures choose the attribute that results in the most
homogeneous child nodes after splitting the data.
Gini Index
The Gini Index measures the homogeneity (purity) of a dataset.
If a population is completely pure (all records belong to one class), the probability that
two randomly selected items belong to the same class is 1.
Key points:
 Used for categorical target variables (e.g., Success / Failure).
 Supports binary splits only.
  Higher Gini value ⇒ higher homogeneity (as per this formulation).
 Used by CART (Classification and Regression Tree) algorithm.

Steps to Calculate Gini for a Split

1. Calculate Gini for each sub-node
2 2
Gini= p + q
where
p= probability of success,
q = probability of failure.
2. Calculate weighted Gini for the split:
Weighted Gini=∑ (sub-node size
total samples
× Gini of sub-node )

3. Choose the split with higher Gini value (more homogeneous).

Conclusion from Example

 Split on Gender gives higher Gini score than split on Class.
 Therefore, the decision tree splits on Gender.

Information Gain (Entropy)

Entropy measures the degree of disorder or impurity in a dataset.
Key points:
 Entropy = 0 → completely pure node

 Lower entropy ⇒ better split

 Entropy = 1 → completely impure (50%–50%)

 Information Gain is derived from entropy.

Entropy Formula
Entropy=− p log ⁡2 p−q log ⁡2 q

Steps to Calculate Information Gain

1. Calculate entropy of the parent node.
2. Calculate entropy of each child node after the split.
3. Compute weighted average entropy of the split.
4. Lower entropy means better split.

Information Gain
 Information Gain =1−Entropy

Data quality is a perception or an assessment of data’s fitness to serve its purpose in a

given context
1. Completeness: Completeness means that all required and mandatory data
fields are filled as expected, even if optional data is missing.
2. Consistency: Consistency ensures that the same data has identical values
across different systems and does not conflict with itself.
3. Conformity: Conformity means that data follows predefined standards such
as correct format, data type, and size.
4. Accuracy: Accuracy is the extent to which data correctly represents real-
world objects, events, or values.
5. Integrity: Integrity means validity of data across the relationships and
ensures that all data in a database can be traced and connected to other data.
6. Timeliness: Timeliness refers to the availability of up-to-date data at the
right time when it is required.
Examples of Data Quality Problems
 Noise
 Outliers
 Missing values
 Duplicate or redundant data
Noise refers to unwanted modification or disturbance in original data values.
Examples:
 Distorted voice during a poor phone call
 “Snow” on a television screen
Noisy Data
 Noisy data is meaningless or corrupt
 Difficult for machines to interpret
 Increases storage cost
 Negatively affects data mining and ML results
Causes: Faulty instruments, data entry errors, transmission errors, inconsistent
formats or codes.
Handling Noise
  Data smoothing (binning, regression, clustering)
 Collect more data
 Use PCA
 Apply regularization and cross-validation
Outliers
Outliers are data values that are significantly different from most other values in a
dataset.

Detecting Outliers
 Box plot
 Histogram
 Scatter plot
Handling Outliers
 Drop outlier records: Remove abnormal data points
 Cap outlier data: Limit extreme values to a threshold
 Assign a new value: Replace with mean or predicted value
 Transformation: Apply functions (log, sqrt) to reduce impact
Missing Values
Missing values occur when some attribute values are not recorded.
Reasons:
 Data not collected
 Attribute not applicable
Handling Missing Values
 Remove data records
 Estimate values (mean/median/mode/prediction)
 Ignore during analysis
 Replace with probable values

Duplicate or Redundant Data

Duplicate data refers to repeated records in a dataset.
Problems
 Causes bias
 Leads to overfitting
Handling
 Remove duplicates carefully
  Ensure duplicates are not real or intentionally oversampled

1️⃣ Noise
Noise refers to unwanted modification or disturbance in the original data values.
2️⃣ Outliers
Outliers are data objects whose values are significantly different from most other data
objects in a dataset.
3️⃣ Missing Values
Missing values occur when some attribute values are not recorded.
4️⃣ Duplicate or Redundant Data
Duplicate data refers to repeated records in a dataset.
How to Handle Outliers
Outliers can be handled using several techniques, depending on the nature of the data and the
analysis requirements.
1. Drop the outlier records:
If outliers occur due to data entry or measurement errors, removing those records helps
prevent them from skewing or biasing the analysis results.
2. Cap the outlier values:
In cases where extreme values behave similarly to normal values, outliers can be capped
at a certain threshold to reduce their influence on the analysis.
3. Assign a new value (Imputation):
When outliers are caused by incorrect data, they can be replaced with estimated values
such as the mean, median, or values predicted using regression techniques.
4. Apply data transformation:
Transforming the data using techniques like scaling or logarithmic transformation can
reduce the impact of outliers without removing data.

 Noisy data refers to meaningless or corrupt information.

 It cannot be correctly interpreted by machines.
 It increases storage requirements and reduces analysis accuracy.
 Causes include faulty instruments, data entry errors, transmission errors, and
inconsistent data formats.

Quantitative Data Qualitative Data

Number-based and measurable data Description-based and categorical data

Can be counted or measured Cannot be measured numerically

Answers how many, how much, or how often Answers why, how, or what

Analyzed using statistical methods Analyzed by grouping into categories

Used in calculations and mathematical analysis Used for interpretation and understanding

Examples: distance, height, weight, time, cost, Examples: gender, nationality, marital status, color,
number of visitors education level

Discrete Data Continuous Data

Data that can be counted Data that can be measured

Has a finite or countably infinite number of Can take any real value within a given range
values

Values are distinct and separate Values are continuous without gaps

Usually represented by integer variables Usually represented by real or floating-

point variables

Examples: number of students in a class, Examples: temperature, height, weight

number of words in a document, number of
heads in coin flips

Nominal Data Ordinal Data

Categories have no inherent order Categories have a meaningful

order or ranking

Cannot be quantified or ranked Can be ranked or ordered

Numeric operations are not meaningful Comparison based on order is

possible

Used only for labeling or classification Used for ranking or prioritization

Examples: hair color, eye color, gender, Examples: exam grades (A, B, C), competition
ranking (1st, 2nd, 3rd), economic status (high,
nationality, marital status medium, low), education level

1. Record Data: Record data is a collection of data objects where each

object (record) consists of a fixed set of attributes. Each record is usually
represented as a row and each attribute as a column in a table.
 1. Data Matrix: A data matrix is a representation of data where all
objects have the same fixed set of numeric attributes. The data is
organized as an m × n matrix, where m is the number of objects and n
is the number of attributes.
2. Document Data: Document data is a type of data in which each
document is represented as a vector of terms, where each term is an
attribute and the value of each attribute indicates the number of times
that term appears in the document.
3. Transaction Data: Transaction data is a special type of record data in
which each record represents a transaction containing a set of
items, such as products purchased together during a single shopping
trip.
2. Graph Data: Graph data represents data objects as nodes and the
relationships between them as edges, and is commonly used to model
networks such as social networks, web links, and molecular structures.
6. Sequential Transaction Data: Sequential transaction data is an extension
of transaction data where each transaction is associated with a time stamp,
allowing the analysis of the order in which transactions occur.
7. Sequence Data; Sequence data consists of an ordered list of individual
elements, such as characters, words, or biological symbols, where the order
or position of elements is important.
8. Time Series Data: Time series data is a collection of observations
recorded at regular time intervals, and it is mainly used to analyze trends and
predict future values.
9. Spatial Data :Spatial data contains information about the physical
location or geographical position of objects, such as coordinates, areas, or
regions.
[Link]-Temporal Data: Spatio-temporal data is a type of data that
combines spatial and temporal information, representing how the location
of objects changes over time.
Classification
Definition:
Classification is a supervised learning technique in which the output (target)
variable is categorical, and the model predicts one or more class labels for
each input instance.

Binary Classification
Definition:
Binary classification is a type of classification where each instance is
assigned to one of two possible classes.

Multiclass Classification
Definition:
Multiclass classification is a type of classification where each instance is
assigned to one of three or more classes.

Multi-Label Classification
Definition:
Multi-label classification is a type of classification where multiple class
labels can be assigned to a single instance at the same time.

Regression
Definition:
Regression is a supervised learning technique in which the output (target)
variable is a continuous numerical value, such as price, weight, or score.

Linear Regression
Definition:
Linear regression is a regression technique that models the relationship
between input variables and a continuous output using a linear (straight-
line) equation.

Polynomial Regression
Definition:
Polynomial regression is a regression technique that models the relationship
between variables using a polynomial equation to handle non-linear data.

Supervised Learning
Definition:
Supervised learning is a machine learning approach in which the model is
trained using labeled data, meaning each input has a corresponding output.

Unsupervised Learning
Definition:
Unsupervised learning is a machine learning approach in which the model is
trained using unlabeled data and discovers patterns or structures on its own.

Clustering
Definition:
Clustering is an unsupervised learning technique that groups data objects
such that objects within the same cluster are similar and objects in
different clusters are dissimilar.

Association Rule Mining

Definition:
Association rule mining is a rule-based machine learning technique used to
discover frequent patterns and relationships among items in large
datasets.
 How to Handle Outliers:
1. Drop the outlier records: Delete records containing extreme values if they
negatively affect analysis.
2. Cap outliers’ data: Limit extreme values to a maximum or minimum
threshold.
3. Assign a new value: Assign new values using mean, median, or prediction
methods.
4. Try a transformation: Use transformations (log, square root, etc.) to reduce
outlier impact.
 Lecture 18: Clustering
1. Well-Separated Clusters
In well-separated clusters,
a cluster is a set of points such that each point in the cluster is closer to every other point in the
same cluster than to any point outside the cluster.
👉 This means clusters are clearly separated from each other.

2. Center-Based (Partitioned) Clusters

In center-based clustering,
a cluster is a set of objects such that each object is closer to the center of its own cluster than to
the center of any other cluster.
 The center is called:
o Centroid → average of points

o Medoid → most representative point

👉 Example: K-means algorithm

3. Contiguity-Based Clusters (Nearest Neighbor)

In contiguity-based clustering,
a cluster is formed when a point is closer to one or more points in the same cluster than to
points in other clusters.
👉 Clusters are formed based on connectivity or chain of neighbors.

4. Density-Based Clusters
In density-based clustering,
a cluster is a dense region of data points separated by low-density regions.
👉 Features:

 Can find irregular shaped clusters

 Handles noise and outliers
👉 Example: DBSCAN algorithm
Hierarchical Clustering
Hierarchical clustering is a clustering technique that produces a set of nested clusters organized
in the form of a hierarchical tree.
It can be visualized using a dendrogram, which is a tree-like diagram showing the sequence of
merges or splits of clusters.

Types of Hierarchical Clustering

There are two main types:
1. Agglomerative Method (Bottom-Up Approach)
 Start with each data point as a separate cluster
 At each step, merge the two closest clusters
 Continue until only one cluster (or k clusters) remain

2. Divisive Method (Top-Down Approach)

 Start with all data points in one cluster
 At each step, split the cluster into smaller clusters
 Continue until each cluster contains one point (or k clusters)

Agglomerative Clustering Algorithm

1. Consider each data point as a separate cluster
2. Compute the proximity (distance) matrix
3. Repeat:
o Merge the two closest clusters

o Update the proximity matrix

4. Continue until only one cluster remains

Inter-Cluster Similarity Measures

The distance between clusters can be defined in different ways:
  Single Linkage (MIN):
Distance between closest points of two clusters
 Complete Linkage (MAX):
Distance between farthest points of two clusters
 Group Average:
Average distance between all pairs of points
 Centroid Distance:
Distance between cluster centers
 Ward’s Method:
Based on minimizing squared error

Strengths of Hierarchical Clustering

 No need to specify number of clusters in advance
 Can produce meaningful hierarchical structure
 Useful in applications like biological classification

Limitations
 Computationally expensive
 Sensitive to noise and outliers (especially single linkage)
 Some methods may break large clusters (complete linkage)
(a) What is Zero frequency problem in Naïve Bayes Classifier? How to handle
with Zero frequency problem? Explain with an example. 2+3

Zero Frequency Problem in Naïve Bayes Classifier

The zero frequency problem in Naïve Bayes occurs when a particular feature value does not
appear in the training dataset for a given class. In such cases, the conditional probability of that
feature given the class becomes zero.
Since Naïve Bayes calculates the posterior probability by multiplying several conditional
probabilities, the presence of any zero probability causes the entire probability of that class to
become zero. As a result, the classifier may incorrectly ignore that class, even if other features
strongly support it.
Example:
If a word does not appear in any training examples of a class, then
P(word ∣ class)=0,
which makes the overall probability of that class zero.
Handling the Problem (Solution) (3 marks)
The zero frequency problem is handled using Laplace Smoothing (Add-one smoothing).
In this method, we add 1 to each frequency count:
Count +1
P(Feature ∣Class)=
Total +V

Where:
 Count = frequency of the feature
 Total = total number of features in that class
 V = number of possible feature values
This ensures that no probability becomes zero.
Example
Suppose we classify emails as Spam or Not Spam.
Training data:
 Email Word = "Offer"
Type

Spam 0 times

Not Spam 5 times

Without smoothing:
P(Offer ∣Spam)=0

So, the Spam class will always get probability zero if the word “Offer” appears.
With Laplace smoothing:
0+1
P(Offer ∣Spam )=
Total+V

Now the probability is non-zero, so the classifier works correctly.

(a) Write down the properties of maximum likelihood estimators.

Properties of Maximum Likelihood Estimators (MLE)

The important properties of Maximum Likelihood Estimators are given below:
1. Consistency
An estimator is said to be consistent if it converges to the true parameter value as the sample
size increases.
That is, as n → ∞, the estimator approaches the true value of the parameter.
2. Efficiency
An estimator is efficient if it achieves the Cramér–Rao Lower Bound (CRLB), meaning it has
the minimum possible variance among all unbiased estimators.
3. Asymptotic Normality
For large sample sizes, the distribution of the MLE becomes approximately normal (Gaussian
distribution), even if the original data is not normally distributed.
4. Unbiasedness (Asymptotically)
MLE is asymptotically unbiased, meaning its expected value approaches the true parameter
value as the number of samples increases.
5. Minimum Variance (Asymptotic Efficiency)
For large samples, MLE achieves the minimum variance, making it an efficient estimator in
practice.
1) What is the core idea of k Nearest Neighbors (kNN) learning? How
to choose the value of k in kNN?

kNN is a lazy learning algorithm that stores all training data.

When a new data point comes, it finds the k nearest neighbors (based on
distance like Euclidean distance) and predicts the output based on those
neighbors.
The choice of k is very important in kNN because it directly affects the
performance of the model.
 If k is small, the model becomes sensitive to noise and outliers,
which may lead to overfitting.
 If k is large, the model becomes computationally expensive and
may ignore the local structure of data, leading to underfitting.
A common approach is to choose:
k =√ nwhere n is the number of training samples.
Another practical method is to try different values of k (e.g., k = 1, 3, 5, …)
and select the value that gives the best performance on training and test
data.
For binary classification, it is preferable to choose an odd value of k to
avoid ties in voting.

2. How to decide the class label in kNN?

The class label of a new data point in kNN is decided using the following
methods:

 Majority Voting:
The new data point is assigned the class that appears most
frequently among its k nearest neighbors.

 Distance-weighted Voting:
The votes of neighbors are weighted based on their distance.
Closer neighbors have more influence using weight:
1
w= 2 where d is the distance.
d
So, nearer points contribute more to the final decision.
c) Write the working principle of the support vector machine (SVM)
algorithm. How kernel function related to support vector machine?

Working Principle of SVM

Support Vector Machine (SVM) works by finding an optimal hyperplane that
separates data points of different classes.
 It selects the hyperplane that maximizes the margin between two
classes
 The closest data points to the hyperplane are called support vectors
 These support vectors determine the position of the hyperplane
 The goal is to achieve maximum separation with minimum error
Relation of Kernel Function with SVM
 Kernel function is used when data is not linearly separable
 It maps data from low-dimensional space to high-dimensional
space
 This transformation makes it possible to separate data using a linear
hyperplane
 Thus, kernel helps SVM to handle complex and non-linear data

(c) Why does Support Vector Machine (SVM) select the hyperplane for which
the margin is maximum? What are the kernel tricks in SVM?

Why maximum margin?

 Larger margin gives better generalization
 It reduces the chance of misclassification
 Makes the model more robust to noise
 Ensures better performance on unseen data
Kernel Trick in SVM
 Kernel trick transforms input data into higher dimension without
explicitly computing it
 It allows SVM to solve non-linear problems efficiently
Common kernels:
 Linear kernel
 Polynomial kernel
 Radial Basis Function (RBF) kernel

c) How does Support Vector Machines (SVMs) work? What are the kernel
tricks in SVM?
(b) How does Support Vector Machines (SVMs) work? How to identify the
right hyper-plane in SVM?

How to identify the right hyperplane

 Select hyperplane that separates classes correctly

 Among all possible hyperplanes, choose the one with maximum

margin

 Ensure minimum classification error

(c) How can SVM classify non-linearly separable data?

(c) How SVM classifies non-linearly separable data

 For non-linear data, SVM uses kernel functions
 Data is transformed into higher-dimensional space
 In that space, data becomes linearly separable
 Then a hyperplane is used to classify the data
🔷 Cross Validation Techniques (Definitions)

Cross-validation techniques are used to evaluate the performance of a model on unseen data
and to reduce overfitting by ensuring better generalization.

✅ 1. Holdout Method: The holdout method is a model validation technique in which the dataset
is divided into two disjoint sets: a training set and a test set. The model is trained on the training
set and evaluated on the test set.
✅ 2. Random Subsampling Method: Random subsampling is an extension of the holdout
method where the dataset is randomly split into training and test sets multiple times. The model
is trained and tested in each iteration, and the final performance is obtained by averaging the
results over all iterations.
✅ 3. k-Fold Cross Validation: In k-fold cross validation, the dataset is divided into k equal-sized
subsets (folds). The model is trained k times, each time using k−1 folds as the training set and the
remaining one fold as the test set. The final performance is the average of the k evaluation
results.
✅ 4. Leave-One-Out Cross Validation (LOOCV): Leave-One-Out Cross Validation is a special
case of k-fold cross validation where k is equal to the number of data samples. In each iteration,
one sample is used as the test set and the remaining samples are used for training. The final result
is obtained by averaging all iterations.
Cross-validation techniques are used to evaluate the performance of a model on unseen data
and to reduce overfitting by ensuring better generalization.

Generalization: In machine learning, generalization refers to the ability of a model to learn

patterns from the training data and apply them effectively to unseen data from the same domain,
resulting in good prediction performance.

Q.6 (a) What is regression analysis? Why do you need to use regression analysis?
Regression Analysis is a supervised learning technique used to predict a continuous (real-valued)
output by modeling the relationship between a dependent (target) variable and one or more
independent (predictor) variables.
Regression analysis is used for the following purposes:
1. Prediction:
It is used to predict future or unknown values, such as predicting house prices or student
marks.
2. Understanding Relationships:
It helps to understand how one variable affects another variable.
3. Trend Analysis:
It is used to analyze trends and patterns in data over time.
4. Decision Making:
It supports decision-making in fields like business, engineering, and science.
5. Modeling Data:
It is used to create mathematical models of real-world situations.
6. 🔹 Case 1: Noise (small disturbance)
7. Now data becomes:
70, 72, 68, 71, 69, 73, 71.5
8. 👉 That extra 71.5 is just a small variation
✔ It is close to others
✔ It doesn’t break the pattern
9. ➡️This is Noise
10.

11.🔹 Case 2: Outlier (big abnormal value)

12. Now data becomes:
70, 72, 68, 71, 69, 73, 10
13. 👉 That 10 is very different
✔ Far away from other values
✔ Breaks the pattern
14. ➡️This is an Outlier

(b) What is meant by regression line? How do you find a best-fit regression line?
(b) Regression Line
A regression line is a straight line that represents the relationship between an independent
variable (x) and a dependent variable (y). It is used to predict the value of the dependent
variable based on the independent variable. The equation of a regression line is:
y=mx+c where m is the slope and c is the intercept.
Best-Fit Regression Line
The best-fit regression line is the line that best represents the data points by minimizing the
error between the actual values and predicted values.
How to find the best-fit regression line
1. Use Least Squares Method:
The best-fit line is found by minimizing the sum of squared errors between actual and
predicted values.
2. Calculate slope (m):
n ∑ xy−(∑ x)(∑ y)
m=
n∑ x 2−¿¿
3. Calculate intercept (c):
∑ y−m∑ x
c=
n
4. Form the equation:
Substitute m and c into y=mx+c .

(b) What is meant by regression line? How do you find a best-fit regression line?
(b) Simple Linear Regression
Definition (1 mark):
Simple linear regression is a statistical method used to model the relationship between one
independent variable (x) and one dependent variable (y) using a straight line.

Core Concept (3 marks):

1. It assumes a linear relationship between input (x) and output (y).
2. The relationship is represented by the equation:
y=mx+c

where m is the slope and c is the intercept.

3. It predicts the value of y for a given x by fitting a straight line through the data points.

Best-Fit Regression Line (2 marks):

The best-fit regression line is obtained using the least squares method, which minimizes the
sum of squared differences between actual and predicted values.
  Slope:
n ∑ xy−(∑ x)(∑ y)
m=
n∑ x 2−¿¿
 Intercept:
∑ y−m∑ x
c=
n

Clustering

Types of Clusters

1. Well-Separated Clusters:
A cluster where all points are closer to each other than to any point outside the
cluster.
2. Center-Based (Partitioned) Clusters:
A cluster where each point is closer to its cluster center (centroid/medoid) than to other
cluster centers.
3. Contiguous (Connectivity-Based) Clusters:
A cluster where points are connected through nearest neighbors (each point is close to
at least one other point in the cluster).
4. Density-Based Clusters:
A cluster is a dense region of points separated by low-density areas; useful for
irregular shapes and handling noise/outliers.

Produces a set of nested clusters organized as a hierarchical tree

Can be visualized as a dendrogram– A tree like diagram that records the sequences of merges or
splits
Strengths of Hierarchical Clustering
 Do not have to assume any particular number of clusters– Any desired number of clusters can
be obtained by ‘cutting’ the dendogram at the proper level
 They may correspond to meaningful taxonomies– Example in biological sciences (e.g., animal
kingdom, phylogeny reconstruction, …)
Hierarchical Clustering: Two main types of hierarchical clustering
– Agglomerative:
 Start with the points as individual clusters
 At each step, merge the closest pair of clusters until only one cluster (or k clusters)
left
– Divisive:
 Start with one, all-inclusive cluster
 At each step, split a cluster until each cluster contains a point (or there are k
clusters)
 Traditional hierarchical algorithms use a similarity or distance matrix
– Merge or split one cluster at a time
Agglomerative Clustering Algorithm:
2. Let each data point be a cluster
3. Compute the proximity matrix (stores the distances between each pair of clusters)
4. Repeat
 Merge the two closest clusters
 Update the proximity matrix to reflect the proximity between the new
cluster & other clusters
5. Until only a single cluster remains

 Key operation is the computation/update of the proximity of two clusters

– Different approaches to defining the distance between clusters distinguish the different
algorithms
 Starting Situation : Start with clusters of individual points and a proximity matrix
 Intermediate Situation : After some merging steps, we have some clusters
 After Merging :The question is “How do we update the proximity matrix?”
How to Define Inter-Cluster Similarity Similarity?
  MIN / Single Linkage
  MAX / Complete Linkage
  Group Average
  Distance Between Centroids
  Other methods driven by an objective function
 – Ward’s Method uses squared error
Cluster Similarity: MIN or Single Linkage
 Similarity of two clusters is based on the two most similar (closest) points in the different
clusters
– Determined by one pair of points, i.e., by one link in the proximity graph
Strength of MIN : Can handle non-elliptical shapes
Limitations of MIN : Sensitive to noise and outliers

Cluster Similarity: MAX or Complete Linkage

 Similarity of two clusters is based on the two least similar (most distant) points in the
different clusters
– Determined by all pairs of points in the two clusters
Strength of MAX :Less susceptible to noise and outliers
Limitations of MAX : Tends to break large clusters •Biased towards globular clusters
A Proximity Matrix is a table (matrix) that shows how close or similar different items are to
each other.

(b) How to update the proximity matrix after merging two closest clusters?
When two clusters are merged, the proximity matrix is updated as follows:
1. Remove the rows and columns of the two merged clusters.
2. Add a new row and column for the new merged cluster.
3. Compute distances between the new cluster and all other clusters using a linkage
method:
o Single Link: minimum distance
o Complete Link: maximum distance
o Average Link: average distance
4. Keep diagonal values as 0.

Ensemble learning is a method where multiple models are combined instead of using just one.
Even if individual models are weak, combining their results gives more accurate and reliable
predictions.
Simple Ensemble Techniques
Ensemble techniques combine multiple models to improve prediction accuracy. The common
simple ensemble methods are:

1. Max Voting
Max voting is mainly used for classification problems.
 In this method, multiple models make predictions for the same data.
 Each model’s prediction is treated as a vote.
 The final prediction is the one that gets the majority vote (mode).
Example:
If 5 models predict ratings: 4, 5, 4, 4, 5 →
The majority prediction is 4, so final output = 4.

2. Averaging
Averaging is used for both regression and classification.
 Multiple models make predictions.
 The final prediction is the average of all outputs.
Formula:
Final Output = (Sum of all predictions) / (Number of models)
Example:
Predictions: 5, 4, 5, 4, 4
Average = (5 + 4 + 5 + 4 + 4) / 5 = 4.4

3. Weighted Averaging
Weighted averaging is an improved version of averaging.
 Each model is assigned a weight based on its importance or performance.
 Predictions are multiplied by their weights.
 Final output is the weighted sum.
Formula:
Final Output = Σ (Prediction × Weight)
Example:
(5×0.23) + (4×0.23) + (5×0.18) + (4×0.18) + (4×0.18) = 4.41

Advanced Ensemble Techniques

Advanced ensemble techniques combine multiple models to improve accuracy and reduce errors.
The main methods are:
 Bagging
 Boosting
 Stacking
1. Bagging (Bootstrap Aggregating)
Bagging is a technique where multiple models are trained in parallel using different subsets of
data.
 Multiple subsets are created from the original dataset using sampling with replacement
(bootstrapping).
 A base model (usually weak learners like decision trees) is trained on each subset.
 All models are independent and run in parallel.
 Final prediction is obtained by combining all model outputs (voting for classification,
averaging for regression).
Key Idea: Reduce variance and improve stability.

2. Boosting
Boosting is a sequential technique where each model tries to correct the errors of the previous
model.
Steps:
1. Initially, a base (weak) learner is trained by assigning equal weight to all observations.
2. After training, if there are prediction errors, the misclassified observations are given
higher weight (more importance).
3. A new base learner is trained focusing more on these difficult observations.
4. Steps 2 and 3 are repeated until the model achieves good accuracy or a maximum number
of iterations is reached.
5. Finally, all weak learners are combined to form a strong learner that provides more
accurate predictions.
Key Idea: Focus more on difficult (misclassified) data points.

Example: AdaBoost
 Initially, all data points have equal weight.
 First model (B1) misclassifies some points.
 These misclassified points are given higher weight.
 Second model (B2) focuses more on these errors but may misclassify others.
 Third model (B3) again adjusts weights and improves accuracy.
 Finally, all models (B1, B2, B3) are combined to create a strong model (B4).

3. Stacking
Stacking is an ensemble technique where predictions of multiple models are used to train a new
model.
 Multiple base models are trained on the dataset.
 Each model makes predictions.
 These predictions are used as input features for another model called the meta-model.
 The meta-model makes the final prediction.
Key Idea: Learn how to best combine different models.
Advanced Boosting Methods
1. What is GBM?
Gradient Boosting Machine (GBM) is an ensemble learning method where models are built
sequentially, and each new model tries to correct the errors of the previous one using gradient
descent.
2. What is XGBoost?
XGBoost (Extreme Gradient Boosting) is an improved version of GBM. It provides high
performance, regularization, and parallel processing, making it faster and more accurate
3. What is LightGBM?
LightGBM is a boosting algorithm developed by Microsoft. It uses histogram-based learning and
leaf-wise tree growth, making it very fast and memory efficient, especially for large datasets.
4. Advantages of XGBoost and LightGBM
XGBoost:
 High accuracy
 Handles missing values
 Reduces overfitting using regularization
LightGBM:
 Faster training speed
 Low memory usage
 Efficient for large datasets
5. Building Classifiers
GBM, XGBoost, and LightGBM are used to build classification models by:
1. Training the model on data
2. Making predictions
3. Evaluating performance

7. LightGBM vs XGBoost
 LightGBM → Faster and better for large datasets
 XGBoost → More accurate and better control over overfitting
What is CatBoost?
CatBoost (Categorical Boosting) is a gradient boosting algorithm developed by Yandex. It is specially
designed to handle categorical data efficiently.
Features:
 Automatically handles categorical features
 Uses ordered boosting to reduce overfitting
 Provides high accuracy
Holdout Method
 Split data into Training set and Test set
 Use k% for training, (100−k)% for testing
 Train model on training data, evaluate on test data
Limitations of Holdout
 Less data for training
 Result depends on how data is split
 Single split → may give misleading accuracy
Random Subsampling
 Repeat holdout multiple times (k iterations)
 Randomly select test set each time
 Average accuracy:
k

∑ ac c i
ac c ¿= i=1
k
 Some data may not be used
 Number of iterations not fixed
Cross Validation (k-fold)
 Split data into k folds
 For each fold:
o Use 1 fold as test
o Use remaining (k−1) folds as training
 Train and evaluate k times
 Final result = average of all scores
Variations
 Repeated CV → run CV multiple times for better accuracy
 Stratified CV → keeps class ratio same (important for imbalanced data)
LOOCV (Leave-One-Out)
 Special case: k = N (dataset size)
 Each test set has 1 sample
 Uses maximum data for training
 Very slow, but highly reliable

Generalization means a model’s ability to perform well on new, unseen data, not just the data it
was trained on.

Bias and Variance (Basic Idea)

 Total prediction error:
2
Error=Bia s +Variance+ Irreducible Error
  Irreducible error → cannot be removed (noise, unknown factors)
 Reducible error → Bias + Variance (can be improved)

Bias Error
 Difference between actual and predicted values due to wrong assumptions
 High Bias → Underfitting (model too simple, misses patterns)
 Learns training data poorly
Examples:
 High Bias → Linear Regression, Logistic Regression
 Low Bias → Decision Trees, k-NN, SVM

Variance Error
 Error due to sensitivity to small changes in training data
 High Variance → Overfitting (model too complex, memorizes data)
Examples:
 High Variance → Decision Trees, k-NN, SVM
 Low Variance → Linear Regression, Logistic Regression

Bias–Variance Tradeoff
 Goal: Low Bias + Low Variance
 But reducing one often increases the other
 Need a balance to minimize total error

Naive Bayes Classifier — Short Summary

 Used to predict class label (Y) from input data (X) using Bayes’ Theorem
 Basic formula:
P( X ∣ y) P( y )
P( y ∣ X )=
P( X )
 For multiple features X =(x 1 , x 2 ,... , x n):
o Assumes features are independent (naive assumption)
o So,
P( y ∣ x 1 ,... , x n)∝ P(x 1 ∣ y) P(x 2 ∣ y) ... P(x n ∣ y ) P( y)

The curse of dimensionality refers to the problem that arises when the number of features
(dimensions) increases in a dataset. As dimensionality increases, the data points become sparse
in the feature space
Adv. & Disadv. of Naïve Bayes Classifier
Advantage
 Works surprisingly well
 Simple
 Handling missing value is easier
 Robust to irrelevant attributes
Disadvantage
 Can’t handle dependent variables
 Suffers from “Zero Frequency” problem
Adv. & Disadv. of SVM
Advantage
 SVM offers very high accuracy compared to other classifiers such as logistic
regression, decision trees, and Naïve Bayes.
 SVM works well with a clear margin of separation and with high dimensional
space.
 It uses less memory because it uses a subset of training points in the decision
phase.
Disadvantage
 Required training time is higher for large datasets.
 It works poorly with overlapping classes, e.g. when the data set has more noise
 It is sensitive to the type of kernel used
Advantages of Random Forests
 Random forests is considered as a highly accurate and robust method because of the
number of decision trees participating in
the process.
 It likely does not suffer from the overfitting problem because it creates multiple trees
on random subsets, takes the average or most votes of the predictions of the trees, which
cancel out the biases. The randomness and voting or averaging mechanisms in random
forests elegantly solve the overfitting problem.
 It can handle missing data.
 It can be used in both classification and regression problems.
Disadvantages of Random Forests
 Random forests is slow because it builds multiple decision trees and makes the final
prediction by combining the predictions of each individual tree.
 The model is difficult to interpret compared to a decision tree, where you can easily
make a decision by following the path in the tree

Adv. & Disadv. of Decision Trees

Advantage
 Easy to Understand
 Less data cleaning required
 Can handle both numerical and categorical variables
 Useful in Data exploration
Disadvantage
 May contain lots of layers, which makes it complex
 May have an overfitting issue
 Feed-Forward Neural Network (FFNN)

👉 A feed-forward neural network is a type of neural network in which

data flows in one direction—from input layer to hidden layer(s) to output
layer—without any loops or feedback connections.

Working of a Single Neuron

Given input vector:

x=(x 1 , x 2 , ... , x n )

1. Each input x iis multiplied by weight w ij

2. All values are summed:

ne t j=∑ x i wij

3. Add bias (threshold) θ j

4. Apply activation function ϕ

👉 Final output:

o j=ϕ ( ∑ x i wij +θ j )
Multi-Layer Form

For multiple layers:

x l+1=ϕ (W l xl +θ l)

👉 Where:

 W l = weight matrix

 x l= input of layer

 θl = bias

 ϕ = activation function

👉 i = neuron in the previous layer

👉 j = neuron in the current layer