MACHINE LEARNING

Contents
1. Introduction to Machine Learning: .........................................................................................4
Key Parts of Machine Learning ..................................................................................................5
Types of Machine Learning ........................................................................................................5
Applications of Machine Learning .............................................................................................7
2. Machine Learning Classification: ...........................................................................................8
Characteristics of Classification ...............................................................................................8
Types of Classification ..............................................................................................................8
Classification Algorithms ..........................................................................................................9
Evaluation Metrics for Classification ....................................................................................... 10
Steps in Classification Process ............................................................................................... 11
Applications of Classification ................................................................................................. 11
3. Linear Regression ............................................................................................................... 12
Types of Linear Regression...................................................................................................... 12
4. Cost Functions ................................................................................................................... 13
Cost Function for Linear Regression ........................................................................................ 13
Why Mean Squared Error?....................................................................................................... 13
Shape of the Cost Function ..................................................................................................... 13
5. Gradient Descent................................................................................................................ 14
Intuition Behind Gradient Descent .......................................................................................... 14
Gradient Descent Algorithm Steps .......................................................................................... 14
Gradient Descent Update Rule................................................................................................ 14
Learning Rate ......................................................................................................................... 15
Types of Gradient Descent ...................................................................................................... 15
6. Gradient Descent for Classification ..................................................................................... 15
Using Linear Regression for Classification ............................................................................... 15
How It Works .......................................................................................................................... 15
Role of Gradient Descent ........................................................................................................ 15
Limitations ............................................................................................................................. 16
7. Gradient Descent Classification Evaluation: ........................................................................ 16
Why Evaluation is Important ................................................................................................... 16
Evaluation Methods ................................................................................................................ 16

1
 MACHINE LEARNING

8. Bias .................................................................................................................................... 20
9. Variance ............................................................................................................................. 21
Characteristics of High Variance ............................................................................................. 21
10. Bias vs Variance (Comparison Table) .............................................................................. 22
11. Bias–Variance Tradeoff ................................................................................................... 22
12. Underfitting vs Overfitting ............................................................................................... 23
13. Bias vs Variance in Linear Regression .............................................................................. 23
14. How to Control Bias and Variance ................................................................................... 23
15. Clustering ...................................................................................................................... 24
Objectives of Clustering ......................................................................................................... 24
Types of Clustering Methods ................................................................................................... 24
Clustering relies on similarity measures such as: .................................................................... 26
16. K-Means Clustering ........................................................................................................ 28
17. K-Median Clustering ....................................................................................................... 29
18. Comparison Table: K-Means vs K-Median ........................................................................ 30
19. Elbow Method ................................................................................................................ 31
Concept ................................................................................................................................. 31
Steps to Apply the Elbow Method ............................................................................................ 31
Advantages ............................................................................................................................ 32
Disadvantages ....................................................................................................................... 32
Applications ........................................................................................................................... 32
20. Artificial Neural Methods ................................................................................................ 32
Key Components of an ANN .................................................................................................... 32
Popular Activation Functions .................................................................................................. 33
Architecture of ANN................................................................................................................ 33
Working Principle ................................................................................................................... 34
Learning Process .................................................................................................................... 34
Training Algorithm .................................................................................................................. 34
Advantages of ANN................................................................................................................. 34
Disadvantages of ANN ............................................................................................................ 35
Applications of ANN ............................................................................................................... 35
21. Feed Forward Neural Networks (FFNN) ........................................................................... 35
Architecture ........................................................................................................................... 35

2
 MACHINE LEARNING

Working Principle ................................................................................................................... 36

Key Features .......................................................................................................................... 36
Advantages ............................................................................................................................ 36
Disadvantages ....................................................................................................................... 37
Applications ........................................................................................................................... 37
22. Feedback Ward Methods (Recurrent Neural Networks – RNN).......................................... 37
Architecture ........................................................................................................................... 37
Methods of Feedback Ward (Recurrent Neural Networks) ........................................................ 38
23. Feature Reduction Methods ............................................................................................ 40
Why Feature Reduction is Important ....................................................................................... 40
Types of Feature Reduction Methods ...................................................................................... 40
Methods of Feature Selection .............................................................................................. 40
Feature Extraction .................................................................................................................. 41
Popular Feature Extraction Techniques................................................................................ 41
24. Decision Tree ................................................................................................................. 42
Components of a Decision Tree .............................................................................................. 42
Working Principle ................................................................................................................... 42
Splitting Criteria ..................................................................................................................... 43
Types of Decision Trees .......................................................................................................... 43
Advantages ............................................................................................................................ 44
Disadvantages ....................................................................................................................... 44
Applications ........................................................................................................................... 44
25. Distance-Based Methods ............................................................................................... 44
Key Concept........................................................................................................................... 45
Common Distance Measures.................................................................................................. 45
Examples of Distance-Based Methods .................................................................................... 45
Advantages ............................................................................................................................ 46
Disadvantages ....................................................................................................................... 46
Applications ........................................................................................................................... 46

3
 MACHINE LEARNING

1. Introduction to Machine Learning:

Machine learning is the study of computer algorithms that improve their performance at a
task through experience. The experience is typically in the form of data, and the
performance is measured with respect to a task.

A learning system constructs a model from data and uses it to make predictions or
decisions.

According to Tom Mitchell (1997): “A computer program is said to learn from experience E
with respect to some task T and some performance measure P, if its performance on task
T, as measured by P, improves with experience E.”

Task (T): What the computer is trying to do (e.g., classify emails as spam or not spam).

Experience (E): The data the computer learns from (e.g., past emails).

Performance (P): How we check if it’s learning well (e.g., accuracy of predictions).

4
 MACHINE LEARNING

Key Parts of Machine Learning

i. Data ii. Features

iii. Model iv. Learning Algorithm v. Evaluation

i. Data: Data is the foundation of machine learning. It can be structured (tables,

databases), semi-structured (XML, JSON), or unstructured (text, images, audio). The
quality, quantity, and relevance of data significantly affect model performance.

ii. Features: Features are measurable attributes or variables extracted from raw data.
Feature engineering is a critical step that involves selecting, transforming, and
creating features to enhance learning efficiency.

iii. Model: A model is a mathematical representation that maps input features to

output predictions. Examples include linear models, decision trees, neural
networks, and probabilistic models.

iv. Learning Algorithm: The learning algorithm optimizes the model parameters by
minimizing a loss or error function using techniques such as gradient descent or
optimization heuristics.

v. Evaluation: Model performance is evaluated using statistical metrics such as

accuracy, precision, recall, F1-score, mean squared error, and area under the ROC
curve.

Types of Machine Learning

i. Supervised Learning ii. Unsupervised Learning

iii. Semi-Supervised Learning iv. Reinforcement Learning

i. Supervised Learning:

5
 MACHINE LEARNING

Supervised learning involves training a model on labeled data, where both input and
corresponding output are known.

Common tasks:

• Classification (spam detection, disease diagnosis)

• Regression (house price prediction)

Algorithms:

• Linear and Logistic Regression

• Support Vector Machines (SVM)
• Decision Trees and Random Forests
• k-Nearest Neighbors (k-NN)
• Neural Networks

ii. Unsupervised Learning:

Unsupervised learning deals with unlabeled data and aims to discover hidden patterns
or structures.

Common tasks:
• Clustering
• Dimensionality reduction
• Association rule mining

Algorithms:

• K-means Clustering
• Hierarchical Clustering
• DBSCAN
• Principal Component Analysis (PCA)
• Apriori Algorithm

iii. Semi-Supervised Learning

Semi-supervised learning uses a combination of labeled and unlabeled data. This

approach is particularly useful when labeled data is scarce or expensive to obtain.

Applications:

• Medical imaging
• Speech recognition
6
 MACHINE LEARNING

• Web content classification

iv. Reinforcement Learning

Reinforcement learning focuses on learning optimal actions through interaction with

an environment. An agent learns by receiving rewards or penalties based on its
actions.

Algorithms:

• Q-Learning,
• SARSA
• Deep Reinforcement Learning (DQN)

Applications of Machine Learning

• Healthcare: Predict diseases, analyze medical images.

• Finance: Detect fraud, predict stock prices.

• Education: Predict student performance, personalize learning.

• Social media: Recommend content or ads.

• Industry: Predict maintenance needs in machines.

1.2. Advantages

• Ability to handle large and complex datasets

• Automation of decision-making processes

• Continuous improvement with more data

• High accuracy in predictive tasks

7
 MACHINE LEARNING

2. Machine Learning Classification:

Classification is a type of supervised machine learning task where the goal is to predict
the categorical class label of new instances based on past observations. In other words,
given a dataset where each example is labeled with a class, a classification algorithm
learns a mapping function from inputs (features) to discrete outputs (classes).

Formally, suppose we have:

• A dataset D= {(x1, y1), (x2, y2) ..., (xn,yn)}

where xi ∈ R^m is a feature vector with m attributes and yi ∈ {C1, C2..., Ck} is a class
label.

• The task of classification is to learn a function f: R^m → {C1, C2..., Ck} that can
assign a correct class label to new unseen instances.

Characteristics of Classification
i. Supervised Learning:
Classification requires labeled data; each training example must have a known
output.

ii. Discrete Outputs:

Unlike regression (predicting continuous values), classification predicts discrete
categories.

iii. Decision Boundaries:

Most classifiers aim to find boundaries in the feature space that separate different
classes.

iv. Probabilistic Interpretation (optional):

Many classifiers can provide a probability P(y∣x)P(y|x)P(y∣x) of each class for an
instance xxx.

Types of Classification
i. Binary Classification:
Only two classes exist (e.g., spam vs. non-spam, disease vs. healthy).

ii. Multiclass Classification:

More than two classes (e.g., classifying types of flowers: rose, tulip, sunflower).

8
 MACHINE LEARNING

iii. Multilabel Classification:

Each instance can belong to multiple classes simultaneously (e.g., tagging
multiple topics in a news article).

Classification Algorithms
i. Logistic Regression:

o A linear model used for binary or multiclass classification.

o Uses the sigmoid function for binary outcomes and softmax for multiple
classes.

o Probabilistic interpretation

ii. Decision Trees:

o Recursive partitioning of the feature space based on feature values.

o Easy to interpret, but prone to overfitting.

iii. Random Forest:

o An ensemble of decision trees.

o Reduces overfitting and improves predictive accuracy.

iv. Support Vector Machines (SVM):

o Finds the optimal hyperplane that maximizes the margin between classes.

o Effective in high-dimensional spaces.

v. K-Nearest Neighbors (KNN):

o Classifies a sample based on the majority class of its k nearest neighbors.

o Simple but computationally expensive on large datasets.

vi. Naive Bayes:

o Based on Bayes theorem and assumes feature independence.

o Fast and effective, especially for text classification.

vii. Neural Networks:

9
 MACHINE LEARNING

o Deep learning models capable of capturing complex patterns.

o Used in image, speech, and text classification tasks.

Evaluation Metrics for Classification

It’s important to measure how well a classifier performs. Common metrics include:

i. Accuracy:

ii. Precision, Recall, and F1-score:

• Precision: Fraction of relevant instances among retrieved instances.

• Recall (Sensitivity): Fraction of relevant instances that were retrieved.

• F1-score: Harmonic mean of precision and recall.

iii. Confusion Matrix: A table summarizing the true vs. predicted labels.

iv. ROC Curve & AUC: Evaluate classifier performance for binary problems at various
thresholds.

10
 MACHINE LEARNING

Steps in Classification Process

Applications of Classification
1. Medical Diagnosis: Classifying patients as diseased or healthy.

2. Email Filtering: Spam vs. non-spam.

3. Customer Segmentation: Classifying customers by purchasing behavior.

4. Image Recognition: Identifying objects or faces.

5. Sentiment Analysis: Positive, negative, or neutral opinions.

11
 MACHINE LEARNING

3. Linear Regression
Linear Regression is a supervised learning algorithm used to predict a numerical value.

It finds a straight-line relationship between:

• Input (features) → 𝑥

• Output (target) → 𝑦

Example

• Predicting house price from house size

• Predicting salary from years of experience

Types of Linear Regression

i. Simple Linear Regression:

Used when there is only one input feature.

𝑦 = 𝜃0 + 𝜃1 𝑥

X= Input feature

Y= predicted output

𝜃0 = intercept (bias)

𝜃1 = slop (weight)

ii. Multiple Linear Regression:

𝑦 = 𝜃0 + 𝜃1 𝑥1 + 𝜃2 𝑥2 + ⋯ + 𝜃𝑛 𝑥𝑛

iii. Multivariate Linear Regression

o Multiple input variables and multiple output variables

12
 MACHINE LEARNING

4. Cost Functions
A cost function measures how well the model fits the data.

• Small cost → good predictions

• Large cost → poor predictions

The training process aims to minimize the cost function.

Cost Function for Linear Regression

The most commonly used cost function is Mean Squared Error (MSE).
𝒎
𝟏
𝑱(𝜽) = ̂ 𝒊 − 𝒚𝒊 ) 𝟐
∑( 𝒚
𝟐𝒎
𝒊=𝟏

Where:

• 𝑚→ number of training samples

• 𝑦̂𝑖 → predicted value

• 𝑦𝑖 → actual value

Why Mean Squared Error?

• Squaring removes negative signs

• Larger errors are penalized more heavily

• Differentiable → suitable for optimization

Shape of the Cost Function

• The cost function is convex

• Has a single global minimum

• Guarantees convergence using gradient descent

13
 MACHINE LEARNING

5. Gradient Descent
Gradient Descent is an iterative optimization algorithm used to minimize the cost function
by updating model parameters gradually.

Intuition Behind Gradient Descent

Imagine standing on a mountain:

• Height → cost function value

• Direction of steepest slope → gradient

• Moving downhill → minimizing cost

Gradient Descent Algorithm Steps

a. Initialize parameters 𝜽randomly

b. Compute predictions

c. Calculate cost

d. Compute gradient

e. Update parameters

f. Repeat until convergence

Gradient Descent Update Rule

𝛛𝑱(𝜽)
𝜽: = 𝜽 − 𝜶
𝛛𝜽

Where:

• 𝛼→ learning rate
∂𝐽
•
∂𝜃
→ gradient (partial derivative)

14
 MACHINE LEARNING

Learning Rate
• Small 𝛼→ slow convergence

• Large 𝛼→ divergence or overshooting

• Proper tuning is essential

Types of Gradient Descent

1. Batch Gradient Descent – uses entire dataset

2. Stochastic Gradient Descent (SGD) – uses one sample

3. Mini-Batch Gradient Descent – uses subset of data

6. Gradient Descent for Classification

Using Linear Regression for Classification
Although linear regression is meant for regression tasks, it can be adapted for binary
classification.

How It Works
• Model outputs continuous values

• A threshold converts output to class labels

𝟏 if 𝒚
̂ ≥ 𝟎. 𝟓
Class = {
𝟎 if 𝒚
̂ < 𝟎. 𝟓

Role of Gradient Descent

Gradient descent:

• Minimizes the cost function

• Adjusts parameters to reduce misclassification

15
 MACHINE LEARNING

Limitations
• Output not bounded between 0 and 1

• Sensitive to outliers

• Poor probability interpretation

7. Gradient Descent Classification Evaluation:

When gradient descent is used to train a classification model (such as linear regression
with thresholding or logistic regression), we must evaluate how well the trained model
performs.

Evaluation answers questions like:

• How accurate are the predictions?

• How many errors are being made?

• What type of errors are more frequent?

Why Evaluation is Important

• Gradient descent minimizes a cost function, not classification accuracy directly

• A low cost does not always mean good classification

• Evaluation metrics help:

o Compare different models

o Detect overfitting or underfitting

o Choose the best threshold value

Evaluation Methods
i. Confusion Matrix

The confusion matrix is the foundation of most classification evaluation methods.

Binary Classification Confusion Matrix

16
 MACHINE LEARNING

Actual \ Predicted Positive (1) Negative (0)

Positive (1) TP FN

Negative (0) FP TN

Where:

• TP (True Positive) → Correctly predicted positive

• TN (True Negative) → Correctly predicted negative

• FP (False Positive) → Incorrectly predicted positive

• FN (False Negative) → Incorrectly predicted negative

ii. Accuracy

Accuracy measures the overall correctness of the classifier.

Formula
𝑻𝑷 + 𝑻𝑵
Accuracy =
𝑻𝑷 + 𝑻𝑵 + 𝑭𝑷 + 𝑭𝑵

Interpretation

• High accuracy → many correct predictions

• Easy to understand

Limitation

• Misleading for imbalanced datasets

• Example: predicting all samples as the majority class can still give high accuracy

iii. Precision

Precision measures how many predicted positives are actually positive.

Formula

17
 MACHINE LEARNING

𝑻𝑷
Precision =
𝑻𝑷 + 𝑭𝑷

Interpretation

• High precision → few false positives

• Important when false positives are costly

Example Use Case

• Spam detection

• Medical diagnosis where false alarms are harmful

iv. Recall (Sensitivity)

Recall measures how many actual positives are correctly identified.

Formula
𝑻𝑷
Recall =
𝑻𝑷 + 𝑭𝑵

Interpretation

• High recall → few false negatives

• Important when missing a positive case is dangerous

Example Use Case

• Cancer detection

• Fraud detection

v. F1-Score

F1-score is the harmonic mean of precision and recall.

Formula
Precision × Recall
F1-score = 𝟐 ×
Precision + Recall

18
 MACHINE LEARNING

Interpretation

• Balances precision and recall

• Useful when data is imbalanced

• High F1 → good overall classification performance

vi. Specificity (True Negative Rate)

Specificity measures how well the model identifies negative cases.

Formula
𝑻𝑵
Specificity =
𝑻𝑵 + 𝑭𝑷

Interpretation

• High specificity → few false positives

• Important in medical screening

vii. Error Rate

Percentage of incorrect predictions.

Formula

Error Rate = 𝟏 − Accuracy

viii. Threshold Selection and Its Effect

In gradient descent–based classifiers:

• Model outputs continuous values

• A threshold converts output into class labels

Changing the threshold affects:

19
 MACHINE LEARNING

• Precision

• Recall

• Accuracy

Lower threshold → higher recall, lower precision

Higher threshold → higher precision, lower recall

ix. ROC Curve and AUC (Conceptual)

ROC Curve

• Plots True Positive Rate (Recall) vs False Positive Rate

• Shows performance across different thresholds

AUC (Area Under Curve)

• Measures overall classifier performance

• Value ranges from 0 to 1

• Higher AUC → better classifier

8. Bias
Bias is the error caused by wrong or overly simple assumptions made by the model about
the data.

Characteristics of High Bias

• Model is too simple

• Fails to capture data patterns

• Performs poorly on both training and test data

• Leads to underfitting

20
 MACHINE LEARNING

Example

• Linear regression on highly non-linear data

• Shallow decision tree

Effect

• Predictions are consistently far from actual values

• Errors are systematic

9. Variance
Variance is the error caused by a model being too sensitive to small changes in the training
data.

Characteristics of High Variance

• Model is too complex

• Learns noise in the data

• Very low training error

• High test error

• Leads to overfitting

Example

• Very deep decision tree

• High-degree polynomial regression

Effect

• Model predictions change significantly with different training sets

21
 MACHINE LEARNING

10. Bias vs Variance (Comparison Table)

Aspect Bias Variance

Definition Error due to wrong assumptions Error due to sensitivity to data

Model Complexity Too simple Too complex

Main Problem Underfitting Overfitting

Training Error High Low

Test Error High High

Flexibility Low High

Sensitivity to Data Low High

11. Bias–Variance Tradeoff

What is the Tradeoff?

• Increasing model complexity:

o ↓ Bias

o ↑ Variance

• Decreasing model complexity:

o ↑ Bias

o ↓ Variance

The goal is to balance both to minimize total error.

Error Decomposition

Total Error = Bias𝟐 + Variance + Noise

• Bias² → error from wrong assumptions

• Variance → error from sensitivity

22
 MACHINE LEARNING

• Noise → unavoidable error in data

12. Underfitting vs Overfitting

Aspect Underfitting Overfitting

Bias High Low

Variance Low High

Model Fit Poor Too perfect

Generalization Poor Poor

13. Bias vs Variance in Linear Regression

• Linear Regression

o High bias

o Low variance

• Polynomial Regression

o Lower bias

o Higher variance

14. How to Control Bias and Variance

To Reduce Bias

• Use a more complex model

• Add more features

• Reduce regularization

To Reduce Variance

• Collect more data

23
 MACHINE LEARNING

• Use regularization

• Simplify the model

• Use ensemble methods

15. Clustering
Clustering is an unsupervised learning technique used to group data objects into clusters
such that:

• Objects within the same cluster are more similar to each other

• Objects in different clusters are less similar

Clustering does not require labeled data, making it useful for discovering hidden patterns
in datasets.

Objectives of Clustering
• Identify natural groupings in data

• Maximize intra-cluster similarity

• Minimize inter-cluster similarity

• Discover structure in unlabeled data

Types of Clustering Methods

1. Partition-Based Clustering

• Divides data into a fixed number of clusters

• Each data point belongs to exactly one cluster

Examples:

• K-means

• K-median

2. Hierarchical Clustering

24
 MACHINE LEARNING

Creates a hierarchy of clusters.

Types:

• Agglomerative (Bottom-Up):

o Each data point starts as a single cluster

o Clusters are merged step by step

• Divisive (Top-Down):

o All data points start in one cluster

o Clusters are split recursively

Output: Dendrogram

3. Density-Based Clustering

• Groups dense regions of data points

• Identifies noise and outliers

Example:

• DBSCAN

4. Grid-Based Clustering

• Divides data space into finite number of grids

• Clustering is done on grids instead of data points

Example:

• STING

5. Model-Based Clustering

• Assumes data follows a statistical model

• Finds best model fit

25
 MACHINE LEARNING

Example:

• Gaussian Mixture Models (GMM)

Clustering relies on similarity measures such as:

i. Euclidean distance
ii. Manhattan distance
iii. Cosine similarity
iv. Mahalanobis distance

i. Euclidean Distance

Euclidean distance is the most commonly used distance measure that calculates the
straight-line distance between two data points in multidimensional space. It is widely used
for continuous numerical data.

Formula:

𝒅(𝑿, 𝒀) = √∑(𝒙𝒊 − 𝒚𝒊 )𝟐

Features:

• Simple and intuitive

• Sensitive to outliers

• Requires data normalization

Applications:
K-means clustering, KNN, image processing

ii. Manhattan Distance

Manhattan distance, also known as City Block distance, measures distance as the sum of
absolute differences between corresponding attributes.

Formula:

𝒅(𝑿, 𝒀) = ∑ ∣ 𝒙𝒊 − 𝒚𝒊 ∣

26
 MACHINE LEARNING

Features:

• Less sensitive to outliers than Euclidean distance

• Measures distance along axes only

Applications:
K-median clustering, grid-based path problems

iii. Cosine Similarity

Cosine similarity measures the similarity between two vectors based on the angle between
them rather than physical distance. It is commonly used for high-dimensional and sparse
data.

Formula:
𝑿⋅𝒀
Cosine Similarity =
∥ 𝑿 ∥∥ 𝒀 ∥

Features:

• Ignores magnitude

• Values range from –1 to 1

Applications:
Text mining, document similarity, recommendation systems

iv. Mahalanobis Distance

Mahalanobis distance measures the distance between two points while considering
correlations among variables. It is scale-invariant and useful for multivariate data analysis.

Formula:

𝒅(𝑿, 𝒀) = √(𝑿 − 𝒀)𝑻 𝑺−𝟏 (𝑿 − 𝒀)

Features:

• Handles correlated features

27
 MACHINE LEARNING

• Effective for outlier detection

Applications:
Anomaly detection, pattern recognition

16. K-Means Clustering

K-Means is a partition-based clustering algorithm used to divide a dataset into K non-
overlapping clusters, where each cluster is represented by the mean (centroid) of its
points. It is widely used in unsupervised learning.

Objective

• Minimize intra-cluster variance (distance of points from the cluster centroid)

• Maximize inter-cluster separation

Algorithm Steps

1. Select number of clusters K.

2. Initialize K centroids randomly.

3. Assign each data point to the nearest centroid (using Euclidean distance).

4. Recalculate centroids as the mean of assigned points.

5. Repeat steps 3–4 until centroids stabilize or assignments do not change.

Distance Measure

• Typically Euclidean distance is used.

𝒅(𝑿, 𝒀) = √∑(𝒙𝒊 − 𝒚𝒊 )𝟐

Advantages

• Simple and easy to implement

• Fast and efficient for large datasets

• Works well when clusters are spherical and equally sized

28
 MACHINE LEARNING

Disadvantages

• Requires predefined K

• Sensitive to outliers and noise

• Sensitive to initial centroid selection

• Poor performance with non-spherical clusters

Applications

• Market segmentation

• Document clustering

• Image compression

• Customer behavior analysis

17. K-Median Clustering

K-Median clustering is similar to K-Means but uses the median of the cluster points as the
cluster center. It is more robust to noise and outliers.

Objective

• Minimize the sum of absolute distances (Manhattan distance) between points and
cluster center.

𝒅(𝑿, 𝒀) = ∑ ∣ 𝒙𝒊 − 𝒚𝒊 ∣

Algorithm Steps

1. Select number of clusters K.

2. Initialize K medians (cluster centers).

3. Assign each data point to the nearest median.

4. Update medians to minimize total distance (choose point that minimizes total
Manhattan distance).

5. Repeat steps 3–4 until convergence.

29
 MACHINE LEARNING

Distance Measure

• Typically Manhattan distance is used.

Advantages

• Robust to outliers and extreme values

• Works well with skewed data

• Can handle categorical and numerical data

Disadvantages

• Slower convergence than K-Means

• More computationally expensive

• Less efficient for very large datasets

Applications

• Facility location planning

• Medical data clustering

• Customer segmentation with skewed data

18. Comparison Table: K-Means vs K-Median

Feature K-Means K-Median

Cluster center Mean Median

Distance measure Euclidean Manhattan

Sensitivity to outliers High Low

30
 MACHINE LEARNING

Feature K-Means K-Median

Computation speed Fast Slower

Best for Numerical, spherical clusters Skewed data, robust clustering

Use case Large numeric datasets Datasets with extreme values

19. Elbow Method

The Elbow Method is a heuristic technique used in clustering to determine the optimal
number of clusters (K) for algorithms like K-Means or K-Median.

It helps avoid arbitrary selection of K and ensures that clusters are meaningful.

Concept
• Clustering algorithms minimize the within-cluster variation (also called Within-
Cluster Sum of Squares, WCSS).

• As K increases:

o WCSS decreases (clusters are smaller and tighter)

o The reduction in WCSS slows down after a certain point

• The point where the rate of decrease sharply changes direction forms an “elbow”,
which indicates the optimal number of clusters.

Steps to Apply the Elbow Method

1. Run clustering algorithm (e.g., K-Means) for a range of K values (e.g., 1–10).

2. Compute WCSS for each K:

𝑲

𝑾𝑪𝑺𝑺 = ∑ ∑ ∣∣ 𝒙 − 𝝁𝒊 ∣∣𝟐
𝒙∈𝑪𝒊
𝒊=𝟏

3. Plot K (x-axis) vs WCSS (y-axis).

4. Identify the elbow point (where the curve starts to flatten).

5. Choose K at this elbow point as the optimal number of clusters.

31
 MACHINE LEARNING

Advantages
• Simple and intuitive

• Provides visual insight into cluster selection

• Widely used in K-Means and K-Median clustering

Disadvantages
• Elbow may not be clearly visible in some datasets

• Subjective decision (depends on visual interpretation)

• Not suitable for high-dimensional data

Applications
• Determining K in K-Means clustering

• Image segmentation

• Customer segmentation

• Market analysis

20. Artificial Neural Methods

Artificial Neural Networks (ANNs) are computational models inspired by the human brain’s
neural network.

They consist of interconnected processing units called neurons that can learn patterns
from data. ANNs are widely used in supervised, unsupervised, and reinforcement learning.

Key Components of an ANN

1. Neurons (Nodes)

o Basic processing units

o Receives input, processes it, and passes output

32
 MACHINE LEARNING

2. Weights

o Connection strengths between neurons

o Determines the influence of input on output

3. Bias

o Additional parameter added to the weighted sum

o Helps shift the activation function

4. Activation Function

o Determines neuron output based on input

o Introduces non-linearity into the network

Popular Activation Functions

Architecture of ANN
1. Input Layer

o Accepts raw input features

o Number of neurons = number of input features

2. Hidden Layers

o Perform intermediate computations

33
 MACHINE LEARNING

o Number of layers and neurons depends on problem complexity

3. Output Layer

o Produces final result

o Number of neurons depends on output type (e.g., 1 for regression, N for N-

class classification)

Working Principle
1. Each input is multiplied by a weight.

2. Weighted sum is calculated and bias is added.

3. Sum is passed through an activation function.

4. Output is forwarded to the next layer (or final output).

5. Learning occurs by adjusting weights using training data and backpropagation to

minimize error.

Learning Process
• Supervised Learning: ANN learns from labeled data.

• Unsupervised Learning: ANN identifies patterns without labels (e.g., self-organizing

maps).

• Reinforcement Learning: ANN learns based on feedback from environment.

Training Algorithm
• Forward Pass: Input → Hidden layers → Output

• Error Calculation: Compare predicted output with actual output

• Backpropagation: Adjust weights to minimize error using gradient descent

Advantages of ANN
• Can model complex, non-linear relationships

• Adaptive and can learn from data

• Handles large datasets and high-dimensional data

34
 MACHINE LEARNING

• Tolerant to noise and missing data

Disadvantages of ANN
• Requires large training datasets

• Computationally expensive

• Can overfit if network is too complex

• Difficult to interpret (black-box nature)

Applications of ANN
• Pattern Recognition: Handwriting, speech, face recognition

• Medical Diagnosis: Cancer detection, disease prediction

• Image Processing: Object detection, image classification

• Financial Forecasting: Stock price prediction, credit scoring

• Robotics and Control Systems: Autonomous navigation

21. Feed Forward Neural Networks (FFNN)

A Feed Forward Neural Network (FFNN) is a type of Artificial Neural Network where the data
flows only in one direction — from the input layer → hidden layers → output layer.

• No cycles or feedback connections are present.

• FFNN is mainly used for supervised learning tasks like classification and regression.

Architecture
1. Input Layer

o Accepts raw features from the dataset.

o Number of neurons = number of input features.

2. Hidden Layer(s)

o Intermediate layer(s) where computations occur.

35
 MACHINE LEARNING

o Can be single-layer or multi-layer.

o Activation functions like ReLU, Tanh, or Sigmoid introduce non-linearity.

3. Output Layer

o Produces final prediction or classification.

o Number of neurons depends on the number of output classes or regression

targets.

o Softmax activation is commonly used for multi-class classification.

Working Principle
1. Forward Pass:

o Each input is multiplied by a weight, and bias is added.

o Result is passed through an activation function.

o Output moves layer by layer to the final output.

2. Error Calculation:

o Compute the difference between predicted and actual outputs using a loss
function (e.g., MSE or cross-entropy).

3. Weight Update (Learning):

o Weights are adjusted using backpropagation and gradient descent to

minimize error.

Key Features
• Unidirectional flow: Data flows from input to output, no loops.

• Non-linear mapping: Can model complex relationships using hidden layers.

• Layer flexibility: Number of hidden layers and neurons can be adjusted.

Advantages
• Simple and easy to implement

36
 MACHINE LEARNING

• Suitable for a wide range of problems (regression & classification)

• Can approximate any continuous function with enough hidden neurons (Universal
Approximation Theorem)

Disadvantages
• No memory of previous inputs → not suitable for sequential data

• May require large datasets for effective training

• Prone to overfitting if network is too complex

Applications
• Handwritten digit recognition (MNIST)

• Image classification

• Medical diagnosis (e.g., disease prediction)

• Stock price prediction

22. Feedback Ward Methods (Recurrent Neural Networks – RNN)

Feedback Ward Methods, commonly known as Recurrent Neural Networks (RNNs), are a
type of Artificial Neural Network where connections between neurons form cycles or loops,
allowing information to persist over time.

• Unlike Feed Forward Networks, RNNs have memory, meaning the output depends
on current input and previous inputs.

• They are mainly used for sequential or time-dependent data.

Architecture
1. Input Layer

o Accepts sequential or time-series data.

2. Hidden Layer(s) with Feedback

o Contains neurons whose output is fed back to the same layer or next layer.

37
 MACHINE LEARNING

o This feedback loop allows the network to remember previous states.

3. Output Layer

o Produces predictions based on current input and accumulated memory.

Methods of Feedback Ward (Recurrent Neural Networks)

1. Simple Recurrent Network (Elman or Jordan Networks)

Description

• Basic RNN architecture

• Hidden layer has feedback connections to itself (Elman) or to input layer (Jordan)

• Stores short-term memory from previous time steps

Characteristics

• Suitable for short sequences

• Trained with Backpropagation Through Time (BPTT)

Limitations

• Suffers from vanishing/exploding gradient problem

• Cannot capture long-term dependencies

2. Long Short-Term Memory (LSTM)

Description

• Special type of RNN designed to overcome vanishing gradient

• Uses memory cells and gates (input, forget, output) to control information flow

Components

• Input gate: Controls input to memory cell

• Forget gate: Decides what to discard from memory

• Output gate: Controls what to output from memory

Advantages

38
 MACHINE LEARNING

• Handles long-term dependencies effectively

• Robust for long sequences

Applications

• Language modeling, speech recognition, machine translation

3. Gated Recurrent Unit (GRU)

Description

• Simplified version of LSTM with two gates: update and reset

• Combines input and forget gates into one

• Fewer parameters → faster to train than LSTM

Advantages

• Effective for sequences

• Less computationally intensive than LSTM

4. Bidirectional RNN (BRNN)

Description

• Processes sequences in both forward and backward directions

• Output depends on past and future context

Applications

• Speech recognition, NLP tasks (e.g., POS tagging)

• Any application where future context helps prediction

5. Training Methods

Backpropagation Through Time (BPTT)

• Extends standard backpropagation to sequential data

• Gradients are propagated back through all time steps

39
 MACHINE LEARNING

• Used to update weights in recurrent connections

Truncated BPTT

• Limits the number of time steps for backpropagation

• Reduces computation and avoids some gradient issues

23. Feature Reduction Methods

Feature Reduction is the process of reducing the number of input variables (features) in a
dataset while preserving as much relevant information as possible.

• Also called dimensionality reduction

• Improves model performance, reduces computation time, and helps in

visualization.

Why Feature Reduction is Important

1. High-dimensional data can cause the curse of dimensionality.

2. Reduces overfitting by removing irrelevant/redundant features.

3. Improves learning speed of algorithms.

4. Helps in visualizing data in 2D or 3D.

Types of Feature Reduction Methods

Feature reduction can be broadly classified into two types:

1. Feature Selection

• Selects a subset of the original features without changing them.

• Goal: Remove irrelevant or redundant features.

Methods of Feature Selection

1. Filter Methods

o Evaluate features based on statistical measures.

o Examples: Correlation, Chi-square test, Mutual Information, ANOVA.

40
 MACHINE LEARNING

o Independent of the learning algorithm.

2. Wrapper Methods

o Use machine learning models to evaluate subsets of features.

o Examples: Forward Selection, Backward Elimination, Recursive Feature

Elimination (RFE).

o Computationally intensive.

3. Embedded Methods

o Feature selection is done during model training.

o Examples: LASSO (L1 regularization), Decision Tree feature importance.

o Balances accuracy and computation.

Feature Extraction
• Transforms original features into a lower-dimensional space.

• Goal: Reduce dimensionality while retaining maximum information.

Popular Feature Extraction Techniques

1. Principal Component Analysis (PCA)

o Linear technique that projects data onto orthogonal axes (principal

components).

o Components capture maximum variance.

o Reduces dimensionality while retaining information.

2. Linear Discriminant Analysis (LDA)

o Supervised method for classification tasks.

o Maximizes between-class variance and minimizes within-class variance.

o Reduces features while improving class separability.

3. Independent Component Analysis (ICA)

o Finds statistically independent components from data.

o Often used in signal processing.

41
 MACHINE LEARNING

4. t-Distributed Stochastic Neighbor Embedding (t-SNE)

o Non-linear technique for visualizing high-dimensional data in 2D or 3D.

o Preserves local structure of data.

24. Decision Tree

A Decision Tree is a supervised learning algorithm used for classification and regression
tasks.

• It represents decisions and their possible consequences in a tree-like structure.

• The tree consists of nodes, branches, and leaves, making it easy to interpret and
visualize.

Components of a Decision Tree

1. Root Node

o The topmost node

o Represents the entire dataset and is split into sub-nodes based on a feature.

2. Internal/Decision Nodes

o Represent tests on features

o Each node splits the data based on a condition.

3. Leaf/Terminal Nodes

o Represent the final output/class label

o No further splitting occurs here.

4. Branches

o Connect nodes and represent outcomes of a test

Working Principle
1. Select the best feature to split the data using a splitting criterion.

2. Split the dataset into subsets based on the chosen feature.

42
 MACHINE LEARNING

3. Repeat the process recursively for each subset (recursive partitioning).

4. Stop when:

o All data in a node belong to the same class, or

o Maximum tree depth is reached, or

o No further information gain is possible.

Splitting Criteria
1. Information Gain (IG)

o Measures the reduction in entropy after a split.

o Entropy formula:

𝑬𝒏𝒕𝒓𝒐𝒑𝒚(𝑺) = −∑𝒑𝒊 𝐥𝐨𝐠 𝟐 𝒑𝒊

• Higher information gain → better feature for splitting

2. Gini Index

o Measures impurity of a node

𝑮𝒊𝒏𝒊 = 𝟏 − ∑𝒑𝟐𝒊

• Lower Gini → better split

3. Chi-Square Test

o Statistical test to determine if feature is significantly associated with target

Types of Decision Trees

1. Classification Tree

o Target variable is categorical

o Example: Predicting whether a customer will buy a product (Yes/No)

2. Regression Tree

o Target variable is continuous

43
 MACHINE LEARNING

o Example: Predicting house prices

Advantages
• Easy to understand and interpret

• Handles both numerical and categorical data

• Requires little data preprocessing

• Can model non-linear relationships

• Can perform feature selection automatically

Disadvantages
• Prone to overfitting

• Sensitive to small changes in data

• Bias towards features with more levels

• May create complex trees if dataset is large

Applications
• Medical diagnosis: Predict disease based on symptoms

• Customer analytics: Predict purchase behavior

• Finance: Credit risk analysis

• Manufacturing: Fault detection

• Decision-making systems: Policy or strategy selection

25. Distance-Based Methods

Distance-based methods are techniques in machine learning and data mining that rely on
measuring the distance or similarity between data points to make decisions.

• The closer two points are, the more similar they are considered.

44
 MACHINE LEARNING

• Widely used in clustering, classification, and anomaly detection.

Key Concept
• Distance-based methods compute a distance metric between points.

• Based on distance, the algorithm:

o Groups similar points (clustering)

o Predicts labels (classification)

o Identifies outliers (anomaly detection)

Common Distance Measures

1. Euclidean Distance – Straight-line distance in multidimensional space

2. Manhattan Distance – Sum of absolute differences between coordinates

3. Cosine Similarity – Measures angle between vectors (high for similar direction)

4. Mahalanobis Distance – Considers correlations between features

Examples of Distance-Based Methods

1. K-Nearest Neighbors (KNN)

• Supervised learning algorithm

• Classifies a point based on majority label of K nearest neighbors

• Distance metric (usually Euclidean) determines “nearest” neighbors

2. K-Means Clustering

• Unsupervised learning algorithm

• Assigns points to clusters based on distance from cluster centroids

• Minimizes within-cluster sum of squared distances (WCSS)

45
 MACHINE LEARNING

3. Hierarchical Clustering

• Builds a dendrogram based on distance between points or clusters

• Can use single-linkage, complete-linkage, or average-linkage distances

4. DBSCAN (Density-Based Spatial Clustering)

• Groups points close together (density) and identifies outliers

• Distance metric is used to define neighborhood radius (ε)

5. Outlier Detection

• Points that are far from all other points based on distance are considered outliers

• Mahalanobis distance is commonly used for multivariate outlier detection

Advantages
• Simple and intuitive

• Works for both classification and clustering

• Applicable to numerical and vector data

• Flexible – choice of distance metric can be adapted to data

Disadvantages
• Performance depends on distance metric choice

• Sensitive to feature scaling → requires normalization

• High-dimensional data may reduce effectiveness (curse of dimensionality)

• Computationally expensive for large datasets

Applications
• Clustering: Customer segmentation, image segmentation

46
 MACHINE LEARNING

• Classification: KNN, anomaly detection

• Pattern recognition: Handwriting, face recognition

• Recommendation systems: Finding similar users/items

47