AI & Applications — Module 3: Machine Learning

STUDY NOTES

INTRODUCTION TO AI AND
APPLICATIONS
BETC105 / BETC205

MODULE 3: MACHINE LEARNING

Reference Textbook:
Reema Thareja, Artificial Intelligence: Beyond Classical AI, Pearson Education, 2023.

Prepared for VTU Curriculum (BOS — CS/IS)

Compiled as comprehensive, self-contained student study material

Page 1 of 43
 AI & Applications — Module 3: Machine Learning

Table of Contents
TOC \h \o "1-3"

Page 2 of 43
 AI & Applications — Module 3: Machine Learning

How to Use These Notes

These notes are built from the Module 3 (Machine Learning) presentation of the VTU BETC105/205 –
Introduction to AI and Applications syllabus. They are designed to be completely self-contained: you should not
need to refer back to the original slides to understand any concept.

• Sections follow the exact order of the original presentation outline for Module 3.
• Every important definition is highlighted in a shaded definition box.
• Diagrams, flowcharts and comparison tables are placed immediately next to the text they illustrate.
• Callout boxes mark Exam Points, Tips, and things to be careful about (Cautions).
• Each major section ends with a short 'Key Takeaways' summary.
• A full Final Revision Summary appears at the end of the document, along with a formula/keyword cheat-
sheet.

NOTE: Syllabus Note

The original slide deck also lists Module 1 (Introduction to AI) and Module 2 (Prompt Engineering) in its outline
pages, but the detailed slide content provided is entirely for Module 3: Machine Learning. These notes therefore
cover Module 3 in full depth.

Page 3 of 43
 AI & Applications — Module 3: Machine Learning

1. Machine Learning in AI — Overview

1.1 What is Artificial Intelligence? (Recap)

Definition — Artificial Intelligence (AI): A field of computer science focused on creating machines that can
perform tasks typically requiring human intelligence, such as learning, reasoning, and problem-solving.

To understand how AI actually works, it helps to break it down into its key techniques and sub-domains, rather
than treating it as a single black box.

How AI Works
• AI combines three ingredients: large datasets, fast processing power, and intelligent algorithms.
• These three ingredients together allow systems to learn patterns from data and make deductions.
• By processing and analysing vast amounts of information, AI systems keep improving their performance —
without needing every rule to be explicitly programmed by a human.

AI as "Reverse-Engineering" Human Traits

A useful mental model is to think of AI as an attempt to reverse-engineer human traits:

• AI tries to mimic human traits such as reasoning, decision-making, and pattern recognition.
• The goal is to build machines that can learn, adapt, and solve problems autonomously by recognising
patterns in the data they are given.

EXAM POINT: Exam Point

Remember the three ingredients of AI: (1) Data, (2) Processing Power, (3) Algorithms. This is a commonly asked
short-answer question.

1.2 Techniques (Sub-Domains) in AI

AI is not one single technology — it is an umbrella term for several overlapping techniques. The presentation lists
six major techniques, plus two supporting technologies.

Page 4 of 43
 AI & Applications — Module 3: Machine Learning

Figure 1.1 — Sub-domains and supporting technologies of AI

1. Neural Networks
2. Machine Learning (ML)
3. Deep Learning (DL)
4. Natural Language Processing (NLP)
5. Computer Vision
6. Cognitive Computing
Additional technologies that enable and support AI:

1. Graphical Processing Units (GPUs)

2. Internet of Things (IoT)

1.2.1 Neural Networks

Definition — Neural Network: A type of AI system that works loosely like the human brain — a group of
connected 'neurons' (nodes) that pass information to each other to understand complex data and make decisions
or predictions.

Think of each node as a tiny decision-maker:

• A node takes an input, processes it, and passes an output to the next node.
• Example: a neuron that receives weather information (sunny/rainy) and helps decide whether to carry an
umbrella.
• Learning process: the network looks at data many times to find patterns — similar to a student practising a
concept repeatedly until it 'clicks'.

Page 5 of 43
 AI & Applications — Module 3: Machine Learning

Layers of a Neural Network

A neural network is organised into three main types of layers:

1. Input Layer — where data enters the network. Example: for predicting house prices, inputs could be size,
location, and number of rooms.
2. Hidden Layer(s) — where the network actually processes data using algorithms. Weights and biases are
applied to inputs to determine their importance. Example: the hidden layer combines size, location, and
rooms to work out their combined effect on price.
3. Output Layer — gives the final result or prediction. Example: the predicted house price.

Figure 1.2 — Basic structure of a neural network showing input, hidden and output layers

1.2.2 Machine Learning (as a technique inside AI)

Definition — Machine Learning (ML): A field of computer science that focuses on teaching machines to
learn from data and make decisions on their own, without needing explicit, step-by-step human instructions.

How Machine Learning Works:

1. Learning from Data — ML algorithms analyse large sets of data to find patterns (e.g., looking at thousands
of cat and dog images to learn the difference between them).
2. No Human Instructions Needed — instead of hand-coding rules, the machine is fed examples and learns
from them directly.
3. Improvement Over Time — the machine analyses its own errors and self-corrects, getting better with
practice.
4. Making Better Decisions — because ML automates decision-making, it can often analyse data and decide
more quickly and accurately than a human, e.g. in healthcare diagnosis.
Page 6 of 43
 AI & Applications — Module 3: Machine Learning

How ML differs from traditional Human-Written Instructions

Human Instructions (Traditional

Aspect Machine Learning
Programming)

Approach Humans write specific rules for every case Machine learns patterns directly from data

Write code: "If email contains 'buy now', Feed the machine past spam/non-spam
Example task
mark spam" emails; it learns the patterns itself

Improves automatically as more data is

Adaptability Rules must be manually updated
processed

Relationship Between AI, ML, DL and NLP

These four terms are often confused, but they exist in a nested, hierarchical relationship:

Figure 1.3 — AI is the broadest field; ML, DL and NLP are progressively more specialised sub-fields

• AI: the broad field of creating intelligent machines.

• ML: focuses specifically on teaching machines to learn from data.
• DL: a deeper, more powerful level of ML that uses multi-layered neural networks to process large datasets
— especially useful for tasks like image recognition.
• NLP: a branch of AI that lets machines understand, interpret and respond to human language.

1.2.3 Deep Learning (DL)

Definition — Deep Learning (DL): A powerful technique within Machine Learning that uses neural networks
with many layers to process and learn from data, especially good at handling complex patterns in very large
datasets.

Page 7 of 43
 AI & Applications — Module 3: Machine Learning

How Deep Learning Works — four key stages:

1. Neural Networks with Layers — a DL network has multiple layers (nodes/neurons). Each layer processes
data, learns patterns, and passes results to the next layer. Example: in image recognition, the first layer
detects edges, the next recognises shapes, and later layers identify whole objects.
2. Forward Propagation — data moves through each layer, getting processed and refined until it reaches the
output layer, producing the final result (e.g., "cat" or "dog").
3. Backward Propagation — once an output is generated, the model calculates the error (difference between
predicted and actual result). If wrong, the weights of the nodes are updated by sending the error backward
through the network.
4. Training the Model — the model repeats forward and backward propagation, adjusting weights and
improving accuracy with every iteration.

Types of Learning in Deep Learning

Type Description Example

Training a model to identify

Supervised Learning Model trained on labelled data (known outcomes) cats using images labelled
'cat' / 'not cat'

Grouping customers into

Model works with unlabelled data and finds structure segments based on purchase
Unsupervised Learning
on its own behaviour, with no
predefined labels

Applications of Deep Learning

• Image Recognition — identifying objects, faces and scenes (e.g., face recognition in smartphones).
• Speech Recognition — converting spoken audio into text (e.g., Siri, Google Assistant).

Concept Summary Table — Deep Learning

Concept Explanation Example

Deep learning in image recognition

A series of interconnected layers that
Neural Networks processes raw images through
process data in steps
multiple layers

Data moves through layers to generate an Input image processed layer by layer
Forward Propagation
output until the model outputs 'cat'

If a model misclassifies an image,

Adjusting weights based on errors to
Backward Propagation backward propagation updates the
improve accuracy
weights

Predicting house prices from size,

Supervised Learning Model is trained on labelled data
location, number of rooms

Clustering customers based on

Model learns from unlabelled data, finding
Unsupervised Learning buying habits with no predefined
hidden patterns
labels

Page 8 of 43
 AI & Applications — Module 3: Machine Learning

1.2.4 Natural Language Processing (NLP)

Definition — Natural Language Processing (NLP): A field of AI focused on enabling machines to read,
understand, and respond to human language in a way that is both meaningful and useful.

Applications of NLP:

• Chatbots and Virtual Assistants

• Text Translation
• Sentiment Analysis
• Text Summarisation
How NLP Works — three stages:

1. Reading and Understanding — the machine breaks sentences into meaningful units (words/phrases).
Example: understanding that "What's the weather like today?" is a question about weather.
2. Interpreting — the machine works out the meaning behind words considering context. Example: "cold"
could mean low temperature or emotionally distant, depending on context.
3. Responding — the machine replies in a natural way, in text or speech. Example: responding to "Tell me a
joke" with an actual joke.

1.2.5 Computer Vision

Definition — Computer Vision: A branch of AI that focuses on enabling machines to see, understand, and
interpret images and videos, similar to how humans do.

Applications of Computer Vision:

• Facial Recognition
• Autonomous Vehicles
• Retail Stores (e.g., automated checkout)
• Medicine (e.g., scan analysis)
• Financial Institutions (e.g., document verification)
How Computer Vision Works:

1. Breaking Down Images — an image is broken into smaller parts like shapes, colours, and textures.
Example: a cat image broken down into ear shape, eyes, whiskers.
2. Classifying and Learning — the machine uses patterns from many images to learn to classify objects.
3. Making Decisions — based on what it has learned, the machine makes decisions about new images it sees.

1.2.6 Cognitive Computing

Definition — Cognitive Computing: A subfield of AI designed to make machines think and respond in a
human-like way, by analysing text, speech, images, or objects to understand and interact with the world,
mimicking the human brain.

Applications: Virtual Assistants, Healthcare, Customer Service, Education.

Page 9 of 43
 AI & Applications — Module 3: Machine Learning

1.3 Supporting Technologies for AI

1.3.1 Graphical Processing Units (GPUs)
GPUs provide the raw computing power required for iterative processing and training of neural networks. They
handle large volumes of data far more efficiently than traditional CPUs, which makes them ideal for:

• Parallel Processing — GPUs can process many operations at once, speeding up model training.
• Faster Training of Neural Networks — GPUs help process data through multiple deep-learning layers
faster. Example: GPUs are used to train image-recognition or speech-recognition models across millions of
inputs.

1.3.2 Internet of Things (IoT) and AI for Data Analysis

Definition — Internet of Things (IoT): A network of connected devices (smartphones, wearables, home
appliances, industrial sensors) that generate massive amounts of data, much of which remains unprocessed or
under-analysed without AI.

AI and advanced algorithms automate the analysis of this IoT data, extracting useful insights:

• Data Analysis at Scale — AI can analyse data from millions of connected devices quickly and efficiently.
• Predicting Rare Events — AI models can spot patterns that predict rare events, such as equipment failure
in manufacturing.
• Understanding Complex Systems — AI helps make sense of complex systems such as smart cities or
healthcare networks by combining data from multiple sources.

TIP: Key Takeaways — Section 1

AI is built from six core techniques (Neural Networks, ML, DL, NLP, Computer Vision, Cognitive Computing)
supported by GPUs and IoT. ML teaches machines to learn from data without explicit rules; DL is a deeper form
of ML using multi-layer neural networks; NLP and Computer Vision let machines understand language and
images respectively. AI ⊃ ML ⊃ DL, with NLP as a specialised branch of AI that uses ML/DL techniques.

Page 10 of 43
 AI & Applications — Module 3: Machine Learning

2. The Machine Learning Model

2.1 Formal Definition of Machine Learning

Machine Learning involves teaching a computer program to improve its performance at a specific task through
experience (data). The most widely accepted formal definition was given by Professor Tom Mitchell:

Definition — Machine Learning (Mitchell's Definition): “A computer program is said to learn from
experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T,
as measured by P, improves with experience E.”

This definition breaks a Machine Learning model into three essential components:

1. Task (T)
2. Experience (E)
3. Performance (P)

2.1.1 Task (T)

This is the real-world problem we want the machine to solve. Task (T) defines what the machine is trying to learn
or predict.

Examples of tasks:

• Predicting sales of a product

• Classifying emails as spam or not spam
• Other common tasks: classification, regression, clustering, and recognition

2.1.2 Experience (E)

Experience refers to the data that the machine learns from. Just as humans learn from past experience, machines
learn from data — the more data a model is exposed to, the better it generally learns.

Example: to classify emails, you provide the model with a set of labelled emails (spam / non-spam). This set is its
experience (E).

Experience (E) can be gained through different types of learning:

• Supervised Learning (with labelled data)

• Unsupervised Learning (with unlabelled data)
• Reinforcement Learning (through trial and error, receiving rewards or penalties)

2.1.3 Performance (P)

Performance measures how well the model performs the task after learning — essentially, the accuracy or
effectiveness of its predictions.

Common metrics used to measure performance:

• Accuracy — how many predictions were correct.

Page 11 of 43
 AI & Applications — Module 3: Machine Learning

• Precision — how many predicted positive outcomes were actually positive.

• Recall — how many actual positive outcomes were correctly identified.
• F1 Score — the balance (harmonic mean) between precision and recall.
• Confusion Matrix — a table showing true positives, false positives, true negatives and false negatives.

Worked Example — House Price Prediction Model

Component Definition in this Example

Task (T) Predict the price of a house based on features like size, location, and number of rooms

Experience (E) Historical data on house sales (size, location, price)

How accurately the model predicts new house prices, evaluated using metrics like Mean
Performance (P)
Squared Error or R²

EXAM POINT: Exam Point

T, E, P is one of the most frequently asked theory questions. Always be ready to map a given scenario (e.g., email
spam filter, disease prediction) onto Task, Experience, and Performance.

2.2 Types of Machine Learning Algorithms

Based on how a model learns from data (its 'Experience'), Machine Learning algorithms are broadly grouped into
four categories:

Figure 2.1 — The four broad categories of Machine Learning algorithms

1. Supervised Machine Learning Algorithms

Page 12 of 43
 AI & Applications — Module 3: Machine Learning

◦ Classification Algorithm
◦ Regression
2. Unsupervised Learning
◦ Clustering
◦ Association Analysis
◦ Dimensionality Reduction
◦ Outlier Detection / Anomaly Detection
3. Semi-Supervised Learning
4. Reinforcement Learning (RL)
◦ Positive Reinforcement
◦ Negative Reinforcement

2.3 Supervised Machine Learning

Definition — Supervised Machine Learning: A type of learning where the machine learns from labelled data
— like a student learning from a teacher who provides both the questions (inputs) and the correct answers
(outputs).

Figure 2.2 — Supervised learning (labels known) vs. Unsupervised learning (no labels, machine discovers groups)

How Supervised Learning Works

1. Learning from Past Data — the algorithm is trained on labelled examples where both input and output are
provided; it learns the relationship between inputs (X) and outputs (Y). Example: a basket of labelled fruits
(apple, banana, orange) — the machine learns to associate features like colour, shape and size with the
correct label.
2. Mapping Function (Y = f(X)) — the relationship between input and output is expressed as a function. Once
learned, the model can predict output for new, unseen data. Example: input color=red, shape=round →
model predicts "apple".

Page 13 of 43
 AI & Applications — Module 3: Machine Learning

Worked Example — Classifying Fruits

1. Step 1: Train the model using labelled data — Input: fruit features (colour, shape, size); Output: correct
labels (apple, banana, orange).
2. Step 2: The machine learns the mapping function Y = f(X).
3. Step 3: Given a fruit with unknown features (colour = yellow, shape = elongated), it predicts "banana".
Supervised Learning is split into two broad categories (covered fully in Sections 3 and 4):

Category Output Type Example

Classification A category or label (discrete) Is an email spam or not spam?

Regression A continuous value Predict the price of a house

(Classification is covered in Section 4; Regression is covered in Section 3.)

2.4 Unsupervised Machine Learning

Definition — Unsupervised Learning: A type of machine learning where the model is trained on data that is
neither labelled nor classified. The machine has no predefined answers to learn from and must discover patterns
or relationships in the data on its own.

How Unsupervised Learning Works

1. No Labelled Data — the machine only has raw data and must find patterns itself. Example: given unlabeled
images of mangoes and oranges, the machine doesn't know what a "mango" is, but must group images by
similarities like colour or shape.
2. Finding Patterns and Grouping — the machine notices patterns (e.g., all mangoes are yellow, oranges are
orange) and groups images into clusters based on such features.
3. Clustering — the process of grouping similar data points together, without knowing the real-world name of
each group.

What Unsupervised Learning Cannot Do

CAUTION: Caution
Unsupervised learning can group similar data points, but it CANNOT assign real-world labels to those groups. It
can tell you that mangoes and oranges form two different clusters, but it cannot tell you which cluster is called
"mango".

Applications of Unsupervised Learning

• Customer Segmentation — grouping customers with similar purchasing behaviour for targeted marketing.
• Image Segmentation — automatically grouping pixels that form objects or regions in an image.
• Anomaly Detection — detecting outliers or unusual data points, useful for fraud detection or network
security.

2.4.1 Clustering (Overview — detailed in Section 5)

Clustering groups similar data points together into clusters so that data within the same cluster is similar, while
data in different clusters is distinct.

Page 14 of 43
 AI & Applications — Module 3: Machine Learning

• Example: a cell-phone company can use clustering to find where most of its customers live, to decide the
best locations to build new cell towers.
(A full, detailed treatment of clustering algorithms — Partitional, Hierarchical, and Density-Based — appears in
Section 5.)

2.4.2 Association Analysis

Definition — Association Analysis: A data-mining technique used to find relationships or patterns between
different items in large datasets, expressed as rules describing how items or events are associated.

How it works:

1. Finding Relationships — e.g., if customers frequently buy bread, they might also buy butter.
2. Creating Association Rules — rules take the form “If X happens, then Y happens.” Example: “If a
customer buys a laptop (X), they are likely to buy a mouse (Y).”
Applications:

• Retail and E-commerce — e.g., placing diapers and baby wipes near each other because they are often
bought together.
• Market Basket Analysis — e.g., “If a customer buys shampoo, they are likely to buy conditioner.”
• Recommender Systems — e.g., Amazon/Netflix suggestions: “Customers who bought this also bought...”

2.4.3 Dimensionality Reduction

Definition — Dimensionality Reduction: A technique used to simplify a dataset by reducing the number of
features (variables) without losing important information — especially useful when datasets have millions of
features.

Why it matters:

• Simplifies Data — too many features are overwhelming and computationally expensive.
• Reduces Complexity — fewer features make models faster to train and help avoid overfitting (where a
model is too complex and fails to generalise to new data).
• Improves Visualization — makes it possible to visualise the dataset in 2D or 3D.

2.4.4 Outlier Detection (Anomaly Detection)

Definition — Outlier Detection: A technique used to find rare or unusual events in a dataset that do not follow
the normal pattern. These unusual observations, called outliers, can indicate fraud, errors, or new trends.

How it works:

• Identifying Anomalies — e.g., a sudden large credit-card withdrawal in an unusual location could be
flagged as possible fraud.
• Clustering (KNN-based) — K-Nearest Neighbors can be used to detect anomalies by measuring the
distance between data points — if a point is far from its neighbours, it is considered an outlier.
Applications:

• Fraud Detection (banking/credit card systems)

Page 15 of 43
 AI & Applications — Module 3: Machine Learning

• Healthcare (detecting rare diseases or unusual symptoms)

• Manufacturing (detecting defects or malfunctioning machinery)

2.5 Semi-Supervised Learning

Definition — Semi-Supervised Learning: A type of machine learning that combines the strengths of
supervised and unsupervised learning. It uses both a small amount of labelled data and a large amount of
unlabelled data, improving model accuracy while requiring less labelled data than pure supervised learning.

Two Approaches to Semi-Supervised Learning

Approach 1: Supervised Model + Unsupervised Model

1. Step 1: Build a supervised model using a small amount of labelled data.
2. Step 2: Apply the trained model to a large amount of unlabelled data to generate predictions.
3. Step 3: Use the predicted labels from the unlabelled data to add more labelled data.
4. Step 4: Iterate this process to progressively improve the model's accuracy.
Example: a small dataset of images labelled "cat"/"dog" trains an initial model, which then predicts labels for a
much larger set of unlabelled images; those predicted labels expand the training set.

Approach 2: Unsupervised Clustering + Annotation

1. Step 1: Use unsupervised learning to group similar unlabelled data into clusters.
2. Step 2: Annotate/label these clusters based on the patterns observed.
3. Step 3: Use the newly labelled clusters to train the model.
Example: cluster unlabelled customer data by purchasing habits, then assign labels to each cluster and use that to
train the model.

2.6 Reinforcement Learning (RL)

Definition — Reinforcement Learning (RL): A type of machine learning where an agent learns how to make
decisions by interacting with an environment. Unlike supervised learning, RL involves trial and error, with the
goal of maximizing rewards and minimizing penalties.

Page 16 of 43
 AI & Applications — Module 3: Machine Learning

Figure 2.3 — The Agent–Environment interaction loop in Reinforcement Learning

How Reinforcement Learning Works

1. Agent–Environment Interaction — an agent interacts with its environment by taking actions and receiving
feedback as rewards or penalties. Example: in a game, the agent might be a robot, and the environment is the
game world.
2. Rewards and Penalties — rewards are given for correct actions, penalties for incorrect ones. The agent's
goal is to maximise total reward over time. Example: a robot that avoids fire and reaches a diamond earns a
reward; touching fire loses reward.
3. Trial and Error — the agent starts with no knowledge and learns by trying actions, observing results, and
adjusting its strategy over time.

Key Features of Reinforcement Learning

Feature Description Example

The initial state from which the agent begins

Input (State) The robot's starting point in the game
learning

The action the agent can take given the current

Output (Action) Move forward, turn left, or turn right
state

The agent learns by interacting with the

Repeating a rewarded action;
Training Process environment and adjusting behaviour based on
avoiding a penalised one
feedback

The agent's ultimate goal — find the path that Finding the best path to the diamond
Maximizing Reward
gives the highest total reward with fewest hurdles

Page 17 of 43
 AI & Applications — Module 3: Machine Learning

Reinforcement Learning vs. Supervised Learning

Aspect Supervised Learning Reinforcement Learning

Labelled data with correct answers already Agent's own experience via rewards
Source of Learning
provided & penalties

Train a model on a dataset where labels are A robot learns the best path to a
Example
known diamond through trial and error

Types of Reinforcement
Reinforcement refers to giving feedback to an agent based on its actions, to encourage or discourage certain
behaviours. There are two types:

Effect on
Type Definition Example
Behaviour

Strengthens
Increases behaviour by giving a A robot gets a reward for the behaviour,
Positive Reinforcement
reward for a correct action avoiding a fire in a game encouraging it
to be repeated

Strengthens
the behaviour
Increases behaviour by removing an A car avoids a penalty by by helping
Negative Reinforcement
unpleasant condition driving safely avoid
negative
consequences

Worked Example — Reinforcement Learning in a Game

Scenario: A robot needs to reach a diamond (reward) while avoiding fires (penalties) in a game.

• The robot starts at a random point and tries different paths.

• Right path = Reward (diamond); Wrong path = Penalty (fire).
• The robot learns over time by repeating the process and adjusting its behaviour to maximise rewards and
avoid penalties.
Final Goal: the robot learns the best path in the game, successfully reaching the diamond while maximising total
reward.

TIP: Key Takeaways — Section 2

A machine learning model is defined by Task (T), Experience (E), and Performance (P). ML algorithms fall into
four families: Supervised (Classification, Regression), Unsupervised (Clustering, Association, Dimensionality
Reduction, Outlier Detection), Semi-Supervised, and Reinforcement Learning (Positive/Negative reinforcement).
Supervised learning uses labelled data; unsupervised learning discovers structure in unlabelled data; RL learns by
trial-and-error interaction with an environment.

Page 18 of 43
 AI & Applications — Module 3: Machine Learning

3. Regression Analysis in Machine Learning

3.1 What Is Regression Analysis?

Definition — Regression Analysis: A statistical method used to study the relationship between a dependent
(target) variable and one or more independent (predictor) variables. Its primary goal is to predict the value of
the dependent variable based on the given predictors.

Figure 3.1 — Linear regression fits a best-fit line through data points

How Regression Analysis Works

1. Identifying Relationships — determine how changes in independent variables affect the dependent
variable. Example: how does advertising expenditure affect sales?
2. Predicting Continuous Values — unlike classification (which predicts discrete categories), regression
predicts continuous values such as price, salary, or temperature.

Steps in Regression Analysis

1. Data Collection — collect data for the independent and dependent variables.
2. Model Creation — create a mathematical equation defining the dependent variable as a function of the
independent variables, e.g. Y = f(X1, X2).
3. Finding the Best Fit — plot a regression line/curve to find the best fit through the data points, minimising the
vertical distance between points and the line.
4. Evaluating the Model — measure how well it predicts new, unseen data.

Page 19 of 43
 AI & Applications — Module 3: Machine Learning

3.2 Case Studies (Linear Equations in Practice)

Case Study 1 — Auto Fare Calculation
Problem: The cost of an auto fare depends on a fixed charge plus a per-kilometre rate.

Definition — Linear Equation: y = 11x + 30, where y is the total cost, x is the distance travelled (km), and 30
is the fixed charge.

For a 10 km trip: y = 11 × 10 + 30 = ₹140.

Case Study 2 — Monthly Rental Cost

Problem: A company's office rental cost is based on a fixed cost plus a per-employee charge.

Definition — Linear Equation: y = 10000x + 20000, where x is the number of employees and y is the total
monthly rental cost.

For 20 employees: y = 10000 × 20 + 20000 = ₹220000.

EXAM POINT: Exam Tip

These two case studies are classic numerical problems. Practice substituting different values of x to compute y
quickly — this type of question is commonly asked in exams.

3.3 Model Evaluation Metrics for Regression

To evaluate how good a regression model is, we focus on the prediction error using various metrics.

Key Metrics
• Root Mean Squared Error (RMSE) — measures the difference between observed and predicted values.
Lower RMSE indicates a better model. Formula: RMSE = √(Σ(observed − predicted)² / n).
• Adjusted R-Square (R²) — represents the proportion of variation in the data explained by the model.
Higher R² indicates a better model.

Evaluation Methods
• Train-Test Split — split data into 80% for training and 20% for testing to evaluate model performance.
• K-Fold Cross-Validation — (1) split the data into k subsets (e.g., 5 subsets for k = 5); (2) train the model
on k−1 subsets and test on the remaining subset; (3) repeat for all subsets and calculate the average
prediction error. The best model is the one with the lowest error.

3.4 Types of Regression

Regression analysis includes several models, each suited for specific types of data and relationships between
variables:

1. Linear Regression
2. Logistic Regression
3. Ridge Regression
4. Lasso Regression (Least Absolute Shrinkage and Selection Operator)

Page 20 of 43
 AI & Applications — Module 3: Machine Learning

5. Polynomial Regression
6. Stepwise Regression
7. ElasticNet Regression

3.4.1 Linear Regression

• Use Case: predicting a continuous dependent variable with a linear relationship to one or more independent
variables.
• Example: predicting house prices based on size, location, and number of rooms.
Definition — Linear Regression Equation: Y = b0 + b1X + e, where b0 is the intercept, b1 is the coefficient
of the predictor variable (X), and e is the error term. (Also written as Y = bX + C for a single predictor.)

• Advantages: simple, fast, and easy to understand.

• Limitations: can suffer from overfitting if data is too complex; for more than one predictor, the model
extends to Multiple Regression: Y = b0 + b1X1 + b2X2 + e.

3.4.2 Logistic Regression

• Use Case: predicting a binary outcome (yes/no, true/false) or the probability of an event occurring.
• Example: predicting whether an email is spam or not spam based on certain features.
• Mechanism: the output is transformed using the logit function (log-odds).
• Applications: categorical data classification tasks (e.g., disease diagnosis, fraud detection).

3.4.3 Ridge Regression

• Use Case: used when there is multicollinearity (high correlation between independent variables). It helps
prevent overfitting by shrinking coefficient sizes.
• How It Works: adds a penalty to the size of coefficients to reduce their impact and stabilise the model.
• Example: predicting sales where price, promotion, and distribution are highly correlated.

3.4.4 Lasso Regression

• Use Case: variable selection and regularisation in regression models.
• How It Works: reduces the coefficients of less important predictors to exactly zero, effectively eliminating
them from the model. Also known as L1 regularisation.
• Example: in a dataset with many features, Lasso can select the most important ones, improving model
efficiency.

3.4.5 Polynomial Regression

• Use Case: used when the relationship between independent and dependent variables is non-linear.
• Example: predicting temperature changes over time (seasonal variations follow a curve, not a straight line).
Definition — Polynomial Regression Equation: Y = b0 + b1X1 + b2X2² + ... + bnXnⁿ

• Best Fit: used when data cannot be fit by a straight line but a curve is needed.

Page 21 of 43
 AI & Applications — Module 3: Machine Learning

3.4.6 Stepwise Regression

• Use Case: builds the regression model by adding or removing variables step by step based on performance.
• Forward Selection — starts with no variables and adds them one at a time.
• Backward Elimination — starts with all variables and removes them step by step.
• Bidirectional Elimination — combines both approaches.

3.4.7 ElasticNet Regression

• Use Case: a combination of Ridge and Lasso regression, especially useful when there are many predictors
compared to observations.
• How It Works: combines the L1 penalty of Lasso and the L2 penalty of Ridge to create a more balanced
model.
• Example: used in Support Vector Machines (SVM) or document optimisation, where there are many features
but not enough data points.

Comparison Table — Types of Regression

Type Use Case Key Feature Example

Predict continuous data with Predicting house prices based

Linear Regression Simple, fast, straight-line fit
a linear relationship on size

Predict binary outcomes or Spam vs non-spam email

Logistic Regression Used for classification tasks
probabilities classification

Adds penalty to reduce Predicting sales with correlated

Ridge Regression Handle multicollinearity
overfitting features

Variable selection & Feature selection in a large

Lasso Regression Reduces coefficients to zero
regularisation dataset

Polynomial Predicting temperature

Non-linear relationships Fits a curve, not a line
Regression variations

Build models by Selecting predictors for sales

Stepwise Regression Automates variable selection
adding/removing variables forecasting

ElasticNet Used in SVM and document

Combines Ridge and Lasso Balances L1 and L2 penalties
Regression optimisation

TIP: Key Takeaways — Section 3

Regression predicts continuous values, unlike classification which predicts categories. Linear Regression is the
simplest and most common technique. Ridge/Lasso/ElasticNet add regularisation to prevent overfitting;
Polynomial Regression handles curved (non-linear) relationships; Stepwise Regression automates feature
selection. Model quality is judged using RMSE and R², validated with Train-Test Split or K-Fold Cross-
Validation.

Page 22 of 43
 AI & Applications — Module 3: Machine Learning

4. Classification Techniques

4.1 What Is Classification?

Definition — Classification: A type of machine learning task where the goal is to predict which category or
class an observation belongs to. The model is trained using labelled data, where each input is already tagged
with the correct class. Based on this, the model learns patterns and uses them to predict the class of new, unseen
data.

Example: in an email classification task, the model predicts whether an email is spam or not spam, based on
features like subject, sender, and content.

Overview of Key Classification Algorithms

Algorithm How It Works (Summary)

Splits data into branches based on feature values, forming a tree-like structure;
Decision Trees
each leaf gives a predicted class.

An ensemble of many decision trees; final prediction is decided by majority

Random Forest
vote.

Classifies a point based on the majority class among its 'k' nearest neighbours
K-Nearest Neighbors (KNN)
in feature space.

Finds the hyperplane that best separates data into different classes, maximising
Support Vector Machines (SVM)
the margin.

Uses Bayes' Theorem, assuming features are independent, to compute the class
Naïve Bayes
with the highest probability.

Logistic Regression Predicts the probability of a binary outcome (0/1) despite the name 'regression'.

4.2 K-Nearest Neighbors (KNN)

Definition — K-Nearest Neighbors (KNN): A supervised learning method that classifies data points based on
the similarity of nearby data points. It is used for both classification and regression tasks by finding the 'k'
nearest neighbours to a new data point and predicting based on their majority class (or average value, for
regression).

How KNN Works

1. Data Points and Labels — KNN requires labelled training data. It looks at the 'k' nearest neighbours and
assigns a class (or predicts a value) based on them.
2. Distance Measure — the distance between data points is usually calculated using the Euclidean distance
metric, which determines the closeness of the new data point to training points.

Steps in the KNN Algorithm

1. Choose 'k' Value — the number of nearest neighbours to consider (e.g., k = 3 means looking at the three
closest neighbours).

Page 23 of 43
 AI & Applications — Module 3: Machine Learning

2. Calculate Distance — measure the distance between the new data point and every point in the training set.
3. Find Nearest Neighbours — identify the 'k' closest data points.
4. Classify or Predict — for classification, assign the most common class among the k neighbours; for
regression, take the average of their values.

Choosing an Appropriate 'k'

k value Effect Example

A single neighbour decides the class,

Higher variance — very sensitive to
Small k (e.g., k = 1) which can be misleading if it's an
noise
outlier

Lower variance — more stable, but may More neighbours smooth predictions
Large k (e.g., k = 10)
miss small/rare patterns but can wash out rare patterns

Worked Example — Classifying Students Using KNN (k = 3)

Student Academic Score EC Score

A 8 7

B 6 5

C 9 8

D 5 4
New Student: Academic Score = 7, EC Score = 6. We want to classify this new student using k = 3.

Page 24 of 43
 AI & Applications — Module 3: Machine Learning

Figure 4.1 — KNN worked example: the new student (star) is classified by its 3 nearest neighbours

Step 1: Calculate Euclidean Distance

Definition — Euclidean Distance Formula: distance = √[(x2 − x1)² + (y2 − y1)²], where (x1, y1) is the new
student's data (7, 6) and (x2, y2) is each existing student's data.

Step 2: Sort the Distances (Increasing Order)

1. Distance to Student A: 1.41
2. Distance to Student B: 1.41
3. Distance to Student C: 2.83
4. Distance to Student D: 2.83

Step 3: Find the 3 Nearest Neighbours (k = 3)

• Student A (Distance: 1.41)
• Student B (Distance: 1.41)
• Student C (Distance: 2.83)

Step 4: Classify the New Student (Majority Vote)

• Student A belongs to Group 1 ("Outstanding")
• Student B belongs to Group 2 ("Sporty")
Page 25 of 43
 AI & Applications — Module 3: Machine Learning

• Student C belongs to Group 1 ("Outstanding")

Majority Class: Group 1 ("Outstanding") appears twice; Group 2 ("Sporty") appears once.

EXAM POINT: Result

Since Group 1 ("Outstanding") is the majority class among the 3 nearest neighbours, the new student is classified
as “Outstanding”.

Pros and Cons of KNN

Pros Cons

Simple and easy to understand; requires little training High prediction time for large datasets (compares to all
time training data)

Sensitive to data scaling — features on different scales

Makes no assumptions about data distribution
bias results; data needs standardising

Works for both classification and regression Memory intensive — stores all training data

Sensitive to outliers and noisy data; struggles with high-

Good for multi-class problems
dimensional data

4.3 Decision Trees

Definition — Decision Tree: A supervised learning algorithm used for classification and regression tasks. It
works by creating a tree-like structure that splits data based on different features to make predictions.

How Decision Trees Work

• Root Node — the starting point where the first split happens, using the feature that best separates the data.
• Decision Nodes — points where data is split further based on conditions.
• Leaf / Terminal Nodes — contain the final prediction/class (e.g., "Yes"/"No", "Cancerous"/"Benign").
• Branches — edges connecting nodes, representing decision rules (e.g., "Is Age > 30?").

Steps to Build a Decision Tree

1. Select a Feature to Split — choose the feature that best separates the data into classes.
2. Split the Data — divide data into two or more sub-nodes based on the chosen feature.
3. Repeat — keep splitting at each node using the most relevant feature until no further division is useful.
4. Assign Class to Each Leaf — once a leaf node is reached, assign the majority class of the data at that leaf.

Worked Example — Predicting Product Purchase

Person Age Income Bought Product

A 30 50K Yes

B 22 30K No

C 45 70K Yes

Page 26 of 43
 AI & Applications — Module 3: Machine Learning

Person Age Income Bought Product

D 60 20K No
Step 1 — Root Node: split based on Age. If Age > 40, predict Yes (Person C). If Age ≤ 40, go to the next node.

Step 2 — Decision Node: split based on Income for people aged ≤ 40. If Income > 40K, predict Yes (Person A);
if Income ≤ 40K, predict No (Person B).

Figure 4.2 — Decision tree for predicting product purchase based on Age and Income

Types of Decision Trees

Type When Used Example

Target variable is categorical Predicting whether an email is spam or

Classification Trees
("Yes"/"No") not

Regression Trees Target variable is continuous Predicting the price of a house

Key Terminology
• Root Node — the first node in the tree.
• Terminal (Leaf) Node — where the final decision/classification is made.
• Branches — edges connecting nodes, showing decisions/rules.
• Splitting — dividing data into sub-groups based on features.
• Parent Node — a node that splits into sub-nodes (has children).

Page 27 of 43
 AI & Applications — Module 3: Machine Learning

Advantages and Disadvantages of Decision Trees

Advantages Disadvantages

Easy to understand and implement — visual, interpretable Overfitting — can easily overfit, especially if the tree is
model very deep (solved by Pruning)

No need for data scaling; works with numerical and Instability — a small change in data can produce a
categorical data completely different tree

Bias toward features with more levels can dominate

Can handle missing data (some algorithms)
splitting (solved using Random Forests)

Can model complex, non-linear relationships

NOTE: Note — Pruning

Pruning removes branches that provide little predictive power, in order to avoid overfitting and to
simplify/generalise the tree.

Real-World Applications of Decision Trees

• Medical Diagnosis — predicting whether a tumour is cancerous or benign.
• Finance — predicting whether a loan will be approved or denied.
• Marketing — predicting if a customer will buy a product based on demographics.

4.4 Random Forest

Definition — Random Forest: An ensemble learning method used for both classification and regression tasks.
It combines multiple decision trees to improve overall performance and reduce the overfitting seen in individual
decision trees.

Figure 4.3 — Random Forest combines many decision trees and aggregates their votes

Page 28 of 43
 AI & Applications — Module 3: Machine Learning

Steps in Random Forest

1. Randomly sample K cases from the dataset for training each decision tree (bootstrap sampling).
2. Select m features from the p available features at each node (random feature selection).
3. Grow each tree as deep as possible, without pruning.
4. For each new data point, every tree produces a prediction.
5. Combine predictions via voting (classification) or averaging (regression) to decide the final output.

How Random Forest Differs from Bagging

In ordinary bagging, all features are considered at every split. In Random Forest, only a random subset of m
features is considered at each node — this prevents the trees from becoming too similar (correlated) to each other.

Advantages and Limitations

Advantages Limitations

Can still overfit on very noisy data, especially in

Handles missing data well
regression tasks

Resilient to overfitting even with noisy/imbalanced data Slower prediction time — many trees must vote

Works well with large, high-dimensional datasets Harder to interpret than a single decision tree

Flexible — used for both classification and regression

4.5 Naïve Bayes Classification

Definition — Naïve Bayes: A probabilistic machine learning algorithm based on Bayes' Theorem. It is simple,
fast, and widely used for classification tasks such as spam detection, sentiment analysis, and text classification.

Figure 4.4 — Naïve Bayes decision flow for spam classification

Bayes' Theorem
Definition — Bayes' Theorem: P(A|B) = [P(B|A) × P(A)] / P(B), where P(A|B) is the probability of A given B
(posterior), P(B|A) is the likelihood, P(A) is the prior probability, and P(B) is the evidence.

Page 29 of 43
 AI & Applications — Module 3: Machine Learning

Understanding Conditional Probability (Warm-Up Example)

What is the probability of drawing the Queen of Spades from a deck of cards, given the card is a Spade? There are
52 cards total, 13 Spades, and only 1 of them is the Queen. This gives the conditional probability of drawing the
Queen given that it is a Spade = 1/13.

The 'Naïve' Assumption

Naïve Bayes assumes that all features are independent of one another. In reality, features are often dependent —
but this simplifying ("naïve") assumption makes the model much faster and easier to implement.

• Example: in spam detection, the presence of each word is treated as independent, even though certain words
often appear together.

How Naïve Bayes Works

1. Given labelled data (e.g., spam / not spam emails), calculate the probability of each feature (word) occurring
in each class.
2. For a new email, compute the probability of it being spam or not spam based on its words, and classify it
into the class with the higher probability.

Worked Example — Spam Email Classification

Word Spam Count Non-Spam Count

offer 3 1
A new email contains the word “offer”. We want to classify it as Spam or Not Spam.

1. Step 1 — P(offer|Spam): "offer" appears in 3 of 5 spam emails → P(offer|Spam) = 3/5 = 0.6.

2. Step 2 — P(Spam): assume 60% of all emails are spam → P(Spam) = 0.6.
3. Step 3 — P(offer): "offer" appears in 4 of 10 total emails → P(offer) = 4/10 = 0.4.
4. Step 4 — Apply Bayes' Theorem: P(Spam|offer) = [P(offer|Spam) × P(Spam)] / P(offer) = (0.6 × 0.6) / 0.4 =
0.9.

EXAM POINT: Result

The probability of the email being Spam given that it contains the word “offer” is 0.9 (90%). Since this is high, the
email is classified as SPAM.

Laplace Correction
Definition — Zero Conditional Probability Problem: When a word never appears in a category, its
probability becomes zero, which would wrongly force the entire product of probabilities to zero.

Solution — Laplace Correction: add 1 to the count of each word so that probabilities never become exactly zero.
Example: if "offer" appears in 0 spam emails, without correction P(offer|Spam) = 0; with Laplace correction, we
add 1 to the count, avoiding the zero probability.

Page 30 of 43
 AI & Applications — Module 3: Machine Learning

Pros and Cons of Naïve Bayes

Pros Cons

Assumption of feature independence is rarely true in

Simple, fast, and works well with large datasets
real-world data

Zero Probability Problem (handled by Laplace

Works with categorical and continuous data
correction)

Handles missing data reasonably well

Works even with small datasets

Applications of Naïve Bayes

• Text Classification — e.g., spam detection.
• Sentiment Analysis — classifying tweets/reviews as positive, negative, or neutral.
• Recommendation Systems — predicting whether a user will like a product.
• Medical Diagnosis — classifying disease presence based on medical features.

4.6 Deep Learning and Neural Networks (Classification Context)

Neural Networks (NN) are inspired by the human brain, consisting of layers of interconnected neurons. Neurons
process input data and learn to make predictions or classifications by detecting patterns; the network adjusts its
internal weights to minimise error during training.

• Example: to recognise a dog in an image, the neural network learns to identify shapes, edges and colours by
processing the image through multiple layers of neurons.

Steps in Training a Neural Network

1. Input Layer — input data is fed to the first layer of neurons.
2. Processing in Hidden Layers — each layer processes data using weights and an activation function.
3. Output Layer — the final result is produced after all layers have processed the data.
4. Backpropagation — errors from the output layer are passed backward through the network, adjusting
weights for better future predictions.

Activation Functions
Purpose: transform the input into an output within a certain range (typically between 0 and 1, for binary
classification). Common activation functions: Sigmoid, Tanh, ReLU.

Types of Neural Networks

Type Description

The most basic type; information moves in one direction from input to
Feedforward Neural Network
output.

Convolutional Neural Networks (CNNs) Specialised for image recognition and processing.

Page 31 of 43
 AI & Applications — Module 3: Machine Learning

Type Description

Recurrent Neural Networks (RNNs) Used for sequence-based data, such as time series or speech.

Training a Neural Network — Gradient Descent

Gradient Descent is the most common optimisation technique; it minimises error by adjusting weights. Two
common types:

Type Description

Batch Gradient Descent Uses the entire dataset to update the weights in one go.

Stochastic Gradient Descent (SGD) Updates the weights after every individual data point.

Pros and Cons of Neural Networks

Pros Cons

Flexible — used for both regression and classification Complex and computationally expensive

Works well with complex, non-linear data (image Training can take a long time, especially for large
recognition, NLP) datasets

Scalable to large numbers of inputs and hidden layers Requires large amounts of data to perform well

Interpretability issues — often called a "black box"

Deep Learning: A Deeper Level of Neural Networks

Deep Learning is a subset of machine learning that specifically uses deep neural networks with multiple hidden
layers.

• Benefit: can automatically extract features from raw data, removing the need for manual feature engineering.
• Example: in facial recognition, early layers detect edges; later layers recognise more complex features like
eyes, nose, and mouth.

Worked Example — Fraud Detection with Deep Learning

Task: identify fraudulent transactions using transaction data (amount, location, time).

Figure 4.5 — Deep learning pipeline for detecting fraudulent transactions

Page 32 of 43
 AI & Applications — Module 3: Machine Learning

1. Input Layer — transaction details (amount, IP address, location, time) are fed into the network. Example: a
sudden large spend in an unfamiliar country is suspicious.
2. First Hidden Layer — processes transaction amount, learning basic patterns like “large transactions in
unfamiliar locations may indicate fraud”.
3. Second Hidden Layer — processes IP address, checking if it matches the user's expected location;
unexpected IP raises a red flag.
4. Third Hidden Layer — compares the user's usual location with the transaction's location; large mismatches
suggest a fraud trigger.
5. Output Layer — the network classifies the transaction as fraudulent or not fraudulent; if flagged, the bank
may freeze the account or block the transaction.
Why this works: the model learns from vast amounts of historical data to detect patterns humans might miss, and
it improves over time as more data is processed.

Applications of Neural Networks and Deep Learning

• Pattern Recognition — facial recognition, object detection, handwriting recognition.
• Anomaly Detection — identifying fraudulent activities or rare events.
• Time-Series Prediction — stock prices, weather forecasting.
• Natural Language Processing — sentiment analysis, machine translation, speech recognition.
• Recommendation Systems — suggesting products/content.
• Medical Diagnosis — identifying diseases from medical images or patient data.

4.7 Support Vector Machine (SVM)

Definition — Support Vector Machine (SVM): A popular supervised learning algorithm primarily used for
classification. It works by finding the best hyperplane that separates two classes of data points in an n-
dimensional space, where n is the number of features. The hyperplane is the decision boundary used to classify
new data points.

Key Concepts in SVM

• Hyperplane — a line (in 2D), a plane (in 3D), or a general decision boundary in higher dimensions that
separates data into two classes.
• Support Vectors — the data points closest to the hyperplane; these are critical in determining the optimal
hyperplane.
• Margin — the distance between the hyperplane and the closest points from each class. SVM tries to
maximise this margin, since a larger margin means better class separation and a more accurate model.

How Does SVM Work?

1. Linear Classification
For linearly separable data, SVM finds a hyperplane that maximises the margin between the two classes.
Example: given red and blue data points, SVM draws the line that maximises the distance (margin) between them.

Page 33 of 43
 AI & Applications — Module 3: Machine Learning

2. Non-Linear Classification — The Kernel Trick

When data is not linearly separable, SVM uses the kernel trick to map data into a higher-dimensional space,
where a separating hyperplane becomes possible.

Types of Kernels in SVM

• Linear Kernel — used when data is linearly separable; decision boundary is a straight line/hyperplane.
• Polynomial Kernel — used when the relationship is non-linear but can be represented as a polynomial.
• Radial Basis Function (RBF) Kernel — one of the most commonly used kernels for non-linear problems;
maps data to a higher-dimensional space, creating a curved decision boundary.

Worked Example — Batsman vs. Bowler Classification

We classify cricket players as Batsman or Bowler based on Runs and Wickets.

Category Data Points (Runs, Wickets)

Batsmen (250, 10), (300, 12), (400, 8)

Bowlers (50, 30), (60, 40), (80, 20)

New player: Runs = 350, Wickets = 20 (or in some variants, Wickets = 15) — we need to classify this player.

Figure 4.6 — SVM hyperplane separating batsmen from bowlers, with the new player classified

1. Step 1: Plot the data points — Runs on the X-axis, Wickets on the Y-axis. Batsmen and bowlers form two
visually distinct groups.

Page 34 of 43
 AI & Applications — Module 3: Machine Learning

2. Step 2: Find the Optimal Hyperplane — SVM computes the line that best separates the two classes while
maximising the margin (distance to the nearest support vectors).
3. Step 3: Classify the New Player — check which side of the hyperplane the new player's point falls on.

Tuning Parameters in SVM

• Cost Parameter (C) — controls the trade-off between maximising the margin and minimising classification
error. High C → narrower margin, fewer misclassifications (risk of overfitting). Low C → wider margin,
more misclassifications (risk of underfitting).
• Gamma (for RBF Kernel) — defines the influence of a single training example. Low gamma → far-
reaching influence; high gamma → local influence. High gamma risks overfitting; low gamma risks
underfitting.

Advantages and Disadvantages of SVM

Advantages Disadvantages

Works well with high-dimensional data Can be slow with large datasets

Effective for both linear and non-linear classification Does not perform well with overlapping classes

Memory efficient — uses only support vectors for

Sensitive to the choice of kernel and parameter tuning
classification

Applications of SVM
• Image Recognition — recognising patterns and objects in images.
• Text Classification — spam email classification, document categorisation.
• Face Detection — classifying pixel data as face or non-face.
• Bioinformatics — classifying protein sequences and gene expression data.

TIP: Key Takeaways — Section 4

Classification predicts discrete categories using algorithms such as KNN (majority vote among nearest
neighbours), Decision Trees (rule-based splits), Random Forest (ensemble of trees), Naïve Bayes (probability via
Bayes' Theorem, assuming feature independence), Neural Networks / Deep Learning (layered pattern learning with
forward & backward propagation), and SVM (maximum-margin hyperplane separation, extended to non-linear
data via the kernel trick).

Page 35 of 43
 AI & Applications — Module 3: Machine Learning

5. Clustering Techniques

5.1 Introduction to Clustering

Definition — Clustering: A machine learning technique used to group similar data points together based on
their features, such that data points within the same cluster are similar to each other, while data points in
different clusters are dissimilar.

Clustering helps identify meaningful patterns in data and is used across healthcare, business, and marketing
applications.

5.2 Types of Clustering Algorithms

There are three main families of clustering algorithms, each suited to different types of data:

1. Partitional Clustering
2. Hierarchical Clustering
3. Density-Based Clustering

Figure 5.1 — Visual comparison of Partitional, Hierarchical and Density-Based clustering

5.2.1 Partitional Clustering

Partitional clustering divides the dataset into non-overlapping groups (clusters), where each data point belongs to
exactly one cluster.

• K-Means — divides data into k clusters, where k is predefined by the user.

• K-Medoids — similar to K-Means, but chooses actual data points as the centre of each cluster (medoids),
rather than computed averages.
Advantages: works well when clusters are spherical in shape; scalable and efficient with large datasets.

Limitations: struggles with complex (non-spherical) shapes; cannot handle clusters of different densities.

5.2.2 Hierarchical Clustering

Hierarchical clustering builds a tree-like structure (a dendrogram) that represents the hierarchy of data clusters.

Page 36 of 43
 AI & Applications — Module 3: Machine Learning

• Agglomerative (Bottom-up) — starts with individual data points and progressively merges them into larger
clusters.
• Divisive (Top-down) — starts with all data points in one cluster and splits them into progressively smaller
clusters.
Advantages: shows relationships between data points at different levels; resulting clusters are easy to interpret.

Limitations: computationally expensive for large datasets; sensitive to noise and outliers.

5.2.3 Density-Based Clustering

Clusters are formed based on the density of data points in a region; this approach does not require you to specify
the number of clusters beforehand.

• DBSCAN — (Density-Based Spatial Clustering of Applications with Noise) forms clusters wherever data
points are close enough to each other, and explicitly handles outliers (points that don't belong to any cluster).
• OPTICS — (Ordering Points to Identify the Clustering Structure) similar to DBSCAN, but can handle
clusters of varying densities.
Advantages: works well with non-spherical shapes and outliers; no need to specify the number of clusters in
advance.

Limitations: not ideal for high-dimensional data; struggles with clusters of very different densities.

Comparison Table — Clustering Types

Clustering Type Description Advantages Limitations

Divides data into non- Works well with Struggles with

Partitional Clustering overlapping groups (e.g., spherical clusters; complex shapes or
K-Means) scalable with large data varying densities

Results are easy to

Computationally
Builds a tree-like structure interpret; shows
Hierarchical Clustering expensive; affected
showing relationships relationships at various
by noise and outliers
levels

Handles non-spherical
Struggles with high-
Density-Based Forms clusters based on clusters and outliers; no
dimensional data and
Clustering density (e.g., DBSCAN) need to specify cluster
varying densities
count

5.3 K-Means Algorithm — Detailed Walkthrough

Definition — K-Means: A popular unsupervised learning algorithm used for clustering. It groups data into k
clusters, where each cluster contains data points that are more similar to each other than to points in other
clusters.

How K-Means Works

1. Choose the Number of Clusters (k) — e.g., decide k = 2 or k = 3.
2. Initialize Centroids — randomly pick k points in the dataset as the initial centroids (cluster centres).

Page 37 of 43
 AI & Applications — Module 3: Machine Learning

3. Assign Points to Clusters — for each data point, calculate the Euclidean distance to each centroid and assign
the point to the nearest one.
4. Update Centroids — after all points are assigned, recalculate each cluster's centroid as the mean of all points
in that cluster.
5. Repeat — repeat assignment and updating until the centroids stop changing (convergence).

Figure 5.2 — The three key stages of K-Means: raw data → initial centroids → converged clusters

Worked Example — Customer Segmentation with K-Means (k = 2)

Customer Income Debt

A 50K 10K

B 40K 15K

C 70K 25K

D 90K 20K

E 60K 30K
We want to group these five customers into 2 clusters based on Income and Debt.

1. Update Centroids: Cluster 1 (near A) = average of A, B, E → new centroid at (55K, 15K). Cluster 2 (near D)
= average of C, D → new centroid at (80K, 22K).
2. Reassign Points to New Clusters: recalculate distances using the new centroids and reassign each customer
accordingly. Repeat the process until the centroids no longer change.

Properties of Good Clusters

• Intra-Cluster Similarity — data points within a cluster should be as similar as possible.
• Inter-Cluster Dissimilarity — data points in different clusters should be as different as possible.

K-Means Evaluation Metrics

• Inertia — measures the sum of squared distances between data points and their respective centroids. Lower
inertia indicates better clustering.
• Dunn Index — measures the ratio of inter-cluster distance to intra-cluster distance. Higher values indicate
better-defined clusters.

Page 38 of 43
 AI & Applications — Module 3: Machine Learning

Pros and Cons of K-Means

Pros Cons

Simple and fast — easy to implement and runs Needs a predefined 'k' (number of clusters must be
efficiently chosen in advance)

Scalable — works well with large datasets Sensitive to outliers, which can distort the centroids

Random initialisation — final clusters can vary

Works well for spherical clusters
depending on the starting centroids

Non-optimal for non-spherical clusters or clusters of

different shapes/densities

Real-World Applications of Clustering

• Customer Segmentation — grouping customers by buying behaviour for targeted marketing (e.g., high
spenders vs. low spenders).
• Image Segmentation — grouping pixels into segments for image processing.
• Recommendation Systems — suggesting products based on similar user preferences.
• Document Clustering — grouping similar documents for better organisation.
• Location-Based Decisions — e.g., a mobile company using clustering to find the best cities for new cell
towers based on customer density.
• Gene Sequence Analysis — grouping similar gene sequences to identify genetic relationships.

TIP: Key Takeaways — Section 5

Clustering is unsupervised — it groups similar data points without predefined labels. Three families exist:
Partitional (K-Means, K-Medoids), Hierarchical (Agglomerative/Divisive, visualised via dendrograms), and
Density-Based (DBSCAN, OPTICS). K-Means is the most widely used algorithm: choose k, initialise centroids,
assign points, update centroids, and repeat until convergence. Quality is judged using Inertia (lower is better) and
the Dunn Index (higher is better).

Page 39 of 43
 AI & Applications — Module 3: Machine Learning

Final Revision Summary — Module 3: Machine Learning

This section brings together the entire module in a single, quick-revision format. Use it the night before an exam
to refresh every key idea covered in Sections 1–5.

A. Big-Picture Concept Map

AI is the umbrella field; ML, DL and NLP are progressively specialised sub-fields within it.

• AI = the broad goal of building intelligent machines.

• ML = teaching machines to learn from data instead of hard-coded rules.
• DL = ML using deep, multi-layer neural networks for very complex patterns.
• NLP / Computer Vision / Cognitive Computing = specialised AI branches for language, images, and human-
like reasoning respectively.

B. The Machine Learning Model in One Line

Definition — ML Model (Mitchell): A program learns from Experience (E) to perform a Task (T), and we
judge success using a Performance measure (P).

C. Complete Map of ML Algorithm Types

Category Sub-Types Nature of Data

Labelled (input + correct output

Supervised Learning Classification, Regression
known)

Page 40 of 43
 AI & Applications — Module 3: Machine Learning

Category Sub-Types Nature of Data

Clustering, Association Analysis,

Unsupervised Learning Unlabelled
Dimensionality Reduction, Outlier Detection

Supervised+Unsupervised combo; Small labelled + large

Semi-Supervised Learning
Clustering+Annotation combo unlabelled

No dataset — learns via trial &

Reinforcement Learning Positive Reinforcement, Negative Reinforcement
error with an environment

D. Regression vs. Classification — The Core Distinction

Aspect Regression Classification

Output Type Continuous numeric value Discrete category / class label

Example Predicting house price Predicting spam vs. not spam

Linear, Logistic*, Ridge, Lasso, KNN, Decision Trees, Random Forest,

Key Algorithms
Polynomial, Stepwise, ElasticNet Naïve Bayes, SVM, Neural Networks
*Note: Logistic Regression predicts probabilities/binary outcomes, so despite its name it is used as a classification
technique.

E. Classification Algorithms — Quick Comparison

Algorithm Core Idea Best Suited For

Simple problems, small-to-medium

KNN Majority vote of k nearest neighbours
datasets

Interpretable models, mixed data

Decision Tree Rule-based splits forming a tree
types

Reducing overfitting, higher

Random Forest Ensemble/vote of many decision trees
accuracy

Bayes' Theorem with independence

Naïve Bayes Text classification, spam filtering
assumption

Maximum-margin hyperplane (with High-dimensional data, clear margin

SVM
kernel trick) problems

Neural Networks / Deep Layered weighted computations with Complex, non-linear patterns
Learning backpropagation (images, speech)

F. Clustering — Quick Comparison

Type Representative Algorithm Key Trait

Non-overlapping groups; needs

Partitional K-Means, K-Medoids
predefined k

Hierarchical Agglomerative, Divisive Tree/dendrogram of nested clusters

Page 41 of 43
 AI & Applications — Module 3: Machine Learning

Type Representative Algorithm Key Trait

No need to predefine k; handles outliers

Density-Based DBSCAN, OPTICS
and irregular shapes

G. Formula & Key-Term Cheat Sheet

Term / Formula Meaning

Y = b0 + b1X + e Simple Linear Regression equation

Y = b0 + b1X1 + b2X2 + e Multiple Linear Regression equation

Y = b0 + b1X1 + b2X2² + ... + bnXnⁿ Polynomial Regression equation

RMSE = √(Σ(observed − predicted)² / n) Root Mean Squared Error — lower is better

Proportion of variance explained by the model — higher is

R² (R-Square)
better

Euclidean Distance = √[(x2−x1)²+(y2−y1)²] Used in KNN and K-Means to measure closeness

P(A|B) = [P(B|A) × P(A)] / P(B) Bayes' Theorem — foundation of Naïve Bayes

Laplace Correction Adds 1 to counts to avoid zero probability in Naïve Bayes

Distance between the hyperplane and nearest support

Margin (SVM)
vectors; SVM maximises this

Sum of squared distances of points to their cluster centroid

Inertia (K-Means)
— lower is better

Ratio of inter-cluster to intra-cluster distance — higher is

Dunn Index
better

Random sampling with replacement, used in Random

Bootstrap Sampling
Forest / Bagging

Pruning Trimming a decision tree to reduce overfitting

Mapping data to higher dimensions so SVM can separate

Kernel Trick
non-linear data

Optimisation technique that adjusts weights to minimise

Gradient Descent
error

H. Frequently Confused Pairs

Concept 1 Concept 2 Key Difference

Classification Regression Predicts categories vs. continuous values

Bagging uses all features at every split;

Bagging Random Forest Random Forest uses a random subset of
features

Supervised Learning Reinforcement Learning Learns from labelled data vs. learns from trial-

Page 42 of 43
 AI & Applications — Module 3: Machine Learning

Concept 1 Concept 2 Key Difference

and-error rewards/penalties

Lasso Regression Ridge shrinks coefficients toward zero; Lasso

Ridge Regression can shrink them exactly to zero (feature
elimination)

Negative Reinforcement Positive = reward given; Negative =

Positive Reinforcement unpleasant condition removed (both
strengthen behaviour)

K-Means is unsupervised clustering; KNN is

K-Means KNN
supervised classification/regression

Partitional gives flat, non-overlapping groups;

Partitional Clustering Hierarchical Clustering
Hierarchical builds a nested tree (dendrogram)

I. Suggested Revision Checklist

• Can you state Mitchell's definition of Machine Learning and identify T, E, P in any given scenario?
• Can you list and briefly explain all four categories of ML algorithms?
• Can you write the equation for Linear Regression and solve a simple numeric example (e.g., auto fare, rental
cost)?
• Can you manually work through a KNN classification example using Euclidean distance?
• Can you draw a simple decision tree for a 2-feature dataset?
• Can you explain how Random Forest differs from a single Decision Tree and from Bagging?
• Can you apply Bayes' Theorem to a simple spam classification example, including Laplace correction?
• Can you explain forward and backward propagation in a neural network?
• Can you describe the SVM margin, support vectors, and the purpose of the kernel trick?
• Can you list the steps of the K-Means algorithm and compute one iteration of centroid updates?
• Can you compare Partitional, Hierarchical, and Density-Based clustering with one advantage/limitation
each?

EXAM POINT: Final Exam Tip

Numeric worked examples (KNN distance calculation, Naïve Bayes probability, auto-fare/rental regression, K-
Means centroid update) are very likely to appear as problem-solving questions. Practice each of these calculations
by hand at least once before the exam.

Page 43 of 43