Introduction to AI and Applications(25ET15)

Module-3: Artificial Intelligence Technologies

Machine Learning: Techniques in AI, Machine Learning Model, Regression
Analysis in Machine Learning, Classification Techniques, Clustering
Techniques, Naïve Bayes Classification, Neural Network, Support Vector
Machine (SVM).

3.1 Techniques in AI

Artificial Intelligence (AI)

Artificial Intelligence (AI) works by combining massive amounts of data with fast, iterative
processing and intelligent algorithms, enabling software to automatically learn patterns and
make decisions with minimal human intervention.

Concept
Building an AI system involves reverse-engineering human traits and capabilities into a
machine—essentially enabling computers to think, learn, and act like humans.

To understand how AI works in practice, it is divided into several sub-domains, each focusing
on a specific capability or approach. All sub-domains share a common foundation: processing
large datasets with intelligent algorithms to enable automatic learning.

1. Neural Networks
Neural Networks are designed to work similar to the human brain, where each
neuron (node) processes information and passes it to other neurons.
They help computers recognize relationships and patterns in data to derive meaning
from it.
• A Neural Network is made up of layers of nodes (neurons).
• Each neuron receives inputs, applies weights and biases, processes them, and sends the
output to the next layer.
• The network learns by making multiple passes through the data, adjusting weights
each time to reduce errors and improve accuracy.

Department of ISE 2025-2026 P a g e 1 | 40

 Introduction to AI and Applications(25ET15)

Structure of a Basic Neural Network

1. Input Layer
o The first layer where data enters the network.
o Each input node represents a feature or variable from the
dataset.
2. Hidden Layer(s)
o One or more layers where processing occurs.
o The system applies weights, biases, and activation
functions to inputs to detect patterns. This is where the
learning happens.
3. Output Layer
o The final layer that produces the result or prediction after
processing.
o Outputs can be in the form of a category (e.g., “spam” or
“not spam”) or a value (e.g., “price = ₹500”).
2. Machine Learning (ML)
Machine Learning is a branch of AI that enables computers to analyze data, identify
patterns, and make decisions without human intervention.

How It Works
• ML algorithms learn from experience
instead of following fixed instructions.
• They analyze data, compare results, and
adjust automatically to improve accuracy.
• The process repeats until the system
produces reliable outputs.
Features
• Learns and improves automatically.
• Makes data-driven decisions.
• Saves human time and effort.
• Forms the base for NLP, Deep Learning, and Computer Vision.

3. Deep Learning (DL)

Deep Learning is an advanced form of Machine Learning that teaches a computer to
process inputs through multiple layers to classify, infer, and predict outcomes accurately.

How It Works
• Uses deep neural networks with many hidden
layers.
• Each layer processes data from the previous one — this
is called forward propagation.
• If the output is inaccurate, the system adjusts the
weights through backward propagation to improve
performance.
• Learns complex patterns from large datasets using
powerful computing resources.

Department of ISE 2025-2026 P a g e 2 | 40

 Introduction to AI and Applications(25ET15)

Key Features
• Learns automatically from large amounts of data.
• Works with both labeled (supervised) and unlabeled (unsupervised) data.
• Continuously improves accuracy through repeated training.
Applications
• Image recognition
• Speech recognition
• Natural language processing
• Autonomous systems (e.g., self-driving cars)

4. Natural Language Processing (NLP)

Natural Language Processing (NLP) is a branch of AI that enables machines to read,
understand, interpret, and respond to human language.

The goal of NLP is to make computers communicate with humans in a natural and
meaningful way, just like people do with each other.

Key Functions
• Reading and processing text or speech.
• Understanding meaning and context.
• Responding appropriately in human language.

Applications
• Chatbots and virtual assistants (e.g., Siri, Alexa)
• Language translation (e.g., Google Translate)
• Speech recognition
• Sentiment analysis

5. Computer Vision
Computer Vision is a branch of AI that enables machines to see, analyze, and understand
images or videos just like humans do.

How It Works
• The system breaks down images into parts, studies each part, and learns patterns.
• It then classifies and interprets new images based on previous learning.

Applications
• Facial recognition for security and surveillance
• Autonomous vehicles (self-driving cars)
• Medical diagnosis using image analysis
• Retail and finance for tracking inventory and preventing fraud

6. Cognitive Computing
Cognitive Computing is a subfield of AI that aims to mimic the human brain by
processing text, speech, images, or objects to produce human-like responses.

Purpose: The goal is to create systems that can think, reason, and respond naturally,
allowing human-like interaction with machines.

Department of ISE 2025-2026 P a g e 3 | 40

 Introduction to AI and Applications(25ET15)

Applications
• Virtual assistants and chatbots
• Customer service automation
• Healthcare (understanding medical data)
• Business analytics (intelligent decision support)

Additional Technologies Supporting AI

Apart from core AI techniques, several supporting technologies enhance AI’s performance
and capabilities.

1. Graphical Processing Units (GPUs)

• GPUs provide the high computing power needed for iterative processing and
training neural networks.
• They enable AI systems to process large volumes of data quickly and efficiently.

2. Internet of Things (IoT)

• IoT devices generate massive amounts of real-time data from sensors and connected
systems.
• AI models help analyze and extract useful insights from this data.
• Combined AI–IoT systems can:
o Predict rare events
o Understand complex systems
o Optimize unique scenarios

3.2 Machine Learning Model

Definition (by Prof. Mitchell)

“A computer program is said to learn from experience (E) with respect to some class of tasks
(T) and performance measure (P), if its performance at tasks in T, as measured by P,
improves with experience E.”

Key Components of a Machine Learning Model

1. Task (T)
o The real-world problem to be solved using ML.
o Examples:
▪ Predicting product sales
▪ Classifying emails as spam or not spam
▪ Recognizing images or speech
o Technically includes classification, regression, clustering,
etc.
2. Experience (E)
o The data or knowledge gained by the algorithm during training.
o The model learns patterns from data through repeated analysis.
o Learning types: Supervised learning, Unsupervised learning, Reinforcement
learning
3. Performance (P)
o Measures how well the ML model performs the task.
o Evaluated using metrics such as: Accuracy, Precision, Recall, F1 Score,
Confusion Matrix
Department of ISE 2025-2026 P a g e 4 | 40
 Introduction to AI and Applications(25ET15)

Types of Machine Learning Algorithms

Machine Learning algorithms are broadly classified into Supervised and Unsupervised
learning.

1. Supervised Machine Learning Algorithms

Supervised learning works with a teacher or labeled data — the algorithm learns from input-
output pairs to make predictions on new data.
Concept:
If input = X and output = Y, then the model learns a mapping function:
𝑌 = 𝑓(𝑋)

This helps the model predict the output for any new input.
Example:
Training a model to identify fruits — after learning what apples and bananas look like, it can
correctly classify new fruit images.

Types of Supervised Learning

a) Classification
• A Classification Algorithm is a type of supervised machine learning algorithm that is
used to classify data into specific categories or groups.
• It predicts discrete (categorical) outcomes rather than continuous values.
• The algorithm learns from labeled training data, where both inputs and their
corresponding class labels are known.
• During the training phase, the model learns patterns and relationships in the data.
• In the testing or prediction phase, it uses what it learned to assign new data to one of
the predefined classes.
Example
• Identifying a fruit as either an apple or a banana based on its features (like color, shape,
and size).
Real-World Applications
• Medical imaging: Classify whether a tumor is benign or malignant.
• Speech recognition: Identify spoken words or commands.
• Handwriting recognition: Recognize characters or digits.
• Email filtering: Detect whether an email is spam or authentic.
• Credit scoring: Predict if a person is a good or risky borrower.
When to Use Classification
• When data can be tagged, categorized, or separated into distinct classes.
• Suitable for problems where the output is a label (e.g., “Yes/No”, “Spam/Not Spam”,
“Positive/Negative”).
b) Regression
• Regression is a type of supervised machine learning algorithm used to predict
continuous (real-valued) outcomes based on input data.
• Unlike classification algorithms, which predict discrete labels, regression models
predict numerical values.
Examples of Regression Applications
• Predicting the cost of a product
• Estimating the value of a stock
• Forecasting temperature changes
• Predicting power demand fluctuations

Department of ISE 2025-2026 P a g e 5 | 40

 Introduction to AI and Applications(25ET15)

How Regression Works

1. Training Phase:
o The model is trained using input data (features) and corresponding output values
(labels).
o The model adjusts its parameters to minimize prediction errors.
2. Prediction Phase:
o After training, the model can predict outcomes for new, unseen data.
3. Feedback:
o The accuracy of predictions is evaluated, and the model can be improved
through feedback.

Applications of Supervised Learning (Regression)

• Customer churn prediction: Identify customers likely to stop using a service.
• Insurance prediction: Estimate the probability of accidents or losses and calculate
insurance value.

2. Unsupervised Machine Learning Algorithms

• Unsupervised learning is a type of machine learning where the model is trained on data
that is neither classified nor labeled.
• The algorithm learns patterns, structures, and relationships from the data without any
guidance or prior training.
• The machine works on unlabeled and unclassified data and tries to find hidden patterns
or groupings within it.
• It groups or organizes information based on similarities, patterns, and differences.
• Unlike supervised learning, there are no predefined output labels or target values.

Example
• If images of mangoes and
oranges are given to the
algorithm without labels:
o Initially, the system does
not know what a mango or
orange looks like.
o It observes patterns and
groups similar images together
— one cluster for mangoes and
another for oranges.
o However, it cannot name
or label them as “mango” or
“orange” — it only recognizes
similarity.

Learning Mechanism
• The algorithm learns through observation by detecting structures and relationships in
the dataset.
• It automatically forms clusters or associations among data points.

Categories of Unsupervised Learning Algorithms

1. Clustering

Department of ISE 2025-2026 P a g e 6 | 40

 Introduction to AI and Applications(25ET15)

o Objective: Discover inherent groupings or hidden patterns in data.

o Finds similarities and relationships in the underlying dataset.
o Examples & Applications:
▪ Grouping customers by purchasing behavior.
▪ Deciding optimal cell tower locations based on user density.
▪ Gene sequence analysis, market research, and object recognition.

2. Association Analysis (Association Mining)

o Objective: Discover rules that describe relationships among items in large
datasets.
o Example: “Customers who buy product X also tend to buy product Y.”
o Applications: Market basket analysis, retail recommendations.

3. Dimensionality Reduction
o Objective: Reduce the number of features or variables in a dataset while
retaining essential information.
o Helps in simplifying large datasets with millions of features.
o Techniques: Principal Component Analysis (PCA), Singular Value
Decomposition (SVD).
o Applications: Data visualization, noise reduction, and speeding up
computations.

4. Outlier Detection (Anomaly Detection)

o Objective: Identify rare or unusual observations that differ significantly from
the norm.
o Helps detect fraudulent transactions, defective products, or unusual behavior.
o Often uses clustering techniques like K-Nearest Neighbors (KNN) to detect
anomalies.

Applications of Unsupervised Learning

• Image recognition
• Speech-to-text conversion
• Natural Language Processing (NLP)
• Medical diagnosis: Predicting disease probabilities
• Retail analytics: Discovering frequently bought product combinations
• Fraud detection and cybersecurity

Characteristics
• Works without labeled data.
• Uses iterative and exploratory approaches, often involving deep learning.
• Suitable for complex tasks like pattern recognition and feature extraction.
• The algorithm explores data to find hidden relationships without prior clues.

3. Semi-Supervised Learning
• Semi-Supervised Learning (SSL) is a hybrid machine learning approach that combines
elements of both supervised and unsupervised learning.
• It uses a small amount of labeled data and a large amount of unlabeled data for training.
• The goal is to improve model accuracy when labeling large datasets is expensive or
time-consuming
Working Principle

Department of ISE 2025-2026 P a g e 7 | 40

 Introduction to AI and Applications(25ET15)

Semi-supervised learning can be implemented using two main approaches:

1. Approach 1: Supervised → Unsupervised Combination

o A supervised model is first built using a small labeled dataset.
o This model is then applied to large amounts of unlabeled data to generate
pseudo-labels.
o These newly labeled data points are used to retrain and refine the model.
o The process is iterative — repeated several times to improve accuracy.

2. Approach 2: Unsupervised → Supervised Combination

o The algorithm first applies unsupervised learning (e.g., clustering) to group
similar data samples.
o These clusters are annotated or labeled manually or automatically.
o The labeled clusters are then used to train a supervised learning model.

Advantages
• Improves accuracy compared to purely
unsupervised learning.
• Reduces labeling cost by requiring fewer labeled
examples.
• Utilizes large volumes of unlabeled data
effectively.
• Works well in scenarios where data labeling is
expensive or limited.

Applications
• Speech recognition: Using few transcribed samples with large amounts of raw audio.
• Web content classification: Labeling only a few pages and classifying millions.
• Medical diagnosis: Limited labeled patient data combined with large unlabeled
datasets.
• Image recognition: Annotating a few images while leveraging many unlabeled ones.
• Text classification: Categorizing documents or emails with minimal labeled data.

[Link] Learning (RL)

• Reinforcement Learning (RL) is a type of machine learning where an agent learns by
interacting with its environment through a system of rewards and penalties.
• The goal of the agent is to maximize cumulative rewards by learning the best actions to
take in a given [Link] is different from supervised and unsupervised learning.
• It is based on dynamic programming, where the agent improves its performance through
trial and error without human supervision.
• The agent receives:
o A reward for correct actions.
o A penalty for incorrect actions.
• Over time, the agent learns optimal behavior (policy) to achieve maximum reward.

Department of ISE 2025-2026 P a g e 8 | 40

 Introduction to AI and Applications(25ET15)

Example :Consider a robot (agent) in a maze:

o Its goal is to reach a diamond (reward) while avoiding
fire (penalty).
o Each step the robot takes gives feedback:
▪ Correct move → reward increases
▪ Wrong move → reward decreases
o After several trials, the robot learns the best path to reach the diamond with
minimal penalties.
Working of RL Algorithm
1. Input: Initial state from which the agent begins.
2. Output: List of possible actions or decisions for that problem.
3. Process:
o The agent takes an action based on its current state.
o The environment provides feedback (reward or punishment).
o The agent updates its knowledge to improve future decisions.
o This continues iteratively until the agent finds the optimal policy (maximum
total reward).

Mathematical View
• RL problems are modeled as Markov Decision Processes (MDPs) consisting of:
o States (S) – different situations the agent can be in.
o Actions (A) – choices available to the agent.
o Rewards (R) – feedback received after each action.
o Policy (π) – strategy used by the agent to decide actions.

Types of Reinforcement
1. Positive Reinforcement
o Strengthens a behavior by rewarding the agent when it performs correctly.
o Encourages the agent to repeat good behavior.
o Effect: Increases performance and maintains desired actions for a longer period.
o Example: Giving points to a robot for moving closer to a goal.
2. Negative Reinforcement
o Strengthens behavior by removing or avoiding negative outcomes.
o Encourages the agent to act in a way that avoids penalties or losses.
o Example: Stopping a penalty once the robot avoids an obstacle.

Applications of Reinforcement Learning

1. Robotics:
o Used for industrial automation, path planning, and robot control systems.
2. Machine Learning Systems:
o Enables machines to learn from experience and improve performance
autonomously.
3. Data Processing:
o Optimizes resource allocation and decision-making in complex data
environments.
4. Education & Training Systems:
o Used to build adaptive learning platforms that provide customized
instructions and feedback.
5. Game Playing:

Department of ISE 2025-2026 P a g e 9 | 40

 Introduction to AI and Applications(25ET15)

o RL is used in AI game agents (e.g., chess, Go) to learn strategies through

repeated play.
6. Self-Driving Cars:
o The car (agent) interacts with the road (environment) and learns through
feedback:
▪ Reward: Safe driving to destination.
▪ Penalty: Collisions or leaving the lane.

3.3 Regression Analysis in Machine Learning

Regression Analysis is a statistical method used to study the relationship between a dependent
(target) variable and one or more independent (predictor) variables. The primary goal of
regression analysis is to predict the value of the dependent variable based on the given
predictors.

Identifying Relationships: Regression analysis aims to

determine how changes in the independent variables affect
the dependent variable. Example: A company may want to
find out how advertising expenditure affects sales.

Predicting Continuous Values: Unlike classification, which

predicts discrete categories, regression predicts continuous
values like price, salary, or temperature. Example: Predicting
the price of a house based on its size, location, and age.

How Regression Analysis Works?

Regression analysis builds a mathematical equation that expresses the dependent variable y
as a function of one or more predictor variables x.

This equation is then used to predict the value of y when the values of the predictors are
known.

Linear Regression – The Simplest Form

Linear regression assumes a linear relationship between the outcome
(y) and the predictor variable(s) (x). Linear regression is the most
basic, simple and widely used technique for predicting values of a
continuous variable. It assumes that there exists a linear relationship
between the outcome and the predictor variables as shown in Fig.

Equation (Simple Linear Regression):

𝑦 = 𝑏0 + 𝑏1 𝑥 + 𝑒

Where:
• b₀ = intercept
• b₁ = coefficient (slope) of the predictor x
• e = residual error (difference between actual and predicted value)

Department of ISE 2025-2026 P a g e 10 | 40

 Introduction to AI and Applications(25ET15)

The regression line is fitted so that the residual error is minimized.

Method Used:
Ordinary Least Squares (OLS)
OLS selects coefficients 𝑏0 , 𝑏1 such that the sum of squared errors is minimum.

Multiple Linear Regression

When more predictors are involved, the equation becomes:
𝑦 = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥2 + ⋯ + 𝑒

• Predictors may have interaction effects

(e.g., one predictor enhances the effect of another).
• Selecting the best combination of predictors is important.
Common techniques:
✔ Best subset regression
✔ Stepwise regression
These help in choosing the optimal model with the lowest error.
Linear regression models work well with both continuous and categorical predictor variables.

When Linear Regression Is Not Suitable

Before applying linear regression, check if the data supports a linear relationship.
If the relationship is non-linear, other models are preferred:
• Polynomial regression
• Spline regression

For large datasets with correlated predictors:

Dimensionality Reduction Techniques
• Principal Component Regression (PCR)
• Partial Least Squares (PLS)
Penalized Regression Methods
Used to simplify complex models by penalizing too many variables:
• Ridge Regression
• Lasso Regression

Choosing the Best Regression Model

Multiple models can be compared, and the best one is chosen using statistical metrics.

Case Studies (Simple Linear Regression Examples)

1. Auto Fare Calculation
Fixed charge = ₹30
Variable charge = ₹11 per km
Equation:
𝑦 = 11𝑥 + 30

For 10 km:
𝑦 = 11(10) + 30 = 140

Department of ISE 2025-2026 P a g e 11 | 40

 Introduction to AI and Applications(25ET15)

2. Rental Space Cost

Fixed monthly rent = ₹20,000
Variable = ₹500 per employee per day
5 days/week → 20 days/month
Equation:
𝑦 = 10000𝑥 + 20000

For 20 employees:
𝑦 = 10000 × 20 + 20000 = 220000

3. Bakery Profit Calculation

Annual expenditure = ₹500,000
Monthly sales = ₹450,000
Regression equation:
𝑦 = 450000𝑥 − 500000

After 6 months:
𝑦 = 450000(6) − 500000 = 2200000

Model Evaluation Metrics

RMSE (Root Mean Squared Error)
• Measures the average prediction error.
• It is calculated as the average difference between the observed known values of the
outcome and the predicted value by the model.

• Lower RMSE → better model

Adjusted R-square
• Shows how much variation in the data is explained by the model.
• Adjusted for number of predictors.
• Higher Adjusted R-square→ better model

Train–Test Split
• Data is often split in 80:20 ratio
o 80% → Training
o 20% → Testing
• Test data checks predictive performance on unseen data.

k-Fold Cross-Validation
Useful even for small datasets.
Steps:
1. Split data into k subsets.
2. Use one subset as test data, train on remaining k–1 subsets.
3. Record the prediction error.
4. Repeat until each subset has been used as test data.
5. Compute the average of all errors → Cross-validation error.
Steps with Example (k = 5)

Department of ISE 2025-2026 P a g e 12 | 40

 Introduction to AI and Applications(25ET15)

Suppose you have a small dataset with 10 records:

Data = [1,2,3,4,5,6,7,8,9,10]
You choose k = 5, so the data will be split into 5 folds (subsets).
Each fold will have 2 records.
Step 1: Split the dataset into k subsets
For k = 5:
• Fold 1: [1, 2]
• Fold 2: [3, 4]
• Fold 3: [5, 6]
• Fold 4: [7, 8]
• Fold 5: [9, 10]
Step 2: Reserve one subset as test data and train on the rest
Example for Iteration 1:
• Test Set → Fold 1: [1, 2]
• Training Set → Folds 2, 3, 4, 5: [3–10]
Step 3: Test the model & record error
Train your machine learning model using training data
and then test it on Fold 1.
The result will give you Error 1.
Step 4: Repeat for all 5 folds
Iteration Test Fold Train Folds Error
1 Fold 1 2,3,4,5 E1
2 Fold 2 1,3,4,5 E2
3 Fold 3 1,2,4,5 E3
4 Fold 4 1,2,3,5 E4
5 Fold 5 1,2,3,4 E5
You get 5 different errors.
Step 5: Calculate average error (Cross-Validation Error)
𝐸1 + 𝐸2 + 𝐸3 + 𝐸4 + 𝐸5
CV Error =
5
This average is called the cross-validation error.
• It tells how well the model performs overall.
• The model with the lowest cross-validation error (e.g., RMSE) is considered the best
model.
Types of Regression
1. Linear Regression
• Definition:
Linear regression is the simplest regression technique used to
predict the value of a dependent variable (Y) based on the
value of an independent variable (X) when the relationship is
linear.
• General Equation:
𝑌 = 𝑏𝑋 + 𝐶
o Y → Dependent variable
o X → Independent variable
o b → Slope of the regression line
o C → Intercept
Department of ISE 2025-2026 P a g e 13 | 40
 Introduction to AI and Applications(25ET15)

• Key Points:
o Easy to model and evaluate.
o Useful when the relationship is not complex or data is limited.
o Prone to overfitting if too many features are used.
o Helps in detecting outliers.
o Fast and interpretable.

2. Logistic Regression
• Definition:
Logistic regression is used when the
dependent variable is binary, such as 0/1,
yes/no, true/false.
• Usage:
Used to calculate the probability of an
event (success or failure).
• How it Works:
o It applies a logit function to convert
linear output into a probability
between 0 and 1.
o Best suited for categorical data.
• Example Applications:
o Spam vs. not spam
o Disease detection (positive/negative)

3. Ridge Regression
• Definition:
Ridge regression is used to handle multicollinearity (high correlation between
predictors) and overfitting in multiple regression.
• How it Works:
o Adds a penalty term (L2 regularization) to shrink large coefficients.
o Reduces model complexity and variance.
• Key Point:
A small bias is introduced, but the overall model performance improves.

4. Lasso Regression (Least Absolute Shrinkage and Selection Operator)

• Definition:
Lasso regression performs variable selection and regularization.
• How it Works:
o Uses L1 regularization which applies a penalty that can shrink some
coefficients to zero, thereby eliminating unnecessary features.
• Advantages:
o Helps identify the most important variables.
o Reduces model complexity.
• Use Case:
Useful when the model contains many predictors but only a few are important.

5. Polynomial Regression
• Definition:
Polynomial regression fits a curved line to data that has a non-linear relationship
between the dependent and independent variables.

Department of ISE 2025-2026 P a g e 14 | 40

 Introduction to AI and Applications(25ET15)

• Equation:
𝑌 = 𝑏0 + 𝑏1 𝑋 + 𝑏2 𝑋 2 + ⋯ + 𝑏𝑛 𝑋 𝑛

• Uses:
o Works well for curvilinear data patterns.
o Typically applied with least squares fitting.
• Example:
Predicting growth curves, price trends, temperature
variations.

6. Stepwise Regression
• Definition:
A technique to build predictive models by automatically adding or removing
variables step-by-step.
• Methods:
o Forward Selection: Add variables one by one; stop when no improvement
occurs.
o Backward Elimination: Start with all variables; remove one at a time.
o Bidirectional Elimination: Combination of both methods.
• Benefit:
Helps identify an optimal set of predictors without manual selection.

7. ElasticNet Regression
• Definition:
ElasticNet is a combination of Ridge (L2) and Lasso (L1) regression.
• Key Points:
o Useful when the number of predictors is greater than the number of
observations.
o Handles correlated predictors effectively (grouping effect).
o Often used in:
▪ Support Vector Machines
▪ Document classification
▪ Metric learning
• Advantage:
Combines the feature selection ability of Lasso and the stability of Ridge.

3.4 Classification Techniques

Classification techniques are used to predict the category or class to which a given data instance
belongs. These algorithms estimate the probability that a data point falls into one of the
predefined classes based on training data.

K-Nearest Neighbour (K-NN) Algorithm

The K-Nearest Neighbour (K-NN) algorithm is a supervised learning classification
technique in which the class labels of training data are known, but the mapping function to
obtain the output is not explicitly learned.
Instead of learning a model, K-NN classifies a new data point by comparing it with existing
labeled samples and assigning it to the class to which it is most similar.

Department of ISE 2025-2026 P a g e 15 | 40

 Introduction to AI and Applications(25ET15)

Basic Idea
• Similar objects belong to the same class.
• For an unknown instance, the algorithm:
o Measures similarity with training samples.
o Assigns the class of the nearest neighbours.
• Classification is done using majority voting.

Nature of the Algorithm

• Supervised Learning – uses labeled training data
• Non-Parametric – no assumption about data distribution
• Lazy Learning Algorithm – no training phase; computation is done at prediction time
• Uses entire dataset during classification

Working of K-NN
1. Store all training samples.
2. Select the value of K (number of neighbours).
3. Calculate the distance between the new instance and each training sample.
4. Identify the K nearest neighbours.
5. Assign the class that appears most frequently among the K neighbours (mode).

Distance Measure
Commonly used distance:
• Euclidean Distance
𝑑 = √(𝑥2 − 𝑥1 )2 + (𝑦2 − 𝑦1 )2

Other distances: Manhattan, Minkowski, Hamming.

Choosing the Value of K

• Small K:
o High sensitivity to noise
o May overfit
• Large K:
o Reduces noise effect
o May ignore smaller important patterns

Example: K-Nearest Neighbour (K-NN) Algorithm

Given Data
Students are evaluated based on:
• Academic Score (AS)
• Extra-Curricular Score (EC)
(Both scores range from 1 to 10)
Student Name Academic Score (AS) EC Score (EC) Grade
Ria 8 8 Outstanding
Khushank 9 1 Academically Sound
Mehar 4 8 Sporty
Nagma 2 1 Below Average
A new student has the following scores:
• Academic Score = 8

Department of ISE 2025-2026 P a g e 16 | 40

 Introduction to AI and Applications(25ET15)

• EC Score = 2
Coordinates of New Student = (8, 2)

Step 1: Choose the Value of K

Let K = 1 (Nearest Neighbour).

Step 2: Euclidean Distance Formula

Distance = √(𝑥2 − 𝑥1 )2 + (𝑦2 − 𝑦1 )2

Step 3: Distance Calculation

1. Distance from Ria (Outstanding) (8, 8)
√(8 − 8)2 + (8 − 2)2 = √0 + 36 = √36 = 6.00

2. Distance from Khushank (Academically Sound) (9, 1)

√(9 − 8)2 + (1 − 2)2 = √1 + 1 = √2 = 1.41

3. Distance from Mehar (Sporty) (4, 8)

√(4 − 8)2 + (8 − 2)2 = √16 + 36 = √52 = 7.21

4. Distance from Nagma (Below Average) (2, 1)

√(2 − 8)2 + (1 − 2)2 = √36 + 1 = √37 = 6.08

Step 4: Summary of Distance Values

Category Distance
Outstanding 6.00
Academically Sound 1.41 (Minimum)
Sporty 7.21
Below Average 6.08

Step 5: Classification
Since the minimum distance is with Academically Sound, the new student is classified as:
Academically Sound

Advantages (Pros) of K-NN

1. Simple and Effective
K-NN is easy to understand and works effectively for basic classification problems.
2. Easy to Implement
The algorithm is straightforward to code, making it popular among data professionals.
3. No Prior Assumptions
Being a non-parametric algorithm, K-NN makes no assumptions about the
underlying data distribution, reducing bias.

Department of ISE 2025-2026 P a g e 17 | 40

 Introduction to AI and Applications(25ET15)

4. Supports Multi-Class Classification

K-NN works well when the dataset contains more than two classes.
5. Applicable to Classification and Regression
It can be used for both classification (majority voting) and regression (average of
neighbours).
6. Quick Training
Since there is no model training phase, training time is very low.

Disadvantages (Cons) of K-NN

1. Limited Predictive Power
The simplicity of the algorithm limits its ability to capture complex patterns.
2. High Prediction Time
Although training is fast, prediction is slow because distances must be calculated for
all training samples.
3. Ignores Potential Patterns
Important insights may be ignored, especially when dominant classes overshadow
minority classes.
4. Sensitive to Feature Scaling
Performance depends on data scale; features must be standardized or normalized.
5. Poor Performance with High Dimensional Data
Accuracy decreases when there are many independent variables (curse of
dimensionality).
6. Memory Intensive and Computationally Expensive
Requires storing the entire dataset and performing large distance calculations.
7. Not Suitable for Skewed Target Variables
Works poorly when class distributions are uneven.
8. Dependent on Value of K
o Small K → high variance and overfitting
o Large K → high bias and underfitting

Decision Tree
• A Decision Tree is a supervised learning technique used in data mining and machine
learning.
• It provides explicit rules for prediction and classification.
• Works well with heterogeneous data (numerical + categorical).
• Represents decisions in the form of a tree structure:
▪ Nodes → questions/conditions
▪ Edges/Branches → decision rules
▪ Leaves → final outcomes

Principle of Decision Trees (Working Steps)

1. Select the best variable that separates the classes → Root node
2. Split data based on this variable → Child nodes
3. Select the next best variable for further splitting
4. Repeat until:
o No further separation is possible, or
o Stopping criteria is met
• Uses forward selection → once a split is made, it cannot be reversed.
• Each individual finally belongs to exactly one leaf node.

Department of ISE 2025-2026 P a g e 18 | 40

 Introduction to AI and Applications(25ET15)

• After assignining all the leaf nodes we calculate : Error rate of tree = sum of error rates
of all leaf nodes.

Output Nature

• Regression Tree → Continuous output

For example, predict the price of a product or income of a customer (based on
his occupation, experi ence, etc.)
• Classification Tree → Categorical output
Example, to if the target value is ‘Yes’ or ‘No’- whether a customer will buy a
product or not.

Pruning the Tree – Notes

• When a decision tree becomes very deep, it may contain irrelevant or noisy leaf
nodes.
• Such trees may overfit the training data and perform poorly on new data.

Purpose of Pruning
• Removes unnecessary branches from the decision tree.
• Prevents the creation of very small nodes with no statistical significance.
• Improves generalization, accuracy, and simplicity of the model.

Good Decision Tree Algorithm

A decision tree algorithm is considered good if it:
• First creates the largest possible tree.
• Automatically prunes the tree after identifying the optimal pruning threshold.
• Eliminates nodes that do not contribute meaningfully to predictions.

Role of Cross-Validation
• Uses cross-validation technique to evaluate performance.
• Calculates error rates for all possible sub-trees.
• Compares these error rates and selects the best sub-tree with minimum error.

Binary Tree
• A tree in which each node has at most two child nodes.
• Most decision tree algorithms generate trees in binary form.

Example
• Information Gain (IG) at a node = −10 (loss of 10).
• Further split gives IG = +20.
• A simple decision tree stops splitting at −10.
• In pruning, overall gain is considered:
−10 + 20 = +10

• Since the total gain is positive, both leaf nodes are retained.

Department of ISE 2025-2026 P a g e 19 | 40

 Introduction to AI and Applications(25ET15)

Key Terminology

1. Root node: This is the node that performs the first split.
2. Terminal nodes/Leaves: These nodes predict the outcome.
3. Branches: They are depicted by arrows that connect nodes and shows the flow from question
to answer. Technically, a branch is a sub sec tion of entire tree
4. Splitting: This is the process of dividing a node into two or more sub nodes. In a decision
tree, splitting done until a user-defined stopping criteria is reached. For example, the
programmer may specify that the algorithm should stop once the number of items per node
becomes less than 30.
5. Decision node: This is a sub-node that splits into further sub-nodes
6. Terminal or Leaf node: It is a sub-node that does not split further.
7. Parent node: A node which splits into sub-nodes is called a parent node of the sub-nodes (or
child of the parent node).

Advantages of Decision Trees

1. Easy to implement and simple to construct.
2. Easy to understand and interpret; no statistical knowledge is required to read results.
3. The resulting model can be easily coded and deployed by programmers.
4. Fast execution when applied to new or unseen data.
5. Outliers or extreme values have minimal effect on decision trees.
6. Certain decision tree algorithms can handle missing data effectively.
7. Allows visualization of results and clearly represents factors involved in decision-
making.
8. Provides a clear picture of data structure and relationships among variables.
9. Helps in identifying the most significant variables in a dataset.
10. Closely resembles human decision-making compared to other regression and
classification methods.
11. Requires less data cleaning and preprocessing than many other techniques.
Department of ISE 2025-2026 P a g e 20 | 40
 Introduction to AI and Applications(25ET15)

12. Fairly robust to missing values in the dataset.

13. Can handle both numerical and categorical variables.
14. Uses a non-parametric approach, with no assumptions about data distribution or
classifier structure

Limitations of Decision Trees

1. Decision trees often find local optima rather than global optima.
2. Independent variables are evaluated sequentially, not simultaneously.
3. A small change in a single variable near the root can significantly alter the entire tree
structure.
4. Decision trees lack robustness.
o This can be reduced using resampling techniques (building multiple trees and
aggregating results).
o However, this leads to complex trees that are difficult to read and interpret.
o Overfitting can be reduced by tree pruning and by setting constraints on tree
size.
5. With each split, the number of remaining records decreases; therefore later splits rely
on very few records, resulting in low statistical power.
6. Due to forward variable selection and constant node splits, predictions may be less
accurate compared to other algorithms.
7. Overfitting is a major challenge in decision tree modeling.
o Without limiting tree growth, a decision tree may achieve 100% accuracy on
training data,
o Since, in the extreme case, each observation may form a separate leaf.

Random Forests
Random Forest is a powerful and versatile machine learning technique that, like decision
trees, can be used for both classification and regression problems. However, it generally
provides better performance than a single decision tree.

Why Random Forests Perform Better

Random forests:
• Reduce the effect of high dimensionality (large number of variables)
• Handle missing values effectively
• Are less sensitive to outliers
• Explore data more efficiently
• Improve accuracy by reducing overfitting

Random Forest vs Bagging

• Both bagging and random forests build multiple decision trees using random samples
of the training data.
• Bagging uses all predictors at each split.
• Random Forest selects only a random subset (m predictors) from all predictors (p
predictors) at each split.
• This de-correlates the trees, leading to better performance.

Department of ISE 2025-2026 P a g e 21 | 40

 Introduction to AI and Applications(25ET15)

Working Principle of Random Forest

Let:
• K = number of cases in the training dataset
• p = total number of input variables
• m = number of randomly selected variables (where m < p)
Steps:
1. If the training set contains K cases, a random sample of these K cases is selected.
2. From p input variables, choose m random variables at each node (m < p).
3. The best split among these m variables is selected to split the node.
4. Each decision tree is grown to the maximum possible extent (no pruning).
5. Predictions from all trees are combined:
o Regression: Final prediction = average of predictions of all trees.
o Classification: Each tree votes for a class; the majority vote is selected.

Simple Working Steps

• Step 1: Random samples
are selected from the dataset.
• Step 2: A decision tree is
built for each sample.
• Step 3: Each tree provides
a prediction.
• Step 4: Voting
(classification) or
averaging (regression) is
performed to get the final
prediction.

Advantages of Random Forest

1. Effectively estimates missing data.
2. Maintains high accuracy even when a large portion of data is missing.
3. Balances errors well in imbalanced datasets.
4. Handles large datasets with high dimensionality efficiently.

Limitations of Random Forest

1. May overfit noisy datasets, especially in regression problems.
2. Trees are grown fully without pruning, increasing complexity.
3. Slow prediction time due to construction of many trees and voting process.
4. Difficult to interpret when compared to a single decision tree.

Department of ISE 2025-2026 P a g e 22 | 40

 Introduction to AI and Applications(25ET15)

Clustering Techniques
Clustering is an unsupervised learning technique used to group
similar data objects into clusters.
• Objects in the same cluster → more similar
• Objects in different clusters → dissimilar
Clustering helps identify meaningfulness (insights) and
usefulness (preprocessing for further tasks).
Applications
• Healthcare: Group patients based on treatment
response
• Business: Cluster customers by purchase behaviour for
targeted marketing

Three main categories:

1. Partitional Clustering
2. Hierarchical Clustering
3. Density-Based Clustering

1. Partitional Clustering
Concept
• Divides data into non-overlapping clusters
• Each object → belongs to only one cluster
• User specifies k clusters
• Clusters formed iteratively
• Algorithms are nondeterministic (different results on each run)
Examples
• k-means
• k-medoids
Advantages
1. Works well for spherical-shaped clusters
2. Scalable in terms of computation
Limitations
1. Poor performance for complex or irregular cluster shapes
2. Does not work well with clusters of different densities
Department of ISE 2025-2026 P a g e 23 | 40
 Introduction to AI and Applications(25ET15)

2. Hierarchical Clustering
Concept
Builds clusters in a tree-like structure (dendrogram).
Two approaches:
(a) Agglomerative (Bottom-Up)
• Start with each point as a single cluster
• Merge the most similar points step-by-step
• End with one cluster
(b) Divisive (Top-Down)
• Start with all data as one cluster
• Split least similar points repeatedly
• End with individual points
Cluster formation is done by cutting the dendrogram at a chosen depth (gives k clusters).
Hierarchical clustering is deterministic → same output on every run.
Advantages
1. Shows relationships among data objects
2. Results are easy to interpret
Limitations
1. Computationally expensive
2. Highly affected by noise and outliers

3. Density-Based Clustering
Concept
Clusters are formed based on high-density regions separated by low-density areas.
• No need to specify k
• Uses a distance threshold to decide cluster membership
Examples
• DBSCAN
• OPTICS
Advantages
1. Works well for non-spherical cluster shapes
2. Effective even with noise and outliers
Limitations
1. Not suitable for high-dimensional data
2. Difficulty in identifying clusters with varying densities

Department of ISE 2025-2026 P a g e 24 | 40

 Introduction to AI and Applications(25ET15)

K-Means Algorithm
What is K-Means?
• K-Means is an unsupervised machine learning technique used to divide a dataset into
k clusters.
• It does not use labeled data; instead, it identifies patterns by grouping similar
observations.
• A cluster is a group of data points that are:
o Similar within the group (intra-cluster similarity)
o Different from other groups (inter-cluster dissimilarity)

[Link] Properties of Clusters

Consider a bank clustering customers based on income (X-axis) and debt (Y-axis).
Properties
1. High similarity within a cluster
o All points inside a cluster should be similar.
o Helps in tasks like targeted marketing.
2. High dissimilarity between clusters
o Different clusters must be as far apart as possible.

2.5.3 Applications of Clustering in Real-World Scenarios

1. Customer Segmentation
• Used by telecom, e-commerce, advertising, banking, etc.
• Helps target specific customer groups.
2. Document Clustering
• Groups similar documents together.
• Used in search engines, news filtering, topic modelling.
3. Image Segmentation
• Pixels with similar properties are assigned to the same group.
• Used in medical imaging, computer vision.
4. Recommendation Engines
• Groups users or products based on similarity.
• Helps recommend friends, songs, products.

2.5.4 Evaluation Metrics for Clustering

Clustering should not only
create groups but meaningful
and well-separated groups.

Department of ISE 2025-2026 P a g e 25 | 40

 Introduction to AI and Applications(25ET15)

1. Inertia (Intra-Cluster Distance)

• Measures how far points are from the centroid of their cluster.
• Lower inertia = better clustering.
Formula Concept:
Sum of all distances of points from the cluster centroid.
2. Dunn Index
Used to check both cluster compactness and separation.

Components:
• Intra-cluster distance: distance between points and centroid (should be small)
• Inter-cluster distance: distance between centroids of different clusters (should be
large)
Dunn Index = (Minimum inter-cluster distance) / (Maximum intra-cluster distance)
• Higher Dunn Index = Better clusters
• Numerator should be large (clusters far apart)
• Denominator should be small (clusters tight and compact)

2.5.5 How K-Means Algorithm Works

Steps
1. Choose number of clusters (k).
2. Initialize k centroids randomly.
3. Cluster Assignment Step
o Assign each observation to the nearest centroid (using Euclidean distance).
4. Centroid Update Step
o Recalculate centroids (mean of all observations in a cluster).
5. Repeat steps 3 & 4
o Continue until no change in cluster assignments (convergence).

2.5.6 Pros and Cons of K-Means Algorithm

Pros
1. Simple and easy to implement.
2. Scales efficiently to very large datasets.
3. Fast convergence in most cases.
Cons
1. Must predefine the number of clusters (k).
2. Sensitive to outliers.
3. Different results may occur if the order of data changes.
4. Random initialization can lead to different clustering outputs.

Department of ISE 2025-2026 P a g e 26 | 40

 Introduction to AI and Applications(25ET15)

DBSCAN(Density-Based Spatial Clustering of Applications with Noise)

• DBSCAN is an unsupervised, density-based clustering algorithm.

• Forms clusters based on regions of high point density separated by low-density regions.
• Unlike K-Means, DBSCAN does not require specifying the number of clusters (k) and
is robust to noise/outliers.

Why DBSCAN?
• K-Means limitations:
o Must pre-define k (number of clusters).
o Sensitive to initialization and outliers.
o Forms only spherical clusters.
o Every point is forced into some cluster.
• DBSCAN advantages:
o No need to specify k.
o Can detect arbitrarily shaped clusters.
o Identifies noise/outliers.
o Less sensitive to small changes in data.

Key Parameters
1. eps (ε):
o Radius used to find neighboring points.
o Chosen using k-distance graph (look for “elbow”).
2. minPts:
o Minimum number of points required to form a dense region.
o Rule of thumb:
▪ minPts ≥ D + 1 (D = dimensions)
▪ For noisy/large datasets: minPts ≈ 2 × D
o Minimum useful value: 3.

Important Concepts
1. Core Point
• Has ≥ minPts points within distance ε.
2. Border Point
• Not a core point but lies within ε of a core point.
3. Noise Point
• Neither core nor border; insufficient neighbors.
Parameters
1. Density Reachability
• A point q is density-reachable from p if:
o q lies within ε of p and
o p is a core point.
2. Density Connectivity
• Points p and q are density-connected if both are density-reachable from a common
point.
Department of ISE 2025-2026 P a g e 27 | 40
 Introduction to AI and Applications(25ET15)

DBSCAN Algorithm Steps

1. Start with an unvisited point p.
2. Mark p as visited and find all points within ε.
3. If neighbors ≥ minPts → form a new cluster with p as core.
4. Recursively expand the cluster by visiting its neighbors.
5. If neighbors < minPts → mark p as noise (may later become border).
6. Repeat until all points are visited.
Output of DBSCAN
• A set of clusters of varying shapes and sizes.
• A set of noise/outlier points.

Naïve Bayes Classification

• Naïve Bayes is a probabilistic classification algorithm based on Bayes Theorem.
• Despite advancements in complex ML models, Naïve Bayes remains popular because
it is:
o Simple
o Fast
o Works well on large datasets
o Highly effective for text-based classification
• Used in spam filtering, document classification, sentiment analysis, etc.
Assumes all features are independent (naïve assumption), making probability computation
simple.
Conditional Probability
Conditional probability of event A given that B has already occurred:
𝑃(𝐴 ∩ 𝐵)
𝑃(𝐴 ∣ 𝐵) =
𝑃(𝐵)

Example 1: Coin Toss

Probability of getting heads = ½.
Example 2: Drawing a queen of spades
Eligible population = 13 cards (spades only).
1
𝑃(𝑄♠ ) =
13

Example 3: Conditional Probability from Dataset

If 12 out of 60 teachers are male:
12
𝑃(Teacher ∣ Male) = = 0.2
60
Tip: When Y has two categories, compute probability of each
class and select the higher one.
Bayes Rule
Bayes Rule helps compute the posterior probability:
𝑃(𝑋 ∣ 𝑌) ⋅ 𝑃(𝑌)
𝑃(𝑌 ∣ 𝑋) =
𝑃(𝑋)

In classification:
• X = features
• Y = class

Department of ISE 2025-2026 P a g e 28 | 40

 Introduction to AI and Applications(25ET15)

• Training gives P(X|Y).

• For each test record, compute probability of Y given X, and choose the class with
maximum probability (= Maximum A Posteriori).

Types of Events in Probability (Simple Explanation)

Independent Events
Two events are independent when the occurrence of one does NOT affect the other.
If
P(B) = P(B | A) → Event A has no effect on B → Independent events
Example:
Toss a coin twice.
• Probability of head on 1st toss = 1/2
• Probability of head on 2nd toss = 1/2
The second toss does not depend on the first → Independent.
So,
P(Head on both tosses) = 1/2 × 1/2 = 1/4

Mutually Exclusive Events

Two events are mutually exclusive if both cannot happen at the same time.
✔️ Conditional probability becomes 0
If P(B|A) = 0 and P(A|B) = 0 → Mutually exclusive events
Example:
A coin toss cannot give Head and Tail at the same time → mutually exclusive.
Only one event can occur.

Dependent Events
Events are dependent when the occurrence of one affects the probability of the other.
✔️ P(A and B) = P(A) × P(B|A)
or
✔️ P(A and B) = P(B) × P(A|B)
Example: Drawing cards without replacement
We want: one Queen and one King.
• P(Queen first) = 4/52
• After drawing one card, only 51 cards left
• P(King | Queen) = 4/51
So,
P(Queen and King) = (4/52) × (4/51)
Since removing one card changes the next probability → Events are dependent.

Non-Mutually Exclusive Events

These are events that can happen together, so we use:
P(A OR B) = P(A) + P(B) – P(A AND B)
(Subtract the overlap because it gets counted twice)
Example:
Throw a dice. Find probability of getting a number that is:
• a multiple of 2 (2,4,6) → 3 numbers
• a multiple of 3 (3,6) → 2 numbers
• Overlap = 6 → 1 number
So,
P(multiple of 2 or 3)
Department of ISE 2025-2026 P a g e 29 | 40
 Introduction to AI and Applications(25ET15)

= 3/6 + 2/6 – 1/6

= 4/6 = 2/3

Type of Event Meaning Example

Independent One event doesn’t affect the other Two coin tosses
Mutually Exclusive Both cannot happen together Head and Tail
One event affects the probability of Drawing cards without
Dependent
the other replacement
Non-Mutually
Both can happen together Multiples of 2 or 3 on dice
Exclusive

Laplace (Laplacian) Correction

Problem:
If we were to classify fruits as mango, orange or banana,
If a feature value has zero frequency for a class:
𝑃(Long ∣ Orange) = 0

Then the entire probability becomes 0.

Solution – Laplace Smoothing:
Add 1 to each frequency:
count + 1
𝑃=
total + 𝑘

where k = number of possible feature values.

Ensures no probability becomes zero.

Pros and Cons of Naïve Bayes

✔ Advantages
1. Simple and easy to implement.
2. Very fast to train and predict.
3. Works well with small training data.
4. Performs well for text-based classification and large datasets.
✘ Disadvantages
1. Zero conditional probability problem without Laplace correction.
2. Strong assumption of feature independence, which is unrealistic for many real-world
datasets.

Applications of Naïve Bayes

1. Text Classification
Categorizing news articles, documents, blogs.
2. Spam Filtering
Used in DSPAM, SpamAssassin, SpamBayes, Bogofilter, ASSP.
3. Sentiment Analysis
Classifying tweets/posts as positive, negative, neutral.
4. Recommendation Systems
Used with collaborative filtering to predict customer preferences.

Department of ISE 2025-2026 P a g e 30 | 40

 Introduction to AI and Applications(25ET15)

Neural Network
• A Neural Network (NN) or Artificial Neural Network (ANN) is a machine-learning
algorithm inspired by the human brain.
• It learns from examples and consists of many interconnected nodes called neurons.
• Information flows non-linearly and is processed in parallel.
• NN adjusts weights (internal parameters) to improve its predictions.
• Neural Networks are especially useful for tasks easy for humans but hard for
machines, such as pattern recognition.
Applications
• Optical character recognition (OCR)
• Facial recognition, object detection
• Pattern matching

Working of Neural Networks

Biological Analogy
• In the brain:
o Dendrites receive signals
o Neuron processes it
o Axon sends output
• Billions of such neurons form an
interconnected decision-making
system.
Artificial Neuron System
• NN consists of interconnected
input/output units.
• Each connection has a weight.
• During training, NN adjusts
weights to reduce error.
• Error is calculated at the output and sent backward using Backpropagation.
Activation Function
• Output of a neuron ranges from –∞ to +∞, so we use an activation function to map it
to a usable range.
• Helps the model learn non-linear patterns.

Neural Network Structure

A typical NN contains:
1. Input Layer – Receives raw data
2. Hidden Layer(s) – Processes the
information
3. Output Layer – Produces final
results
There can be multiple hidden layers (deep
neural networks).
Feedback Neural Network
• Information flows forward and
backward, enabling learning from
errors.

Department of ISE 2025-2026 P a g e 31 | 40

 Introduction to AI and Applications(25ET15)

Pros and Cons of Neural Networks

Pros
1. Flexible – works for both regression and classification.
2. Performs well on non-linear data (especially images).
3. Accepts any number of inputs and layers.
4. Very fast due to parallel computation.
Cons
1. Simpler algorithms (Decision Tree, Regression) are faster and easier.
2. Requires high computation power and more development time.
3. Needs large amounts of data to perform well.
4. Works only with numeric and complete (non-missing) input data.

Applications
Neural Networks are used in:
1. Pattern Recognition
• Facial recognition
• Fingerprint identification
• Object detection
2. Anomaly Detection
• Detect unusual or abnormal patterns
• Used in fraud detection, intrusion detection
3. Time Series Prediction
• Stock market forecasting
• Weather prediction
4. Natural Language Processing (NLP)
• Text classification
• Named Entity Recognition (NER)
• POS tagging
• Speech recognition
• Spell checking

How Neural Networks Work

Forward Propagation
• Inputs are passed through the network layer by layer
• Final output is produced in the output layer
Backward Propagation
• Error is calculated at the output
• Error is sent backward to update weights
• Goal: reduce error using gradient descent

1. Threshold-based output:
If (x1 + x2 + x3 > threshold) → output = 1; else 0
2. Weights:
Importance of inputs.
Example: w3 > w1 means x3 is more important.
3. Bias:
Helps shift the decision boundary for more flexibility.
4. Training Iteration / Epoch:
One complete cycle of forward + backward propagation.

Department of ISE 2025-2026 P a g e 32 | 40

 Introduction to AI and Applications(25ET15)

What is an Activation Function?

An activation function is a small formula inside a neural network that decides what a
neuron should output.
Why do we need it?
The neuron first calculates:
Weighted Sum = (w₁x₁ + w₂x₂ + w₃x₃ + bias)
But this value can be anything — very large, very small, positive, or negative.
A neuron cannot work properly with such raw values.
So, an activation function converts this value into something meaningful and controlled.
Simple Example
Think of it like a switch:
• If input is strong → turn ON
• If input is weak → stay OFF
Activation function helps decide that.
Main Purpose
✔ Makes the neural network non-linear
✔ Helps the model learn complex patterns (like images, speech, language)
Common Activation Functions
• Sigmoid → outputs between 0 and 1
• Tanh → outputs between –1 and +1
• ReLU → outputs 0 or a positive number

Gradient Descent
Gradient Descent is a method used to reduce error in a neural network by updating weights.
Imagine this:
You are on top of a hill (high error).
You want to reach the bottom (low error).
You take small steps downward each time.
These steps = updating weights
Direction = gradient
Reaching bottom = minimum error

Types of Gradient Descent

1. Full Batch Gradient Descent
• Uses all training data at once
• Calculates weight updates using the entire dataset
• More accurate but slow
Example:
If you have 10 data points, all 10 are used together to update w₁ and w₂.

2. Stochastic Gradient Descent (SGD)

• Uses one data point at a time
• Updates weights after each single example
• Faster but can be noisy
Example:
• First data point updates weights
• Then second data point updates weights
• And so on…

Department of ISE 2025-2026 P a g e 33 | 40

 Introduction to AI and Applications(25ET15)

Neural Networks and Deep Learning

Artificial Intelligence (AI) → broad concept (machines mimic
human intelligence)
Machine Learning (ML) → subset of AI (machines learn from
data)
Deep Learning (DL) → subset of ML (uses deep neural
networks)

Deep Learning (also called Deep Neural Learning or Deep

Neural Networks) is a branch of Artificial Intelligence (AI)
that imitates the working of the human brain to process data,
recognize patterns, and make [Link] learns patterns
automatically from large amounts of data.
• It is a subset of Machine Learning (ML)
• Machine Learning itself is a subset of Artificial Intelligence
• Works well with:
o Images
o Text
o Audio
o Videos
• Needs:
o Huge data
o Powerful hardware (GPUs)

How Does Deep Learning Work?

Deep learning uses Artificial Neural Networks (ANNs) with multiple layers.
• Input Layer – receives raw data
• Hidden Layers – process and transform data
• Output Layer – produces final result
A network with many hidden layers is called a Deep Neural Network
Working Process:
1. Input data enters the first layer
2. Each layer extracts more complex features
3. Weights decide importance of signals
4. Final layer gives prediction
5. Error is calculated
6. Weights are adjusted (learning)
Example: Face Recognition
• First layer → edges and lines
• Middle layers → eyes, nose, mouth
• Final layers → complete face

Department of ISE 2025-2026 P a g e 34 | 40

 Introduction to AI and Applications(25ET15)

Machine Learning vs Deep Learning

Machine Learning (ML) Deep Learning (DL)
It is a subset of Artificial Intelligence. It is a subset of Machine Learning.
It is an evolution of Machine Learning, inspired
It is an evolution of AI.
by the human brain.
Requires a huge dataset (may contain millions of
Requires thousands of data points.
data records).
Usually outputs a numerical value like a predicted Can output anything from numerical values to
output or product. free text, sound, or images.
Uses algorithms like Decision Trees, Random Forests, Uses deep neural networks to interpret data
SVM, Naïve Bayes, Neural Networks, etc. features and relations.
Algorithms are largely self-depicted on data
Algorithms examine specific variables in the data.
analysis once deployed.
Models are trained using CPU (Central Processing Requires GPU (Graphics Processing Unit) for
Unit). training.
More human intervention is needed for getting the Less human intervention once the system is in
desired output. production.
Requires additional setup time but gives more
Faster to set up, but effectiveness may be limited.
accurate and effective results.
Training time is less due to limited data size. Training time is high as the dataset is very large.
Feature engineering is not needed; features are
Requires feature engineering by data analysts.
automatically detected.
DL systems use more powerful hardware and
ML applications are simpler compared to DL.
resources.
Results of DL models are difficult to explain
Results of ML models are easy to interpret.
(black box).
Suitable for highly complex and challenging
Suitable for simple to moderately complex problems.
problems.
Breaks a problem into sub-parts and combines results. Follows an end-to-end learning approach.
Works on both structured and unstructured
Mostly works on structured data.
data.

Department of ISE 2025-2026 P a g e 35 | 40

 Introduction to AI and Applications(25ET15)

Applications of Deep Learning

Application Area Description / Use

Uses millions of images and video data for object detection, traffic sign
Autonomous /
recognition, and pedestrian detection. Requires high-performance GPUs and cloud
Driverless Cars
computing.
Identifies objects from satellite images, locates areas of interest, and determines
Aerospace and Defence
safe and unsafe zones for military operations.
Automatically detects cancer cells, diagnoses diseases and tumours, and supports
Medical Field
personalized medicine based on an individual’s genome.
Improves worker safety by detecting people or objects entering unsafe zones near
Industrial Automation
heavy machinery.
Used in automated hearing systems and voice-based applications to recognize and
Speech Recognition
process human speech accurately.
Virtual assistants such as Siri, Cortana, Alexa, and Google Now use deep learning
Digital Assistants
for speech recognition and natural language processing.
Enables real-time translation of text and speech between languages; used in Google
Language Translation
Translate and Skype Translator.
Email services like Gmail use deep learning to identify and filter spam emails
Email Spam Detection
before they reach the inbox.
Used by companies like PayPal to detect fraudulent transactions by analysing time,
Fraud Detection
location, IP address, transaction amount, and retailer type.
Visual Search Apps like CamFind allow users to take pictures of objects and get detailed
Applications information using deep neural networks.
Google DeepMind’s WaveNet generates human-like speech that sounds more
Speech Generation
natural than traditional systems.
Face Recognition Facebook uses deep learning to identify and tag people in uploaded photos.
Chatbots and Service
Used for intelligent customer support, responding accurately to text-based queries.
Bots
Converts black-and-white images into coloured images automatically using
Image Colourization
context and object recognition.
Personalized Used in shopping and entertainment platforms to recommend products, movies,
Recommendations and shows based on user preferences.

Department of ISE 2025-2026 P a g e 36 | 40

 Introduction to AI and Applications(25ET15)

Support Vector Machine (SVM)

Support Vector Machine (SVM) is a
supervised machine learning algorithm
mainly used for classification, though it can
also be applied to regression.
• SVM classifies data by finding the best
decision boundary (hyperplane) that
separates data points of different classes.
• Each data point is represented in an n-
dimensional space, where n is the number of
features.
• The data points closest to the hyperplane are called support vectors.

Department of ISE 2025-2026 P a g e 37 | 40

 Introduction to AI and Applications(25ET15)

Scenario 1
• Three hyperplanes A, B, and C are drawn.
• B and C separate the two classes better than A.
• Not all separating lines are optimal.
Scenario 2
• All three hyperplanes separate the classes.
• The hyperplane with maximum margin is selected.
• Hyperplane C has the highest margin → best decision boundary.
• Low margin → higher chance of misclassification.
Scenario 3
• Hyperplane A is chosen.
• It maximizes margin and minimizes classification error.
Scenario 4
• One data point (star) lies inside the opposite class (circle).
• Perfect linear separation is difficult in this case.

Evaluating Hyperplanes (Non-Linear Case)

Scenario 5
• Linear hyperplane cannot separate the data.
Solution 1: Feature Transformation
• Add new feature:
𝑧 = 𝑥2 + 𝑦2

• All z values are positive.

• Circles appear near the origin (low z).
• Stars appear farther away (high z).
• Data becomes linearly separable.
Solution 2: Kernel Trick
• Kernel function maps data to higher-dimensional space.
• Converts non-linearly separable data into linearly separable.
• In original space, the decision boundary appears circular.

Hyperplane and Margin

• Hyperplane:
o Line in 2D, plane in 3D, and hyperplane in higher dimensions.
• Margin:
o Distance between the hyperplane and the nearest data points from each class.
o Goal of SVM: maximize the margin for better generalization.
• A low-margin hyperplane may cause misclassification.
Handling Non-linear Data
When data is not linearly separable:
1. Feature Transformation
o Add new features (e.g., 𝑧 = 𝑥 2 + 𝑦 2 ) to make data separable.

Department of ISE 2025-2026 P a g e 38 | 40

 Introduction to AI and Applications(25ET15)

2. Kernel Trick
o Transforms data from low-dimensional space to high-dimensional space.
o Common kernels:
▪ Gaussian RBF
▪ Polynomial
▪ Sigmoid
Tuning Parameters
• Cost (C):
o Large C → low bias, high variance (risk of overfitting).
o Small C → high bias, low variance.
• Gamma (for RBF kernel):
o Controls the influence of a single training point.
o Small gamma → underfitting.
• Parameter selection is done using cross-validation and grid search.
How SVM Works
1. Select an optimal hyperplane with maximum margin.
2. Apply a penalty (C) for misclassification.
3. Use kernel functions for non-linearly separable data.
Data Standardization
• SVM is distance-based; hence feature scaling is mandatory.
• Common methods:
o Z-score normalization
o Min–Max scaling
Advantages
• Works well with high-dimensional data.
• Effective for non-linear problems using kernels.
• Memory efficient (uses only support vectors).
• Performs well for text and image classification.
• Free from multicollinearity issues.

Disadvantages
• Computationally expensive for large datasets.
• Does not directly provide probability estimates.
• Linear SVM behaves similar to logistic regression for linearly separable data.

Applications of SVM in Real World

Support Vector Machine (SVM) is widely used in real-world applications due to its high
accuracy and ability to handle high-dimensional and non-linear data.

• Face Detection & Emotion Recognition

SVM classifies image regions as face or non-face based on pixel features. It is also
used to detect facial expressions such as happy, sad, or normal and apply filters
accordingly.
Department of ISE 2025-2026 P a g e 39 | 40
 Introduction to AI and Applications(25ET15)

• Text and Hypertext Categorization

SVM classifies documents such as emails, news articles, and web pages into
categories (e.g., spam vs non-spam, business vs movies).

• Image Classification & Handwriting Recognition

SVM provides high accuracy in image search, handwriting recognition, signature
verification, and distinguishing between human and computer-generated text.

• Bioinformatics
Used to classify proteins, genes, and cancer data. Kernel functions help measure
similarity between protein sequences.

• Email Spam Detection

SVM effectively classifies emails as spam or non-spam with high accuracy (around
97%).

• Geo and Environmental Sciences

Applied to analyze noisy spatial and spatiotemporal environmental data.
• Generalized Predictive Control
SVM helps control chaotic systems, improving system stability and performance.

• Inverse Geo-Sounding Problem

Used to analyze electromagnetic data and determine layered planetary structures.

• Seismic Liquefaction Analysis

SVM is applied to SPT and CPT test data to predict liquefaction occurrence.

• Mathematics (Smooth SVM)

SSVM is used for solving complex mathematical optimization problems.

• Texture Classification
SVM classifies images based on texture, such as smooth or rough surfaces.
• Speech Recognition
Applied to acoustic data for speech recognition systems, especially helpful for
assistive technologies.

• Steganography Detection
Used to detect hidden messages in digital images, important in security and forensic
applications.

Department of ISE 2025-2026 P a g e 40 | 40