Course Title: Introduction to AI and Applications Course Code:

25ET12TL

Module - 3: Techniques in AI
Techniques in AI, Machine Learning Model, Regression Analysis in Machine
Learning, Classification Techniques, Clustering Techniques, Naïve Bayes
C lassification, Neural Network, Support Vector Machine (SVM).

W hat is AI?

AI combines large data sets with fast, iterative processing and intelligent
algorithms to learn patterns in data.
OR
artificial intelligence (AI) as the science and engineering of making intelligent
machines, especially intelligent computer programs.
AI includes:
- Theories
- Methods and
- Tech supporting automatic learning.

Sub- Domains of AI:

1. Cognitive Computing: algorithms try to mimic human brain by analyzing text,

speech, images or objects in the same way to give the desired output.
Examples: virtual assistants and chatbot that can understand natural language,
answer customer queries
2. Computer Vision: is a branch of AI that tries to understand an image by
breaking it down into several parts and then studying each part of the image.
Example: Autonomous robot, surveillance and security systems, retail stores for
tracking inventory and customers etc.
3. Machine Learning: is a branch of computer science that analyses data and
identifies patterns to teach a machine to deduce results and make decisions
without any human intervention.

1
ML algorithms learn from experiences rather than instructions.
Example: Image recognition, Speech recognition, virtual personal assistant.
4. Neural Networks: machine learning models that mimic the complex functions
of the human brain. These models consist of interconnected nodes or neurons
that process data, learn patterns and enable tasks such as pattern recognition and
decision- making.

Example: Content recommendation systems, chatbot processing conversations

5. Deep Learning: ML technique using layered neural networks for improved

classification and inference. U ses multiple hidden layers for complex pattern
recognition.
Examples: Fraud detection, Financial service - stock market, Self- driving car,
Predictive analy tics.
6. Natural Language Processing: is a subfield of computer science and artificial
intelligence (AI) that uses machine learning to enable computers to understand
and communicate with human language.

Example: C omputers and digital devices (smart phone, smart watches, laptop,
monitor etc) to recognize, understand and generate text and speech by
combining computational linguistics (grammar).

Supporting Technologies:

- Graphical Processing U nits (GPU s): Powerful computing for neural network
training.

2
- Internet of Things (IoT): Automates data analysis & extraction from connected
devices.

Machine Learning Model:

Figure: ML Model is a combination of Task, Performance and Experience

A computer program is said to learn from experience with respect to a set of
tasks and performance measure P.

Task (T)
The real- world problem that needs to be solved.
Examples: Predicting sales, Email spam classification, Regression, Clustering,
Recognition (Speech / Face).
ML tasks include classification, regression, clustering, and recognition. 

Experience (E)
Knowledge gained from data provided to the algorithm.
The model learns patterns and relationships from data, similar to humans learning
from experiences.
Types: supervised, unsupervised, reinforcement learning.

Performance ()
A measure indicating how well an ML algorithm completes a task using its
experience.
Evaluated using metrics such as accuracy, F1- score, confusion matrix, precision,
recall, sensitivity.

Performance assessment helps refine models and optimize them for real- world
use.

3
Regression Analysis in Machine Learning:

W hat is Regression Analysis?

• A statistical tool for studying relationships between a dependent (target)
and one or more independent (predictor) variables.
• Helps analysts understand how changes in independent variables affect
the dependent variable.
Predicting Continuous Values
• Regression predicts real, continuous values such as temperature, age,
salary, price.
• Example: Predicting company sales based on advertisement spending.
Applications of Regression
• Predicting rainfall using temperature and other factors.
• Predicting the height of a person given the age of the person.
• Predicting road accidents due to rash driving.
Some important regression techniques are:

1. Linear Regression: Linear regression is the most basic, simple and widely used
technique for predicting values of a continuous variable.

• Regression analysis creates a mathematical equation that defines y as a

function of the x variables. This equation is then used to predict the value
of y on the basis of values of the predictor variables (x).

i.e Y = F(X)

where Y - Dependent variable; X - Independent variable

4
Multiple predictor variables:

Let us consider X 1 and X 2 are two independent variables then multiple regression
equation is given by:

Y = b 0 + b 1X 1 + b 2 X 2 + e

W here b 0 - Y intercept,

b 1 and b 2 are coefficient of X 1 and X 2 and e is the error.

Case Study:

Let us compute auto fare. We know that auto charges are computed by adding a
fixed amount and a variable cost. The fixed amount is set by the government. The
variable cost depends on the distance travelled. It is usually specified as Rs 11 per
km. If the fixed charge of an auto is Rs 30, then a linear regression equation can
be used to find the cost of any auto trip. By using “x” to represent the number of
kilometers travelled, “y”, the cost of that auto ride can be calculated as,
Y = 30 + 11X.
Now, if I say that I took an auto to reach a destination that was 10 KM away from

5
my house, how much would I have to pay?

Solution:
W.K.T Y = 30 + 11X.
Distance travelled to reach destination 10KM.
Therefore,
Y = 30 + 11* 10 = 30 + 110 = Rs140

2. Logistic regression:
It is used when the dependent variable is binary (0/1, True/False, Yes/No) in
nature.

Logistic regression model transforms the linear regression function continuous

value output into categorical value output using a sigmoid function which maps
any real- valued set of independent variables input into a value between 0 and 1.
This function is known as the logistic function.

3. Ridge Regression: regularized version of Linear Regression that adds a penalty

on the size of coefficients.

It helps in reducing overfitting, especially when:

- The dataset has multicollinearity (high correlation between features)
- The number of features is large compared to the number of samples.

6
Ridge is also called L2 Regularization.

4. Lasso (Least Absolute Shrinkage Selector Operator) regression

Lasso Regression is a regularization technique used in machine learning to
prevent overfitting and improve model generalization.
- It reduces the number of dependent variables if the penalty term is huge
- Lasso regression is also known as L1regularization.
For example, if we' re predicting house prices based on features like location,
square footage and number of bedrooms. Lasso Regression can identify most
important features.
5. Polynomial regression:
Polynomial regression is used to construct a model that fits non- linearly
separated data. In such a case, the best- fitted line is not a straight line, but a
curve.
1 2 n
Y= b 0 + b 1X + b 2 x + … … ..+ b n X

6. Step wise Regression:

Systematic method for building predictive regression models.
Adds or removes variables step- by- step to identify the best subset of predictors.
Aimed at improving model accuracy while keeping it simple.

7. Elastic Net Regression:

7
A hybrid of Ridge (L2) and Lasso (L1) regression
Handles multicollinearity and performs feature selection
Produces grouping effect for highly correlated predictors
Suitable when number of predictors > number of observations

Classification Techniques:
1. K- Nearest Neighbours Algorithm(K- NN)
2. Decision Tree
3. Random Forests

1. K- Nearest Neighbours Algorithm(K- NN)

is a supervised learning algorithm in which the output value of data is known but
how to get the output is not known?
This algorithm looks ‘K’ nearest neighbors in the training data and assigns a class
or predict a value based on those neighbors.

This K- Nearest Neighbour Algorithm use Euclidean distance (d) as illustrated

below:

Figure: Euclidean distance between two points

8
Example 1:
C onsider the following table which represent each student academic score and
extracurricular score with their class. Calculate the distance between new student
and all other students, if new student have academic score = 7 and
extracurricular score = 6. Also classify the new student in which class he belongs
to considering K = 3.
Student Academic Score Extracurricular score Class

A 8 7 Outstanding

B 6 5 Sporty

C 9 8 Outstanding

D 5 4 Outstanding

Solution:
We know that from K- Nearest Neighbour (K- NN)
Euclidean distance (D):

9
From the above values, K = 3 Nearest Neighbour for new students are:
Student A, Student B and Student C. i.e 1.41, 1.41 and 2.83.
Out of these, majority class is belonging to Outstanding.
Therefore, the new students belong to Outstanding class.

Let us consider the following example:

2. Consider the following table which represent the Academic Score and Extra
curriculum (EC) score and their Grade.

Assuming that co- ordinates of the New Student are (8, 2). Calculate the distance
between each student name and classify on which grade the New Student is
belong to?

10
Solution:
W.K.T
Euclidean distance between two points is given by:

Pros and C ons of KNN algorithm:

Pros:
1. Simple and easy to understand
2. No assumptions
3. Works for both classification and regression
4. Good for Multi- class problems
C ons:
1. High Prediction time
2. No ideal for high dimensional data
3. Require large memory usage
4. Sensitive to data scaling

2. Decision Tree

11
- It is a supervised learning algorithm used for classification and regression tasks.
It works by creating tree like structure that splits data based on different features
to make the prediction.

Figure: Simple Decision tree.

Let us understand with an example:

12
1. C onsider the following table, predict whether a person will buy a product
based on age and income.

Solution:

Advantages:
1. Easy to understand and implement
2. No need for data scaling
3. Handles missing data
4. Suitable for non- linear relationship

Dis- advantages:
1. Overfitting
2. Instability
3. Random Forests:
- Collective learning method used for both classification and regression tasks.
They combine multiple decision trees to improve the overall performance by

13
reducing the overfitting issue seen in individual decision trees.

Figure: Decision tree

In simpler terms, we can understand the working of a random forest in the

following steps.

Step 1: Random samples are selected from a given dataset.

Step 2: A decision tree is constructed for each sample to obtain a prediction
result from each decision tree as shown in above Figure.
Step 3: Voting for each predicted result is performed.
Step 4: Prediction result with the maximum votes is selected as the final
prediction.

Advantages of using Random forests:

• It can be very effectively used to estimate missing data
• It maintains accuracy when a large proportion of the data is missing
• It can balance errors in datasets where the classes are imbalanced
• It can handle huge datasets with large number of dimensions

Dis- advantages of using Random forests:

• Random forests might easily over fit noisy datasets, especially when
performing regression.
• Random forests are slow in generating predictions as multiple decision
trees are constructed and then the process of voting selects the best

14
 prediction result.
• Random forests are difficult to interpret as and when compared to a
decision tree model.
Consider the below example of training data instance:

Solution:
From the above decision tree the major voting (Bagging) is for Class A therefore
the prediction is Class A.

Clustering Techniques:

C lustering is a machine learning technique used to group similar data points

together based on their features.
It is used in various applications like healthcare, business, and marketing.

There are three main types of clustering algorithms:

a. Partitional Clustering
b. Hierarchical Clustering
c. Density- Based Clustering

a. Partitional Clustering: divides the dataset into non- overlapping groups

(clusters) where each data point belongs to exactly one cluster.
Example Algorithms:
• K- Means: Divides data into K clusters where K is predefined by the user.
K- Means is a popular unsupervised learning algorithm used for clustering. It
groups data into k clusters where each cluster contains data points that are
more similar to each other than to those in other clusters.

15
 • K- Medoids: Similar to K- Means but chooses actual data points as the
center of each cluster (medoids).
Advantages:
• Works well when clusters are spherical in shape. Scalable and efficient
with large datasets.
Limitations:
• Struggles with complex shapes (non- spherical clusters). C annot handle
clusters of different densities.
b. Hierarchical C lustering: builds a tree- like structure (dendrogram) that
represents the hierarchy of data clusters.
● Agglomerative (Bottom- up): Starts with individual data points and merges
them into clusters.
• Divisive (Top- down): Starts with all data points in one cluster and splits
them into smaller clusters.
Advantages:
• Shows relationships between data points at different levels. The resulting
clusters are easy to interpret.
Limitations:
• C omputationally expensive, especially with large datasets. Sensitive to
noise and outliers.

16
C . Density- Based Clustering: Clusters are formed based on the density of data
points in a region. It doesn’t require you to specify the number of clusters in
advance.
Popular Algorithms:
• DBSCAN (Density- Based Spatial Clustering of Applications with Noise):
Forms clusters where data points are close enough to each other, and
handles outliers (points that don’t belong to any cluster).
• OPTICS (Ordering Points to Identify the Clustering Structure): Similar to
DBSCAN but can handle varying densities of clusters.
Advantages:
• Works well with non- spherical shapes and outliers. No need to specify the
number of clusters.
Limitations:
• Not ideal for high- dimensional data. Struggles with clusters of different
densities.

Naive Bayes Classification: is a probabilistic machine learning algorithm

based on Bayes' Theorem. It is simple, fast, and widely used for classification
tasks, such as spam detection, sentiment analysis, and text classification.
C alculate the probability of an event occurring given prior knowledge about
related events.

17
P(A|B): The probability of A occurring given that B is true (posterior probability).
P(B|A): The probability of B occurring given that A is true (likelihood).
P(A): The probability of A occurring (prior probability).
P(B): The probability of B occurring (evidence).

Naive Bayes Assumption:

• Naive means that we assume all features (variables) are independent of
each other.
• In reality, features are often dependent, but the " naive" assumption
simplifies the calculation, which makes the model faster and easier to
implement.
For example: in spam email detection, we treat the presence of each word as
independent, though in reality, some words may often appear together.

How Naive Bayes Works:

1. Given a dataset with labeled categories (e.g., spam or not spam
emails), we calculate the probability of each feature (word)
belonging to each class (spam or not spam).
2. For a new email, we compute the probabilities of it being spam or

18
 not spam based on its features (words) and classify it to the class with
the highest probability.

Applications of Naive Bayes:

[Link] C lassification: Classifying whether a document is spam or not.
[Link] Analysis: Analyzing tweets or reviews to classify them as positive,
negative, or neutral.
[Link] Systems: Predicting whether a user will like a product based on their
past behaviors.
[Link] Diagnosis: Classifying whether a patient has a certain disease based on
medical features.

Example: Naive Baye

Introduction to Neural Networks:

Figure: Feedback neural network (ANN: 3 - 2 - 1)

● Input Layer: Input data is fed to the first layer of neurons which receive the
raw data.
● Processing in Hidden Layers: Each layer processes the data using its
weights and applies the activation function.
● Activation Function:

19
 Purpose: Transforms the input into an output within a certain range,
typically between 0 and 1 (for binary classification).

● Output Layer: The final result is output after all layers have processed the
data.
i.e Gives the final prediction or classification.
● Forward Propagation: Data flows through layers to predict the output.
● Backpropagation: Errors from the output layer are passed backward
through the network, adjusting the weights for better future predictions.

Types of Neural Networks:

1. Feedforward Neural Network
2. Convolution Neural Network (CNNs)
3. Recurrent Neural Networks (RNNs)

2. C onvolution Neural Networks:

Special type of artificial neural network (a deep learning algorithm) used to

recognize images and visual information. It is one of the key techniques in image
recognition and object detection.
It is used for applications such as:
● Recognizing faces in photos
● Distinguishing between cats and dogs
● Reading handwritten digits
● Analysing medical images like X- rays
● Identifying traffic signs for self- driving cars

20
 Figure: Architecture of CNNs
The first few layers are convolution layers, which are responsible for extracting
the basic features of the image such as edges and shape.
1. Input: these will receive the raw data (images) for making prediction.
2. Convolution: it is the core of C NN, designed to find distinctive patterns in the
input data. It applies the set of filters to produce a feature extraction.
After the convolution operation, a non- linear activation function, typically ReLU
(Rectified Linear U nit), is applied to introduce non- linearity into the model.
3. Pooling: The purpose of this pooling layer is to reduce the size of feature maps
while preserving the most important features. This helps to reduce computational
complexity and control overfitting.
4. Fully connected: The final layer of a CNN is typically a fully connected layer
that classifies the CNN' s output. This layer is similar to a traditional neural
network layer, connecting to all neurons from the previous layer.
5. Output: Display the final classification.
3. Recurrent Neural Networks (RNNs): used for sequence- based data, like time
series or Speech.

21
RNNs are organized into layers that consist of interconnected nodes or neurons.
In a simple RNN, the output from one layer becomes an input for the same layer
as the data moves from one time step to the next. 
This feedback loop enables RNNs, maintain an internal state memory, helping
them to recognize patterns in sequences of information.
Consequently, this iterative feedback process gives RNNs the ability to capture
temporal dependencies and recognize patterns across time.

Pros and Cons of Neural Networks:

Pros:
● Flexible
● Works well with complex and non- linear data
● Scalable
Cons:
● Complex and computationally expensive
● Training can take time
● Requires a lot of data
● Interpretability issues

22
 Support Vector Machine (SVM): is a popular supervised learning algorithm
primarily used for classification tasks.
● Its main goal is to classify data points into one of two categories based
on features in the dataset.
● This works by finding the best hyperplane that separates two classes of
data points in an n- dimensional space (where n is the number of features).
● Hyperplane is the decision boundary that helps classify data points into
their respective classes.

Figure: Support Vector Machine (SVM)

In a 2D space, the hyperplane would be a line that divides the data into two parts
(one part for C lass 1 and another for C lass 2).
● Support vectors are the data points that are closest to the hyperplane.
These points are critical in determining the optimal hyperplane.
● Margin: margin is the distance between the hyperplane and the closest
support vectors. The larger the margin, the better the classifier and more
accurate the model.

How Does SVM Work?

1. Linear C lassification: For linearly separable data, SVM finds a hyperplane
that maximizes the margin between two classes.

Example: We have a set of data points (e.g., red and blue circles), and we
need to classify them. SVM draws a line (hyperplane) between the red and

23
 blue circles to separate them.

2. Non- Linear Classification (Kernel Trick): W hen the data is not linearly
separable, SVM uses a method called the kernel trick to map the data into
higher dimensions, where it becomes easier to find a separating
hyperplane.

• Example: In the case of non- linearly separable data, imagine you have
two classes (red and blue), but they can' t be separated by a straight line.
By transforming the data into a higher- dimensional space, the data points
become separable by a hyperplane.

Types of Kernels in SVM:

1. Linear Kernel: U sed when the data is linearly separable. The decision
boundary is a straight line or hyperplane.

2. Polynomial Kernel: U sed when the data has a non- linear relationship, but
we can represent it as a polynomial.

3. Radial Basis Function (RBF) Kernel: One of the most commonly used
kernels, especially for non- linear problems. It maps data points to a higher-
dimensional space and creates a curved decision boundary.

Advantages of SVM:
● Works well with high- dimensional data.
● Effective in both linear and non- linear classifications.
● Memory efficient, as it uses only support vectors for classification.

Disadvantages of SVM:
● It can be slow with large datasets.
● It does not perform well with overlapping classes.
● It is sensitive to the choice of kernel and parameter tuning.

24
Applications of SVM:
● Image Recognition: SVMs are used to recognize patterns and objects
in images.
● Text Classification: SVMs are used for spam email classification or
document categorization.
● Face Detection: SVMs are used to detect faces in images by
classifying pixel data as face or non- face.
● Bioinformatics: SVMs are used for classifying protein sequences and
gene expression data.

25