Reg.

No

SAVEETHA ENGINEERING COLLEGE

(Autonomous, Affiliated to Anna University, Chennai)

Common to M.E CSE

MCS1009 – Machine Learning – Answer Key

Faculty Name Mr. C. Rajesh Department M.E-CSE

Answer All Questions

PART A

QA101 What is overfitting in machine learning? How can it be reduced?

Overfitting happens when a machine learning model learns the training data too much and cannot
perform well on new data.
Methods to Reduce Overfitting:
 Use more training data
 Use a simpler model
 Apply regularization or cross-validation.
QA102 Define conditional probability with an example.
Conditional probability is the probability of an event occurring given that another event has already
occurred.
P( A ∩ B )
P( A ∣ B )=
P( B )
QA103 Differentiate between supervised and unsupervised learning.
Supervised Learning Unsupervised Learning
Uses labeled data (input with correct
Uses unlabeled data (no correct output given).
output).
Model finds hidden patterns or groups in
Model learns to predict the output.
data.
Example: Classification and Regression. Example: Clustering and Association.
QA104 What is entropy in information theory?
It is the expected value of the self-information (surprisal) of a random variable, defined as

H ( X )=− ∑ P( x ) log b P( x )
𝐻(𝑋)=−𝑥∈𝒳𝑃(𝑥)log𝑏𝑃(𝑥)
x∈X

1
 QA105 List any two advantages and two challenges of machine learning.
Advantages:
1. Automation: Machines can perform tasks automatically without human intervention.
2. Improves accuracy: Machine learning models can make better predictions using data.
Challenges:
1. Requires large amount of data for training.
2. Risk of overfitting, where the model performs poorly on new data.
QA106 Define precision and recall. Why are they important in classification?
Precision:
Precision is the ratio of correctly predicted positive observations to the total predicted positives.
Recall:
Recall is the ratio of correctly predicted positive observations to all actual positives.
Importance in Classification:

Precision and recall are important because they measure the performance of a classification model,
especially when dealing with imbalanced datasets. They help evaluate how accurately the model
identifies positive cases.
QA201 Differentiate between Discriminative and Generative models.
Discriminative Models Generative Models
Learn the decision boundary between
Learn the joint probability distribution of data.
classes.
Model the probability **P(Y X)**.
Used mainly for classification tasks. Can generate new data samples.
Examples: Naïve Bayes, Hidden Markov
Examples: Logistic Regression, SVM
Model
QA202 What is the main objective of Linear Regression?
Main Objective of Linear Regression:
The main objective of linear regression is to predict the value of a dependent variable based on one or
more independent variables by fitting a best-fit straight line.
y=β 0 + β 1 x +ϵ
QA203 Define Overfitting and mention one way to prevent it.
Overfitting occurs when a machine learning model learns the training data too well, including noise, and
performs poorly on new or unseen data.
One way to prevent it:
Use regularization or increase the training data.
QA204 What is Cross-Validation in the context of supervised learning?
Cross-validation is a technique used in supervised learning to evaluate the performance of a model by
dividing the dataset into training and testing parts multiple times.
It helps to check how well the model generalizes to new data and reduces overfitting.
QA205 What is the purpose of Lasso Regression in linear models?
Lasso Regression is used in linear models to reduce overfitting and perform feature selection by
shrinking some coefficients to zero.

QA206 State the key difference between Logistic Regression and Linear Regression.
 Linear Regression is used to predict continuous values (e.g., price, temperature).
 Logistic Regression is used for classification problems to predict categorical outcomes (e.g., yes/no,
spam/not spam).
QA301 What is the main objective of clustering in unsupervised learning?
 The main objective of clustering in unsupervised learning is to group similar data points into
clusters based on their similarities without using labeled data.
 It helps to discover hidden patterns or structures in the dataset.
QA302 State any two differences between K-Means and Hierarchical Clustering.
K-Means Clustering Hierarchical Clustering

2
 Number of clusters must be specified in
Number of clusters is not required initially.
advance (K).
Forms clusters using a tree-like structure
Uses centroids to form clusters.
(dendrogram).
QA303 What is cluster validity and why is it important in clustering algorithms?
Cluster Validity:
Cluster validity is the process of evaluating how well the data points are grouped into clusters in a
clustering algorithm.
Importance:
It is important because it measures the quality of clustering and helps determine whether the formed
clusters are meaningful and accurate.
QA304 Define Principal Component Analysis (PCA) and mention one of its uses.
Principal Component Analysis (PCA):
PCA is a dimensionality reduction technique used to transform a large set of variables into a smaller set
of new variables called principal components, while preserving most of the data’s information.
Use:
It is used to reduce the number of features in a dataset, making data analysis and visualization easier.

QA305 What is the role of the EM (Expectation-Maximization) algorithm in unsupervised learning?

The EM algorithm is used in unsupervised learning to estimate the parameters of statistical models
when the data has hidden or missing variables.
Role:
It works by repeatedly performing two steps (Expectation and Maximization) to find the best parameters
and improve the clustering or probability model.
QA306 What is meant by model-based learning in reinforcement learning?
Model-based learning is an approach where the agent builds a model of the environment that describes
how actions affect states and rewards. Using this model, the agent can predict future outcomes of
actions.
QA401 What is the basic assumption behind the Naïve Bayes algorithm?
The Naïve Bayes algorithm assumes that all features are independent of each other given the class label.
This means the presence or absence of one feature does not affect the presence of another feature in the
dataset.
QA402 Define Maximum Likelihood Estimation (MLE) in the context of probabilistic learning.
Maximum Likelihood Estimation is a method used in probabilistic learning to estimate the parameters
of a statistical model that make the observed data most likely.

QA403 What is Maximum A Posteriori (MAP) estimation and how does it differ from MLE?
MAP estimation is a method used to estimate model parameters by considering both the observed data
and prior knowledge about the parameters.
θMLE = arg max θ L(x | θ) = arg max θ Yn i=1 fX(xi | θ)
Difference from MLE:
 MLE considers only the likelihood of the data.
 MAP considers both the likelihood and the prior probability of parameters.

QA404 Mention any two components of a Bayesian Belief Network.

Two components of a Bayesian Belief Network:
1. Nodes – Represent random variables in the network.
2. Edges (Directed links) – Represent probabilistic dependencies between the variables.
QA405 What is the significance of probability density estimation in machine learning?
Probability Density Estimation:
 Probability density estimation is used in machine learning to estimate the probability
distribution of data points in a dataset.
 It is important because it helps in understanding the data pattern and making probabilistic
predictions.
QA406 List two advantages of using Hidden Markov Models (HMMs) in sequence modeling.
 Captures temporal dependencies – HMMs can model sequences where the current state depends
on the previous state.
3
  Handles hidden states – They can represent systems where the underlying states are not directly
observable.
QA501 What is the biological inspiration behind artificial neural networks?
Biological Inspiration of Artificial Neural Networks:
Artificial Neural Networks are inspired by the structure and functioning of the human brain. They
mimic biological neurons, where neurons are connected and transmit signals to process information.
QA502 Define a Perceptron and mention its primary limitation.
Perceptron:
A perceptron is the simplest type of artificial neural network used for binary classification problems. It
combines input features using weights and produces an output.
Primary Limitation:
It can only classify linearly separable data and cannot solve non-linear problems (e.g., XOR problem
QA503 What distinguishes a Multi-Layer Perceptron (MLP) from a single-layer Perceptron?
 A Single-Layer Perceptron has only one layer of weights between input and output and can
solve only linearly separable problems.
 A Multi-Layer Perceptron (MLP) has one or more hidden layers, allowing it to learn complex
and non-linear patterns in data
QA504 What is the role of the activation function in a neural network?
Role of Activation Function:
The activation function determines whether a neuron should be activated or not by transforming the
input into an output signal. It also introduces non-linearity, allowing neural networks to learn complex
patterns in data.
QA505 State any two differences between Feed Forward Networks and Recurrent Neural Networks.
Feed Forward Neural Networks (FNN) Recurrent Neural Networks (RNN)
Data flows only in one direction from Data can flow in loops, allowing feedback
input to output. connections.
Used for static data such as image Used for sequential data such as text or
classification. speech processing
QA506 What is the purpose of the backpropagation algorithm in training neural networks?
Backpropagation:
Backpropagation is an algorithm used to train neural networks by updating the weights of the network.
It works by calculating the error at the output and propagating it backward through the network to
minimize the error.

PART B

QB101 (a) Derive Bayes’ Theorem from the principles of conditional probability. Explain its role in machine
learning with an example
Bayesian Conditional Probability

Conditional Probability
In English, a conditional probability states “what is the chance o( an event E happening given
thal 1 have already observed some other event P’. It is a critical idea in machine learning and
probability because it allows us to update our beliefs in the face of new evidence. The
definition for calculating conditional probability is:
P(EF) I
P(E F) ——
P(F)

This equation implies that: P(EF j = P(ElF)P(F) which we call the Chain Rule. Intuitively it
states that the probability of observing events E and F is the probability of observing F, multiplied
by the probability of observing E, given that you have observed F.

Here is the general form of the Chain rule:

4
 P(E 2' •.E ) = P(Ed)P( 2 t)... P(E |E ... — i)

In the case where the sample space has equally likely outcomes:

#outcomes in E consistent with F _ |EF| _ |EF|

P(ETF) —#outcomes in S consistent with F“ |SFI“ III
Itayes 'Theoxem
Very often we know a conditional probability in one direction, say P(OF), but we would like to
know the conditional probability in the other direction. Bayes Theorem provides a way to convert
from one to the other. There are a lot of things called Bayes Theorem. Here are the two most
common equations:

Most Common Version:

——
P(F|E)

Expanded Version:
P(E|F)P(F)
P(F |E) —— P(E|F)P(F) +P(E|E’”)P(FC)
Conditional Probability Example 1

Machine Learning (sometimes called Data Science) is the love child of: probability. data
and computers. Sometimes machine learning involves complex algorithms. But often it's
just the core ideas of probability applied to large datasets.
As an example let us consider Netflix. a company that has thrived because of well thought
out machine learning. One of the primary probabilities that they calculate is the probability
tbal a user will watch a given movie given nn other information about lhe user. We call this the
prinr,

Let E be the event that a user watches a given movie. We can approximate P(E) using the
definition nf probability from Friday's class:

Using this definition, we can approximate the probability by counting the number of users
who watched thg movie divided by the number of users whn are nn Netflix. Since the
number nf users on Netflix is huge, this is a good approximation.
(OR)
QB101 (b) Elaborate on the types of machine learning problems. Provide examples and describe how each type
handles data and learning objectives.
Types of Machine Learning Problems

5
 QB102 (a) Derive the expression for the joint probability P(A∩B) using marginal and conditional probabilities.

Joint Probability
6
 Probability of events A and B denoted byP(A and B) or P(A ∩ B)is the
probability that events A and B both occur. P(A ∩ B) = P(A). P(B) . This only
applies if Aand Bare independent, which means that if Aoccurred, that doesn’t
change the probability of B, and vice versa.
Conditional Probability
Let us consider A and B are not independent, because if A occurred, the
probability of B is higher. When A and B are not independent, it is often useful to
compute the conditional probability, P (A|B), which is the probability of A given
that B occurred: P(A|B) = P(A ∩ B)/ P(B).
The probability of an event A conditioned on an event B is denoted and defined
P(A|B)
= P(A∩B)/P(B)
Similarly, P(B|A) = P(A ∩ B)/ P(A) . We can write the joint probability of as
A and B as P(A ∩ B)= p(A).P(B|A), which means : “The chance of both things
happening is the chance that the first one happens, and then the second one given
the first happened.”
Bayes’ Theorem
Bayes’s theorem is a relationship between the conditional probabilities of two
events. For example, if we want to find the probability of selling ice cream on a hot
and sunny day, Bayes’ theorem gives us the tools to use prior knowledge about the
likelihood of selling ice cream on any other type of day (rainy, windy, snowy etc.).

any other type of day (rainy, windy, snowy etc.).

where Hand E are events, P(H|E) is the conditional probability that event H occurs
given that event E has already occurred. The probability P(H) in the equation is
basically frequency analysis; given our prior data what is the probability of the
event occurring. The P(E|H) in the equation is called the likelihood and is
essentially the probability that the evidence is correct, given the information from
the frequency analysis. P(E) is the probability that the actual evidence is true.
Let H represent the event that we sell ice cream and Ebe the event of the
weather. Then we might ask what is the probability of selling ice cream on any
given day given the type of weather? Mathematically this is written as P(H=ice
cream sale | E= type of weather) which is equivalent to the left hand side of the
equation. P(H) on the right hand side is the expression that is known as the prior
because we might already know the marginal probability of the sale of ice cream. In
our example this is P(H = ice cream sale), i.e. the probability of selling ice cream
regardless of the type of weather outside. For example, I could look at data that said
30 people out of a potential 100 actually bought ice cream at some shop somewhere.
So my P(H = ice cream sale) = 30/100 = 0.3, prior to me knowing anything about
the weather. This is how Bayes’ Theorem allows us to incorporate prior information
[2].

(OR)
QB102 (b) Explain the concepts of overfitting and underfitting with diagrams and real-world examples. Discuss the
bias-variance trade-off and techniques to achieve optimal model performance.
1. Overfitting and Underfitting:
What is overfitting?
Let’s start with an example, say one day you are walking down a street to buy something, a
7
 dog comes out of nowhere you offer him something to eat but instead of eating he starts
barking and chasing you but somehow you are safe. After this particular incident, you might
think all dogs are not worth treating nicely.
So this overgeneralization is what we humans do most of the time, and unfortunately
machine learning model also does the same if not paid attention. In machine learning, we
call this overfitting i.e model performs well on training data but fails to generalize well.
Overfitting happens when our model is
too complex. Things which we can do to
overcome this problem:
1. Simplify the model by selecting one with fewer parameters.
2. By reducing the number of attributes in training data.
3. Constraining the model.
4. Gather more training data.
5. Reduce the noise.
What is underfitting?
Yes, you guessed it right underfitting is the opposite of overfitting. It happens when our
model is too simple to learn something from the data. For E.G., you use a linear model on a
set with multi-collinearity it will for sure underfit, and the predictions are bound to be
inaccurate on the training set too.
Things which we can do to overcome this problem:
1. Select a more advanced model, one with more parameters.
2. Train on better and relevant features.
3. Reduce the constraints.

🔹
Bias–Variance Trade-Off
Bias
Error due to wrong assumptions in learning algorithm.

🔹
High Bias → Model too simple → Underfitting
Variance
Error due to model sensitivity to small fluctuations.
High Variance → Model too complex → Overfitting
Trade-Off Curve
 As model complexity increases:
o Bias decreases
o Variance increases
 Total error forms a U-shaped curve
 Optimal complexity lies at minimum validation error
Error Decomposition Formula
Total Error=Bia s 2 +Variance+ IrreducibleError
Where:
 Bias² → Error from incorrect model assumptions
 Variance → Error from sensitivity to training data
 Irreducible error → Noise in data (cannot be removed)
Goal: Minimize Bias² + Variance
Techniques to Achieve Optimal Performance
1. Cross-Validation
 K-Fold Cross Validation
 Reduces variance in evaluation
 Ensures robust performance estimation
2. Regularization
Adds penalty term to loss function.
L1 Regularization (Lasso)
Loss=MSE+ λ ∣ w ∣
L2 Regularization (Ridge)
Loss=MSE+ λ w 2
Reduces model complexity.
3. Early Stopping
Stop training when validation loss increases.
Used in Neural Networks.

8
 4. Dropout (Deep Learning)
Randomly deactivate neurons during training.
Prevents co-adaptation.
5. Pruning (Decision Trees)
Remove weak branches.
Reduces variance.
6. Data Augmentation
Used in image datasets.
Generates more training samples.
7. Feature Selection
Remove irrelevant features.
Improves generalization.
8. Ensemble Methods
 Random Forest
 Gradient Boosting
 Bagging reduces variance
 Boosting reduces bias

(OR)
QB103 (a) Explain entropy and mutual information in the context of information theory. How can these metrics be
used in feature selection for classification problems?

Knowledge discovery is a process that helps you engineer features or

attributes in a base dataset to improve the performance of the machine learning
models. Information theory has a lot of 'information measures' or metrics that can
help you evaluate the importance and relevance of each new attribute that you
develop.
We have already seen in the sections above that conditional entropy
determines how two variables are correlated. If H(Y|X=xi) represents the
conditional entropy of a variable 'Y' given X=xi. Let us assume that 'Y' represents
the target variable in a machine learning model and 'X' is an explanatory variable,
say a categorical variable with several labels (xi) where 'i' specifies the number of
unique labels of the variable 'X'. In this case, we need to ensure that H(Y|X=xi)
when aggregated for all 'i' is close to zero. This is because H(Y|X)=0 if and only if
the value of 'Y' is completely determined by the value of X. The value of conditional
entropy H(Y|X) is equal to H(Y) if and only if 'Y; and 'X' are independent random
variables.

Figure 20: Information Measures that can be used as learning and content measures
Information theory treats information as a physical entity, like energy or mass. It deals with

9
 theoretical analyses of how information can be transmitted over any channel: natural or man-
made. Thus, it defines a few laws of information. Let us assume a basic system for information
flow as follows
Variable Length codes:
One traditional way of assigning codewords to data elements would be to
assign codes with fixed lengths, or every data element gets assigned a code of the
same code length. Let us look at a visual representation of the same. If we have 4
values of a given variable, say 'Discount type' with the following 4 values:
'Promotion', 'Priority Customer', 'Repeat buy' and 'None'. If we assume that the 4
types of discounts are equally probable, they can safely be mapped to codewords of
equal length, say 00, 01, 10 and 11, as shown in the image below:

Now, if we know that discounts under 'Promotion' are more frequent than the rest
and have been assigned the following probability values.

We can then, design codes weighted according to these probability values.

We design the code to reduce the total length of the message. Since one would
transmit the most frequent terms (data elements) most number of times, you would
want to associate the least length with the most frequent term. Hence, we design a
code with the least number of bits for the most frequent data element. Let us look at
an example of the same.

10
 (OR)

QB103 (b) Describe the steps in building a classification model using Naïve Bayes. Include dataset preprocessing,
probability estimation, application of Bayes’ theorem, and model evaluation.

Algorithm: Naive Bayes

Let there be a training data set having n features , , f'q. Let lb denote an arbitrary value of
,
/ 2Fo f 2. and so on. Let the set of class labels be •l 2i • • i <p • It there be given a test
instance
having the feature vector

We are required to determine the most appropriate class label that should be assigned to
the test instance.
Step l. Compute the probabilities P{cz) for k -—
1, ... ,p. Step 2. Form a table showing the
conditional probabilities

for all values of / , / , . , /q and for k —- 1......., p.

Step 3. Compute the products

gy = P( |c+) J? 2!*k) P(z |*L)P(<k)

for ñ —— 1, ... , p.
Step 4. Find j such q -- max(qi sz - g)
Step S. Assign the class label cy to the test instance A.
6.3.3 Example
Problem
Consider a training data set consisting of the fauna of the world. Each unit has three
features named “Swim”, “Fly” and “Crawl”. Let the possible values of these features
be as follows:
Swim Fast, Slow, No
Fly Long,
Short, Rarely, No
Crawl Yes, No
For simplicity, each unit is classified as “Animal”, “Bird” or “Fish”. Let the training data
set be as in Table 6.1. Use naive Bayes algorithm to classify a particular species if its features
are (Slow, Rarely, No)?
Sl. No. Swim Fly Crawl Class
1 Fast No No Fish
2 Fast No Yes Animal
3 Slow No No Animal
4 Fast No No Animal
S No Short No Bird
6 No Short No Bird
7 No Rarely No Animal
8 Slow No Yes Animal

11
 9 Slow No No Fish
10 Slow No Yes Fish
II No Long No Bird
12 Fast No No Bird
Table 6.1: Sample data set for naive Bayes algorithm
In this example, the features are
Fi - “Swim”, F2 - “Fly”, F3 - “Crawl”.
The class labels are
1' “Animal”, •2' “ Bird”, •3 • “Fish”.
The test instance is (Slow, Rarely, No) and so we have:
+i - “Slow”, zt = “Rarely”, z3 = “No”.

We construct the frequency table shown in Table 6.2 which summarises the data. (It may be noted
that the construction of the frequency table is not part of the algorithm.)

Features
C la ss Swim (Hi i Fly (N;) Crawl (f3) Total
Fast Slow No Long Short Rarely No Yes No
A n im a l ( i) 2 2 1 0 0 1 4 2 3 5
B ird (‹ 2) 1 0 3 1 2 0 1 1 3 4
Fish (‹ ) 1 2 0 0 0 0 3 0 3 3
Total 4 4 4 1 2 1 8 4 8 12

Table 6.2: Frequency table for the data in Table 6.1

Step 1. We compute following probabilities.
No. of records with class label “Animal”
Total number of examples

No. of records with class label “Bird”

( ) ' T o tal n u m b er o f exam ples
= 4/12
No of records with class label “Fish”
Total number ot examples
= 3/12
Step 2. We construct the following table of conditional probabilities:

Features
Swim ( ) Fly (Ft) Crawl ( t)
Class
F ast S lo w No Long Short Rarcly No Y cs No
A n im a l(rt) 2/5 2/5 1/5 ()/5 ()/5 1/5 4/5 2/5 3/5
B ird (c ,) l/4 0/4 3/4 1/4 2/4 0/4 1/4 0/4 4/4
F ish (c t) 13 2/3 0/3 0/3 0/3 0/3 3/3 0/3 3/3

T a b le 6 .3 : T a b le o f th e c o n d itio n a l p ro b a b ilitie s P f, ct)

Note: The conditional probabilities are calculated as follows:
of records with ’i - Slow and class label
No. of records with class label r t
= 2/5.
Step 3. We now calculate the following numbers:

12
 QB201 (a) Discuss Generative and Discriminative models in the context of supervised learning.

Generative models are a wide class of machine learning algorithms which make
predictions by modelling joint distribution P(y, x).
Discriminative models are a class of supervised machine learning models which
make predictions by estimating conditional probability P(y|x).
In order to use a generative model, more unknowns should be solved: one has to
estimate probability of each class and probability of observation given class. These
probabilities are used to compute joint probability, and finally, joint probability can be used
as a substitute for conditional probability to make predictions.

The discriminative model takes a shorter way: it simply estimates conditional

probability
directly.
There are many pros and cons to each of the models. I just note that generative model
can be
used to generate new samples, but it requires more data. Discriminative model often
superior than generative model, given the same amount of data, but it does not know about
dependencies between features, because it is irrelevant for prediction. Therefore
discriminative model can not generate new samples. Now let’s take a closer look at the
concept of generative models.

Generative model
As I showed earlier, to make predictions, conditional distribution P(y|x) is enough.
But since P(y|x) = P(y, x) / P(x), where P(x) is constant for the given x and all possible y, it
is valid to use joint distribution P(y, x) to make predictions.
By modelling joint distribution P(y, x) is meant that for each pair (yᵢ, xᵢ) a
probability P(yi, xi) is known (modelled). At the beginning it was a bit difficult for me to
understand how it is even possible the range of possible values of X might be enormous,
so it’s gonna be unrealistic to suggest
probabilities for each xi, leave alone pair (yi, xi). How is it supposed to be done?
First. Bayes theorem! It breaks computation of joint probability P(y,x) into
computation of two other types of probabilities: probability of class, P(y), and probability of
observation given class, P(x|y).
P(y, x) = P(y) * P(x|y)
13
 What benefits does it give? This way it is at least easier to figure out probability P(y),
because it can be estimated from the dataset by computing class frequencies. P(x|y) is
trickier, because usually x is not just one feature, but a set of features: x = xi, …, xn, which
might have dependencies between each other.
P(x|y) = П P(xi|y, x1, xi-1, xi+1, xn)

Often the dependencies between the features are not known, especially when they
appear in complex constellations (y, x1, xi-1, xi+1, xn).
So what should be done to estimate P(x|y)? For this, there is the following trick:

Second. Make wild assumptions! Or just some assumptions which make estimation
of P(x|y) tractable. Naive Bayes classifier can serve as a perfect example of a generative
model with such assumption, which makes computation of P(x|y) easier. Namely, it has
independence assumption between the features xi, …, xn.
P(x|y) = П P(xi|y)
With this relaxation, estimation of P(x|y) is tractable, because every P(xi|y) can be estimated
either by finding frequencies of discrete feature xi independently from other features or
using Gaussian distribution, if feature xi is continuous.

(OR)

(b) Consider the time series data given below:

Use the least

xi 8 3 2 10 11 3 6 5 6 8
square method to
determine the equation of
yi 4 12 1 12 9 4 9 6 1 14
line of best fit for the data.

Mean of xi values = (8 + 3 + 2 + 10 + 11 + 3 + 6 + 5 + 6 + 8)/10 = 62/10 = 6.2

Mean of yi values = (4 + 12 + 1 + 12 + 9 + 4 + 9 + 6 + 1 + 14)/10 = 72/10 = 7.2
Straight line equation is y
= a + bx. The normal
equations are
∑y = an + b∑x
∑xy = a∑x + b∑x2
x y x2 xy

8 4 64 32
QB201
3 12 9 36

2 1 4 2

10 12 100 120

11 9 121 99

3 4 9 12

6 9 36 54
5 6 25 30
6 1 36 6
8 14 64 112
∑x = 62 ∑y = 72 ∑x2 = 468 ∑xy = 503

14
 Substituting these values in the normal equations,
10a + 62b = 72….(1)
62a + 468b = 503….(2)
(1) × 62 – (2) × 10,
620a + 3844b – (620a + 4680b) = 4464 – 5030
-836b = -566
b = 566/836
b = 283/418
b = 0.677
Substituting b = 0.677 in equation (1),
10a + 62(0.677) = 72
10a + 41.974 = 72
10a = 72 – 41.974
10a = 30.026
a = 30.026/10
a = 3.0026
Therefore, the equation becomes,
y = a + bx
y = 3.0026 + 0.677x

Now, we can find the sum of squares of deviations from the obtained values as:
d1 = [4 – (3.0026 + 0.677*8)] = (-4.4186)
d2 = [12 – (3.0026 + 0.677*3)] = (6.9664)
d3 = [1 – (3.0026 + 0.677*2)] = (-3.3566)
d4 = [12 – (3.0026 + 0.677*10)] = (2.2274)
d5 = [9 – (3.0026 + 0.677*11)] =(-1.4496)
d6 = [4 – (3.0026 + 0.677*3)] = (-1.0336)
d7 = [9 – (3.0026 + 0.677*6)] = (1.9354)
d8 = [6 – (3.0026 + 0.677*5)] = (-0.3876)
d9 = [1 – (3.0026 + 0.677*6)] = (-6.0646)
d10 = [14 – (3.0026 + 0.677*8)] = (5.5814)
∑d2 = (-4.4186)2 + (6.9664)2 + (-3.3566)2 + (2.2274)2 + (-1.4496)2 + (-1.0336)2 +
(1.9354)2 +
(-0.3876)2 + (-6.0646)2 + (5.5814)2 = 159.27990

QB202 (a) List and explain at least three causes and three mitigation strategies for each:
 Underfitting
 Overfitting
i) Define what constitutes a “good fit” in a machine learning model. At what point during training can
this be identified, and why is it important?
ii) Describe the concept of cross-validation, its role in model training, and explain how it assists in
identifying the optimal model parameters without compromising test data integrity.

(I) Underfitting: A statistical model or a machine learning algorithm is said to have

underfitting when it cannot capture the underlying trend of the data, i.e., it only performs
well on training data but performs poorly on testing data. (It’s just like trying to fit
undersized pants!) Underfitting destroys the accuracy of our machine learning model. Its
occurrence simply means that our model or the algorithm does not fit the data well enough.
Reasons for Underfitting:

15
 1. High bias and low variance
2. The size of the training dataset used is not enough.
3. The model is too simple.
4. Training data is not cleaned and also contains noise in it.

Techniques to reduce underfitting:

1. Increase model complexity
2. Increase the number of features, performing feature engineering
3. Remove noise from the data.
4. Increase the number of epochs or increase the duration of training to get better results.

Overfitting: A statistical model is said to be overfitted when the model does not make
accurate predictions on testing data. When a model gets trained with so much data, it starts
learning from the noise and inaccurate data entries in our data set. And when testing with
test data results in High variance.

Reasons for Overfitting are as follows:

1. High variance and low bias
2. The model is too complex
3. The size of the training data

Examples:

Techniques to reduce overfitting:

1. Increase training data.
2. Reduce model complexity.
3. Early stopping during the training phase (have an eye over the loss over the
training period as soon as loss begins to increase stop training).
4. Ridge Regularization and Lasso Regularization
5. Use dropout for neural networks to tackle overfitting.

Good Fit in a Statistical Model: Ideally, the case when the model makes the predictions
with 0 error, is said to have a good fit on the data. This situation is achievable at a spot
between overfitting and underfitting. In order to understand it, we will have to look at the
performance of our model with the passage of time, while it is learning from the training
dataset.
Cross Validation
Unfortunately, things are a little bit more complicated than that, since we might also
want to know how well the algorithm is generalising as it learns: we need to make sure that
we do enough training that the algorithm generalises well. In fact, there is at least as much
16
 danger in over-training as there is in under-training. The number of degrees of variability in
most machine learning algorithms is huge — for a neural network there are lots of weights,
and each of them can vary. This is undoubtedly more variation than there is in the function
we are learning, so we need to be careful: if we train for too long, then we will overfit the
data, which means that we have learnt about the noise and inaccuracies in the data as well as
the actual function. Therefore, the model that we learn will be much too complicated and
won’t be able to generalise.

(OR)

(b) Discuss how Lasso differs from Ridge Regression in terms of regularization and its impact on model
coefficients. Provide the mathematical formulation of Lasso Regression, explaining the role of the
regularization parameter λ (lambda).

What is Lasso Regression?

Lasso regression is a regularization technique. It is used over regression methods for a more
accurate prediction. This model uses shrinkage. Shrinkage is where data values are shrunk
towards a central point as the mean. The lasso procedure encourages simple, sparse models
(i.e. models with fewer parameters). This particular type of regression is well-suited for
models showing high levels of multicollinearity or when you want to automate certain parts
of model selection, like variable selection/parameter elimination.

Lasso Regression uses L1 regularization technique (will be discussed later in this article).
It is used when we have more features because it automatically performs feature selection.

Lasso Meaning
The word “LASSO” stands for Least Absolute Shrinkage and Selection Operator. It is a
statistical
formula for the regularisation of data models and feature selection.

Regularization
Regularization is an important concept that is used to avoid overfitting of the data,
especially when the trained and test data are much varying.
QB202 Regularization is implemented by adding a “penalty” term to the best fit derived from the
trained data, to achieve a lesser variance with the tested data and also restricts the influence
of predictor variables over the output variable by compressing their coefficients.

Lasso Regularization Techniques

There are two main regularization techniques, namely Ridge Regression and Lasso Regression. They
both differ in the way they assign a penalty to the coefficients. In this blog, we will try to understand
more about Lasso Regularization technique.

L1 Regularization
If a regression model uses the L1 Regularization technique, then it is called Lasso Regression. If it used
the L2 regularization technique, it’s called Ridge Regression. We will study more about these in the later
sections.
L1 regularization adds a penalty that is equal to the absolute value of the magnitude of the coefficient.
This regularization type can result in sparse models with few coefficients. Some coefficients might
17
 become zero and get eliminated from the model. Larger penalties result in coefficient values that are
closer to zero (ideal for producing simpler models). On the other hand, L2 regularization does not result
in any elimination of sparse models or coefficients. Thus, Lasso Regression is easier to interpret as
compared to the Ridge.
Mathematical equation of Lasso Regression
Residual Sum of Squares + λ * (Sum of the absolute value of the magnitude of coefficients)

Where,
 λ denotes the amount of shrinkage.
 λ = 0 implies all features are considered and it is equivalent to the linear regression
where only the residual sum of squares is considered to build a predictive model
 λ = ∞ implies no feature is considered i.e, as λ closes to infinity it eliminates more
and more features
 The bias increases with increase in λ
 variance increases with decrease in λ

(a) Derive the logistic function (sigmoid function) and explain how it maps any input to a probability
between 0 and 1.

QB203

18
 (OR)

(b) Explain how SVM constructs the separating line (or hyperplane) to divide data into classes, and discuss
the importance of choosing the "ideal" hyperplane. Why is there more than one possible separating line,
QB203
and how does SVM select the optimal one?

19
 Machine learning involves predicting and classifying data and to do so we
employ various machine learning algorithms according to the dataset. SVM or
Support Vector Machine is a linear model for classification and regression
20
 problems. It can solve linear and non-linear problems and work well for many
practical problems. The idea of SVM is simple: The algorithm creates a line or a
hyperplane which separates the data into classes.
In this blog post I plan on offering a high-level overview of SVMs. I will talk
about the theory behind SVMs, it’s application for non-linearly separable datasets
and a quick example of implementation of SVMs in Python as well. In the upcoming
articles I will explore the maths behind the algorithm and dig under the hood.

(a) Explain the role of unlabeled and unstructured data in unsupervised learning. Why is unsupervised
learning important, and what are its typical use cases in real-world applications?

The input to the unsupervised learning models is as follows:

 Unstructured data: May contain noisy(meaningless) data, missing values, or unknown
data
 Unlabeled data: Data only contains a value for input parameters, there is
no targeted value(output). It is easy to collect as compared to the labeled one in
the Supervised approach.

Types of Unsupervised Learning are as follows:

 Clustering: Broadly this technique is applied to group data based on different

QB301 patterns, such as similarities or differences, our machine model finds. These
algorithms are used to process raw, unclassified data objects into groups. For
example, in the above figure, we have not given output parameter values, so
this technique will be used to group clients based on the input parameters
provided by our data.
 Association: This technique is a rule-based ML technique that finds out some
very useful relations between parameters of a large data set. This technique is
basically used for market basket analysis that helps to better understand the
relationship between different products. For e.g. shopping stores use
algorithms based on this technique to find out the relationship between the sale
of one product w.r.t to another’s sales based on customer behavior. Like if a
customer buys milk, then he may also buy bread, eggs, or butter. Once trained
well, such models can be used to increase their sales by planning different
offers.
 Some algorithms: K-Means Clustering
DBSCAN – Density-Based Spatial Clustering of Applications
with Noise BIRCH – Balanced Iterative Reducing and Clustering
using Hierarchies Hierarchical Clustering

(OR)

(b) Discuss at least four different objectives or use-cases where clustering is used to draw insights from
unlabeled data. Explain how the “criteria for good clustering” may vary based on the application.
QB301
Introduction to Clustering
It is basically a type of unsupervised learning method. An unsupervised learning
method is a method in which we draw references from datasets consisting of input data
21
 without labeled responses. Generally, it is used as a process to find meaningful structure,
explanatory underlying processes, generative features, and groupings inherent in a set of
examples.
Clustering is the task of dividing the population or data points into a number of
groups such that data points in the same groups are more similar to other data points in the
same group and dissimilar to the data points in other groups. It is basically a collection of
objects on the basis of similarity and dissimilarity between them.
For ex– The data points in the graph below clustered together can be classified into
one single group. We can distinguish the clusters, and we can identify that there are 3
clusters in the below picture.

It is not necessary for clusters to be spherical. Such as :

DBSCAN: Density-based Spatial Clustering of Applications with Noise

These data points are clustered by using the basic concept that the data point lies within the
given constraint from the cluster center. Various distance methods and techniques are used
for the calculation of the outliers.

Clustering Methods :
 Density-Based Methods: These methods consider the clusters as the dense region
having some similarities and differences from the lower dense region of the space.
These methods have good accuracy and the ability to merge two clusters. Example
DBSCAN (Density-Based Spatial Clustering of Applications with Noise), OPTICS
(Ordering Points to Identify Clustering Structure), etc.
 Hierarchical Based Methods: The clusters formed in this method form a tree-type
structure based on the hierarchy. New clusters are formed using the previously formed
one. It is divided into two category
 Agglomerative (bottom-up approach)
 Divisive (top-down approach)

examples CURE (Clustering Using Representatives), BIRCH (Balanced

Iterative Reducing Clustering and using Hierarchies), etc.

 Partitioning Methods: These methods partition the objects into k clusters and each
partition forms one cluster. This method is used to optimize an objective criterion
similarity function such as when the distance is a major parameter example K-
means, CLARANS (Clustering Large Applications based upon Randomized
Search), etc.
 Grid-based Methods: In this method, the data space is formulated into a finite
number of cells that form a grid-like structure. All the clustering operations done on
these grids are fast and independent of the number of data objects example STING
(Statistical Information Grid), wave cluster, CLIQUE (CLustering In Quest), etc.

22
 Clustering Algorithms :
K-means clustering algorithm – It is the simplest unsupervised learning algorithm that
solves clustering problem.K-means algorithm partitions n observations into k clusters
where each observation belongs to the cluster with the nearest mean serving as a prototype
of the cluster.

Applications of Clustering in different fields

1. Marketing: It can be used to characterize & discover customer segments
for marketing purposes.
2. Biology: It can be used for classification among different species of plants and animals.
3. Libraries: It is used in clustering different books on the basis of topics and information.
4. Insurance: It is used to acknowledge the customers, their policies and identifying the
frauds.

(a) Explain the step-by-step process involved in forming clusters using the K-Means algorithm. Describe
how centroids are initialized, updated, and how the convergence is achieved.

QB302

23
24
 (OR)

(b) Describe the main idea behind hierarchical clustering and explain how it builds nested clusters in the
form of a hierarchy or tree structure. Mention the key differences between hierarchical and partitioning-
based clustering methods.

1. Hierarchical Clustering

QB302

25
26
 (a) Discuss how PCA transforms original features into a new set of orthogonal features (principal
components) and how it can be used for visualizing high-dimensional data in 2D or 3D.

1. Principal Component Analysis

QB303

27
 Putting this in more formal terms, we have a data matrix X and we want to rotate it so
that the data lies along the directions of maximum variation. This means that we multiply
our data matrix by a rotation matrix (often writtenPas ) so that Y = P*X, whePre is
chosen so that the covariance matrix of Y is diagonal, i.e.,
0
0

(OR)

(b)

Illustrate how EM can estimate the parameters (mean, covariance, and mixing coefficients) of Gaussian
QB303
components when data is assumed to be generated from a mixture of Gaussians.

28
29
30
 (a) What is conditional probability, and how is it different from unconditional probability? Describe the
formula for calculating the probability of an event AAA occurring given that BBB has occurred, and
provide an example to illustrate its application.

QB401

31
 40 40 *0
40

F ig u re 6 .1 : E v en ts A , B , G w h ic h are n o t m u tu ally in d e p e n d en t: E q s.(6 .1 6.3) are satisfied, but

E g .(h .4 ) is n o t satisfie d .

Figure 6.2: Events A, B, G which are not mutually independent: Eg.(6.4) is satisfied but Eqs.(6.1}—
(6.2) are not satished.

(OR)

(b) What are the core principles behind the Naïve Bayes algorithm? How does it calculate the probability of
a class label given a set of features, and why is it particularly efficient for text classification?

QB401

32
 Suppose we are given a test instance having the feature vector

We are required to determine the most appropriate class label that should be assigned to the test
instance. F or this p urp ose w e co m p ute the fo llo w ing co nd itio nal probabilities

and choose the maximum among them. Let the maximum probability be Plc; A). Then, we choose
as the most appropriate class label for the training instance having A as the feature vector.
The direct computation of the probabilities given in Eq.(6.5) are difficult for anumber of reasons.
The Bayes’ theorem can b applied to obtain a simpler method. This is explained below.
6 .3 .3 Computation of probabilities
Using Bayes’ theorem, we have:

(6.6)

S in c e , b y a ssu m p tio n , th e d a ta h a s cla ss-c o n d itio n a l in d e p e n d e n c e , w e n o te th at th e e v e n ts “ 1 •".

“ 2 k”! --, :s„ ct are independent (because they are all conditioned on the same class label ct).
Hence we have

Using this in Eq,(6.6) we get

Since the denominator P(X) is independent of the class labels, we have

P c X) « P(>1 k)P 21* )- P n k)Pi )

So it is enough to find the maximum among the following values:
P(zi|cp)P(z2|o#)- P(:cq Jet)P(ez), k -- 1, ... , p.
Remarks
The various probabilities in the above expression are computed as follows:
(pp) _ No. of examples with class label ct
Total number of examples
of examples with jth feature equal to z¿ and class label k
No. of examples with class label k

6.3.3 The algorithm

Algorithm: Naive Bayes

Let there be a training data set having n features , , f'q. Let lb denote an arbitrary value of
,
/ 2Fo f 2. and so on. Let the set of class labels be •l 2i • • i <p • It there be given a test
instance
having the feature vector

We are required to determine the most appropriate class label that should be assigned to
the test instance.

33
 Step l. Compute the probabilities P{cz) for k -— 1, ... ,p.
Step 2. Form a table showing the conditional probabilities

fo r all v alu es o f / , / , . , /q an d fo r k — - 1. ...... , p.

Step 3. Compute the products

g y = P( |c+ ) J? 2!*k) P(z |*L)P(<k)

fo r ñ — — 1 , ... , p.
Step 4. Find j such q -- max(qi sz - g)
Step S. Assign the class label cy to the test instance A.

(a) In the context of Naïve Bayes classification, explain how Bernoulli, Multinomial, and Gaussian
distributions are used to model feature likelihoods. What are the key differences in their mathematical
formulations?

Esfimation of p
Consider a random sample N = (z , ... , zq} taken from a Bernoulli distribution with the probability
function /(m[p). The log likelihood function is

(r) s /( ip) + + fi /(*•|r)

= lOg ' (l p) l—" --1- • • -1— BOQ p‘ ( ) l—‘*'

[*i oz p • (* — i) l°s(1 —P)] • • [* loss +( —*, ) los(1 —P)]

To find the valuc of p that maximizes L(p) we set up the cquation
QB402

that is,
1 —z z ____________1
+ + = 0.

2. Multlnomlal density
Suppose that the outcome of a random event is one of R classes, each of which has a
probability of occurring p, with

We represent each outcome by an ordered Z7-tuple x = ( i, - , :r ) where exactly one of i ,

- , « is 1 and all others are 0. m, = 1 if the outcome in the i-th class occurs. The probability
function can be expressed as

Here, pt, ... , y›rr are the parameters.

We choose n random samples. The i-the sumple may be represented by
<i = (=ii. - - - =xi)-
The values of the parameters that maximizes the likelihood function can be shown to be

34
 2. Gaussian (normal) density
A co n tin u o u s ra n d o m v a ria b le X h a s th e G a u ssia n o r n o rm a l d is trib u tio n if its d e n sity fu n ctio n is

Here p and cr are the parameters.

Given asample mt, z2, ... ,zpfromthe distribution. the loglikelihood function is

1
log(2s) — it log
2
Setting up the equations
—- 0, =0
d;t

(OR)

(b) Describe the step-by-step process of the Apriori algorithm. How is candidate generation and pruning
performed to find frequent item sets?

1. Maximum Apriori

QB402

35
 (a) Explain the structure and components of a Bayesian Network. How do directed acyclic graphs (DAGs)
represent conditional dependencies?

1. Bayesian Belief Networks

QB403

36
37
 (OR)

(b)

Explain the role of kernel density estimation (KDE) in probability density estimation. How does the
QB403
choice of kernel and bandwidth affect the output?

38
 1. Probability Density Estimation

(a) In what ways are neural networks applied across different domains such as image recognition, natural
language processing, and medical diagnosis? Provide examples to support your answer.

Neural Networks are powerful machine learning models that can learn complex patterns from
QB501
large datasets. They are widely applied in many domains such as image recognition, natural
language processing (NLP), and medical diagnosis. These networks mimic the working of
the human brain using interconnected neurons and layers (input, hidden, and output layers).

39
 1. Image Recognition
Neural networks, especially Convolutional Neural Networks (CNNs), are widely used in
image recognition tasks. CNNs automatically detect features such as edges, shapes, and
textures from images and classify them.

Applications:

 Face Recognition: Used in smartphones for unlocking devices and security systems.
 Object Detection: Identifying objects such as cars, animals, or people in images and
videos.
 Autonomous Vehicles: Detecting traffic signs, pedestrians, and road lanes.
 Image Classification: Categorizing images into classes (e.g., cat, dog, tree).

Examples:

 Facebook/Instagram use neural networks for automatic face tagging.

 Self-driving cars use CNN models to recognize traffic signs and obstacles.
 Google Photos automatically groups images based on people or objects.

2. Natural Language Processing (NLP)

Neural networks help computers understand, interpret, and generate human language. Models
such as Recurrent Neural Networks (RNNs), LSTM networks, and Transformer models
are commonly used.

Applications:

 Machine Translation: Translating text from one language to another.

 Chatbots and Virtual Assistants: Conversational AI systems.
 Sentiment Analysis: Identifying emotions or opinions in text.
 Speech Recognition: Converting spoken language into text.

Examples:

 Google Translate uses neural machine translation models.

 ChatGPT-like systems generate human-like responses using transformer networks.
 Amazon Alexa and Siri use neural networks for speech understanding.

3. Medical Diagnosis
Neural networks are increasingly used in healthcare to assist doctors in diagnosing diseases
and analyzing medical data.

Applications:

 Medical Image Analysis: Detecting tumors in MRI, CT scans, and X-rays.

 Disease Prediction: Predicting diseases such as diabetes or heart disease using patient
data.
 Drug Discovery: Identifying potential medicines by analyzing biological data.
 Clinical Decision Support: Assisting doctors in making accurate treatment decisions.

40
 Examples:

 Neural networks detect brain tumors in MRI images using CNN models.
 AI systems help identify breast cancer from mammograms.
 Predictive models analyze patient history to predict heart disease risk.

(OR)

(b) Explain the structure and working of a biological neuron. How are concepts like dendrites, axons,
synapses, and action potentials modeled in artificial neural networks?

QB501

41
 by dendrites are electrical impulses. The interneuronal transrnissinn is
sometimes electrical but is really effected hy the release of chemical
transmitters at the synapse. Thus, terminal boutons generate the
chemical that affects the receiving ocwou. The receiving neuron either
genemles an irripulse to its axon, or produces no response.
The neuron is able to mspond to the tntal of its inpurs aggregated
within a short time interval called the period of latent Aerion, The
neuron's response is generated if tin total potcntial of its membrane
reaches a certain level. The mem-brane can be considered as a shell,
which aggregates the magnitude of the incom-ing signals over some
duration. Specifically, the nearoz generous a pulse response and sends it
to its axon only if the conditions necessary for firing are fulfilled.
Let us consider the conditions necessary for the firing of a neuron.
Incom-ing impulses can be excitatory if they cause the firing, or
inhibitory if they hinder the firing of the response. A more precise
condition for firing is that the excitation should exceed the inhibition by
the amount called the threshold of the neuron, typically a value of about
40 mV (Arbib 1987). Since a synaptic connection causes the excitatory or
inhibitory reactions of the receiving neuron, it is practical to assign
positive and negative unity weight values, respectively, to such
connections. This allows us to reformulate the neuron's firing condition.
The neuron fires when the total of the weights to receive impulses exceeds
the threshold value during the latent summation period.
The incoming impulses to a neuron can only be generated by
neighboring neurons and by the neuron itself. Usually, a certain number of
incoming impulses are required to make a target cell fire. Impulses that
are closely spaced in time and arrive synchronously are more likely to
cause the neuron to fire. As mentioned before, observations have been
made that biological networks perform temporal integration and
summation of incoming signals. The resulting spatio-temporal processing
performed by natural neural networks is a complex process and much less
structured than digital computation. The neural impulses are not synchro-
nized in time as opposed to the synchronous discipline of digital
computation. The characteristic feature of the biological neuron is that
the signals generated do not differ significantly in magnitude; the signal in
the nerve fiber is either absent or has the maximum value. In other words,
information is transmitted between the nerve cells by means of binary
signals.
After carrying a pulse, an axon fiber is in a state of complete
nonexcitability for a certain time called the refractory period. For this time
interval the nerve does not conduct any signals, regardless of the intensity
of excitation. Thus, we may divide the time scale into consecutive intervals,
each equal to the length of the refractory period. This will enable a
discrete-time description of the neurons' performance in terms of their
states at discrete time instances. For example, we can specify which
neurons will fire at the instant k + 1 based on the excitation conditions at
the instant k. The neuron will be excited at the present instant if the
number of excited excitatory synapses exceeds the number of excited
inhibitory synapses at the previous instant by at least the number T, where
42
 T is the neuron's threshold value.

(a) Discuss the limitations of the single-layer perceptron. Why is it incapable of solving problems like the
XOR problem, and how did this lead to the development of multi-layer perceptron?

1 . Perceptron

QB502

43
 (OR)

(b) How does the activation function in each layer of a Multi-layer Perceptron impact the model's
performance? Compare common activation functions like sigmoid, ReLU, and tanh.

QB502

44
 Therefore, the network's computations can be written out as:

(1)

(2)
(1)

(3)

The above computations are therefore written in vectorized form us:

h*") a!°) (w(2)h(1! -I- b(2))

y — g(3) (W(3)h(2) -I- b(3))

45
 . The computations are written as follows (note the transposes):

(a) What is a feedforward neural network? Explain its architecture and how information flows through the
layers in such a network.

1. Feed Forward Network

QB503

46
 (OR)

(b) Explain the role of activation functions in the backpropagation process. How do different activation
QB503
functions affect the backpropagation and gradient computation?

47
 1 . Back Propagation

48
 PART C (1 x 15 = 15 marks)

(Case study/Comprehensive type Questions)

C Knowled Difficul
O ge Level ty
49
 (Blooms) Level
(1-5)
(a Explain how the depth of a Decision Tree affects its performance. What happens when CO K4 4
) the tree depth is too shallow or too deep? Discuss the relationship between tree depth, 2
bias, and variance, and explain how it relates to overfitting and underfitting.

QC2
01

Overcast

Humidity Yes Wind

Strong

Yes

playing tennis.
from that node corresponds to one of the possible values
for this attribute. An instance is classified by starting at
the root node of the tree, testing the attribute specified
by this node, then moving down the see branch
corresponding to the value of the attribute in the given
example. This process is then repeated for the subtree
rooted at the new node.
Figure 3.1 illustrates a typical learned decision tree.
This decision tree clas-sifies Saturday mornings according
to whether they are suitable for playing tennis. For

50
 example, the instance

I,Outlook —— Sunn y, Temperat ure —— Hot, Humid i t y —— Hi gh,

Wind —— St r on g

would [Link] down the leftmost branch of this decision

tree and would therefore be classified as a negative
instance (i.e., the tree predicts that Play Tennis —— no).
This tree and the example used in Table 3.2 to illustrate the
ID3 learning algorithm are adapted from (Quinlan 1986).
In general, decision trees represent a disjunction of
conjunctions of con-straints on the attribute values of
instances. Each path from the tree root to a leaf
corresponds to a conjunction of attribute tests, and the
tree itself to a disjunc-tion of these conjunctions. For
example. the decision tree shown in Figure 3. i
corresponds to the expression

( Outlook Sum y H umid it y —- Normal) (Outlook

O nercast
j
v ( Outlook Rain Wind Weak)

(OR)

( Explain the concept of ensemble learning in machine learning. Why are ensemble CO K4 4
b methods used, and how do they improve the performance of individual models? 2
) Discuss the general principle behind combining multiple models to create a more
robust and accurate prediction.

13. Ensemble Methods

QC2
01

putting them together cnn nn informed decision be renched.

Figure 13.1 shows the basic idea of ensemble learning, es thèse
methods are collectively called. Given ri relatively simple bÎnnry
classification problem and some learner thnt puts an ellipse
around a subset of the data, combining the ellipses can provide a
consideraJrly more complex decision boundnry.
There are then only a couple of questions to ask: which

51
 learners should we use, how should we ensure thnt they leorn
different things, and how should we combine their results? The
methods that we are investigating in this chnpter can use any
classifier at all. Although in general they only use one type of
clnssifier et a time, they do not hnve to. A comnion choice of
classifier is the decision tree (see Chapter 12).
Ensuring thnt the leorners see different things con be
performed in different wnys, and it is the primary dilference
between the algorithms that we shell see. However, it con also
come nbout naturnlly depending upon the application oreo.
Suppose thot you hnve lots ond lots of date. In that case you could
simply randomly partition the date und give different sets of dnta
to different clossifiers. Even here there rire choices: do you make
the partitions separate, or include overlaps’? If there is no overlap,
then it could be difficult to work out how to coriibine the
clossifiers, or it might be very simple: if your doctor nlwoys osks for
opinions from two colleagues, one specialising in heart problems
and one in sports injuries,

FIGURE 13.1 By combining lots of simple classifiers (here

that simply put an elliptical decision boundary onto the
data], the decision boundary can be made much more com-
plicated, enabling the difficult separation of the pluses from
the circles.
then upon discovering thnt your leg stnrted hurting nlter you went for u run she
would likely
accord more weight to the diagnosis of the Sports injury expert.

52
 (a Compare EM with K-Means clustering. Highlight differences in their underlying CO K4 4
) assumptions, the ability of EM to handle overlapping clusters, soft clustering vs. hard 3
clustering, and use of probability distributions. Give a real-world application where
the EM algorithm is commonly used.

1. K – Means

QC3
01

53
54
 ( Explain the two main types of hierarchical clustering approaches. CO K4 4
b 3
)  Agglomerative (bottom-up)
 Divisive (top-down)
Include the basic working principles and when each method is typically used.

1. Hierarchical Clustering

QC3
01

55
 (a Consider a training data set consisting of the fauna of the world. Each unit has three CO K4 4
) features named “Swim”, “Fly”, and “Crawl”. Let the possible values of these features 4
QC4 be as follows:
01
Swim Fast, Slow, No
Fly Long, Short, Rarely, No
56
 Crawl Yes, No
For simplicity, each unit is classified as “Animal”, “Bird” or “Fish”. Let the training
data set be as in Table 1.0. Use naïve Bayes algorithm to classify a particular species if
its features are (Slow, Rarely, No)?
[Link] Swim Fly Crawl Class
1 Fast No No Fish
2 Fast No Yes Animal
3 Slow No No Animal
4 Fast No No Animal
5 No Short No Bird
6 No Short No Bird
7 No Rarely No Animal
8 Slow No Yes Animal
9 Slow No No Fish
10 Slow No Yes Fish
11 No Long No Bird
12 Fast No No Bird
Table 1.0: Sample data set for naïve Bayes algorithm
In this example, the features are
F i - “ S w im ” , F 2 - “ F ly ” , F 3 - “ C ra w l” .
The class labels are
1' “Animal”, •2' “ Bird”, •3 • “Fish”.
T he test instance is (S low , R arely, N o ) an d so w e have:
+i - “Slow”, zt = “Rarely”, z3 = “No”.

57
 We construct the frequency table shown in Table 6.2 which summarises the data. (It may be noted
that the construction of the frequency table is not part of the algorithm.)

Features
C la ss Swim (Hi i Fly (N;) Crawl (f3) Total
Fast Slow No Long Short Rarely No Yes No
A n im a l ( i) 2 2 1 0 0 1 4 2 3 5
B ird (‹ 2) 1 0 3 1 2 0 1 1 3 4
Fish (‹ ) 1 2 0 0 0 0 3 0 3 3
Total 4 4 4 1 2 1 8 4 8 12

Table 6.2: Frequency table for the data in Table 6.1

Step 1. We compute following probabilities.
No. of records with class label “Animal”
Total number of examples

No. of records with class label “Bird”

( ) '
T o tal n u m b er o f exam ples
= 4/12
No of records with class label “Fish”
Total number ot examples
= 3/12
Step 2. We construct the following table of conditional probabilities:

Features
Swim ( ) Fly (Ft) Crawl ( t)
Class
F ast S lo w No Long Short Rarcly No Y cs No
A n im a l(rt) 2/5 2/5 1/5 ()/5 ()/5 1/5 4/5 2/5 3/5
B ird (c ,) l/4 0/4 3/4 1/4 2/4 0/4 1/4 0/4 4/4
F ish (c t) 13 2/3 0/3 0/3 0/3 0/3 3/3 0/3 3/3

T a b le 6 .3 : T a b le o f th e c o n d itio n a l p ro b a b ilitie s P f, ct)

Note: The conditional probabilities are calculated as follows:
of records with ’i - Slow and class label
No. of records with class label r t
= 2/5.
Step 3. We now calculate the following numbers:

( CO K4 4
b 4
QC4
) Describe the architecture and working mechanism of Hidden Markov Models
01
(HMMs) as a type of sequence model. What are the primary components of an HMM,
and how does it model sequential dependencies within data?
58
 1. Markov Models – Hidden Markov Models

15.3 Hidden Markov Models

H]DDENMAREOV In a hidden ñfarkov mndef (HMM), the states are not observable, but when
MODEL we visit a state, an observation is recorded that is a probabilisfic function
of the state. We assume a discrete observation in each state from the set
O = ired, red, green, blue, yellow

59
 In the previous case, kno 'ing ihe
observation (ball color), e knew the state
lurnl exactly because there « ere separate urns
for separate crilnrs and cach urn contained
[Link] of onl) one color. The obscri able model
is a special case of the hidden model ›•’here â/
= N and .i >
i' 7

T o su m m a riz e a n d fo rm a liz e , a n H 6 I8 1 h a s th e fo llo w in g e le m e n ts:

1 . N: Number of states in the model

S —— S , S , ... , SN

2 . hI: Number of distinct observation symbols in the alphuhe•t

3 . State transition probabilities:

4 . Obseo ation probabilities:

B=[ (m)]whee (m)-f(0,=v lg=./)

. Initial state probabilities:

60