MACHINE LEARNING

Complete Study Guide — Mumbai University

T.E. Computer Science (AI & ML) | Sem VI | Subject Code: 37474

Based on PYQs: May 2023 · Dec 2023 · May 2024 + Important Questions

■ All Module Answers | ■ Exam Predictions | ■ Numerical Solutions

★★★ Very High Probability ★★■ High Probability ★■■ Medium Probability
■ EXAM PREDICTIONS — Based on PYQ Analysis (May 2023, Dec 2023, May
2024)

Questions that appeared in ALL 3 previous papers are EXTREMELY likely to appear again. Study
these first!

Times
# Topic Module Prediction
Asked

1 Singular Value Decomposition (SVD) + Applications M2 3/3 ★★★

2 Supervised vs Unsupervised Learning M1 3/3 ★★★

3 Error Back Propagation (Block Diagram + Flowchart) M5 3/3 ★★★

4 Activation Functions (Binary/Bipolar/Continuous/Ramp) M5 3/3 ★★★

5 Matrix Diagonalization (Numerical) M2 3/3 ★★★

6 PCA + Curse of Dimensionality M6 3/3 ★★★

7 Least Squares Regression (Theory + Numerical) M3 3/3 ★★★

8 Overfitting and Underfitting M1 2/3 ★★★

9 Performance Metrics (Accuracy/Precision/Recall/F1) M1 2/3 ★★★

10 LMS-Widrow Hoff (Delta Learning Rule) M5 2/3 ★★★

11 Perceptron Neural Network M5 2/3 ★★★

12 ML Definition + Steps in Developing ML Application M1 2/3 ★★★

13 McCulloch Pitts Model (XOR / AND-NOT) M5 2/3 ★★■

14 Multilayer Perceptron (MLP) M5 2/3 ★★■

15 Ridge vs Lasso Regression M3 2/3 ★★■

16 EM Algorithm for Clustering M4 2/3 ★★■

17 MP Neuron Model M5 1/3 ★★■

18 Hebbian Learning Rule M4 2/3 ★★■

19 Feature Selection for Dimensionality Reduction M6 2/3 ★★■

20 OR Perceptron (w1=0.6, w2=1.1, lr=0.5, T=1) M5 2/3 ★★★

21 PCA Numerical (X=[2,1,0,-1], Y=[4,3,1,0.5]) M6 2/3 ★★★

22 SVD Numerical (A=[2,4;1,3] or similar) M2 2/3 ★★★

23 Support Vector Machines M5 1/3 ★★■

24 Trace of Matrix + Properties M2 1/3 ★■■

25 Bias-Variance Trade-off + Cross Validation M1 1/3 ★★■

 MODULE 1 — Introduction to Machine Learning
Q1. Define Machine Learning. Explain the key steps in developing an ML application. [VVVIMP ★★★]
Definition: Machine Learning (ML) is a subfield of Artificial Intelligence that enables computers to learn from data
and improve their performance on tasks without being explicitly programmed. Arthur Samuel (1959) defined ML as:
the field of study that gives computers the ability to learn without being explicitly programmed.
Key Steps in Developing an ML Application:
• 1. Problem Definition: Identify the objective — classification, regression, clustering, etc. Define success
metrics.
• 2. Data Collection: Gather relevant data from databases, APIs, sensors, web scraping, etc.
• 3. Data Preprocessing: Handle missing values, remove outliers, normalize/standardize features, encode
categorical variables.
• 4. Exploratory Data Analysis (EDA): Understand data distributions, correlations, visualize patterns using plots
and statistics.
• 5. Feature Engineering: Select/create relevant features, reduce dimensionality to improve model performance.
• 6. Model Selection: Choose appropriate algorithm (Decision Tree, SVM, Neural Network, etc.) based on
problem type.
• 7. Model Training: Split data into train/test sets (e.g., 80/20). Fit the model on training data.
• 8. Model Evaluation: Evaluate using metrics: Accuracy, Precision, Recall, F1-Score, RMSE, etc.
• 9. Hyperparameter Tuning: Optimize model parameters using Grid Search, Random Search, or
Cross-Validation.
• 10. Deployment: Deploy the model as an API or embedded system. Monitor performance in production.

Q2. Compare Supervised and Unsupervised Learning. [VIMP ★★★]

Feature Supervised Learning Unsupervised Learning

Definition Learns from labeled data Learns from unlabeled data

Output Predicts a label/value Discovers hidden patterns

Training Data Input + Output pairs Input data only

Goal Prediction/Classification Clustering/Association

Examples Linear Regression, SVM, Decision Tree K-Means, PCA, Apriori

Feedback Direct feedback via labels No direct feedback

Complexity Simpler, needs labeled data Complex, no ground truth

Use Case Spam detection, Price prediction Customer segmentation, Anomaly detection

Q3. Write a short note on Overfitting and Underfitting. [VIMP ★★★]

Overfitting: The model learns training data too well — including noise — and performs poorly on new/test data. It
has LOW bias but HIGH variance.
• Signs: High training accuracy, low test accuracy.
• Causes: Too complex model, insufficient training data, too many features.
• Solutions: Regularization (L1/L2), cross-validation, pruning, dropout, more data.

Underfitting: The model is too simple to capture the underlying patterns. It has HIGH bias and LOW variance.
• Signs: Low accuracy on both training and test data.
• Causes: Too simple model, too few features, insufficient training.
• Solutions: Increase model complexity, add more features, reduce regularization, train longer.

Ideal Model: Low bias + Low variance → Generalizes well to unseen data. This is achieved by
finding the right balance using techniques like cross-validation.

Q4. How is Quality of a Model Measured? Explain Performance Evaluation Metrics. [★★★]
A Confusion Matrix forms the basis of classification metrics:

Predicted Positive Predicted Negative

Actual Positive TP (True Positive) FN (False Negative)

Actual Negative FP (False Positive) TN (True Negative)

Metric Formula Interpretation

Accuracy (TP + TN) / (TP + TN + FP + FN)Overall correctness of the model

Precision TP / (TP + FP) Of all predicted positives, how many are actually positive?

Recall (Sensitivity) TP / (TP + FN) Of all actual positives, how many were correctly predicted?

F1-Score Harmonic +
2 × (Precision × Recall) / (Precision mean of Precision and Recall
Recall)

Specificity TN / (TN + FP) True Negative Rate

Q5. Discuss ML Approaches. Explain Cross Validation + Bias-Variance Trade-off. [★★■]

ML Approaches:
• Supervised Learning: Labeled data → Regression, Classification
• Unsupervised Learning: Unlabeled data → Clustering, Dimensionality Reduction
• Semi-Supervised: Mix of labeled and unlabeled data
• Reinforcement Learning: Agent learns via reward/punishment feedback

Cross Validation (K-Fold): Dataset is split into K equal parts. Model is trained on K-1 folds and tested on the
remaining fold. This repeats K times and results are averaged. Reduces overfitting and gives reliable performance
estimate.

Bias-Variance Trade-off:
• Bias: Error from wrong assumptions. High bias → Underfitting.
• Variance: Sensitivity to training data fluctuations. High variance → Overfitting.
• Total Error = Bias2 + Variance + Irreducible Noise
Goal: Find sweet spot where both bias and variance are low → Best generalization.
 MODULE 2 — Mathematical Foundation for ML
Q1. Explain Singular Value Decomposition (SVD) and its Applications. [VIMP ★★★]
Definition: SVD decomposes any real matrix A (m×n) into three matrices:
A = U · Σ · VT

• U (m×m): Left singular vectors — orthogonal matrix (columns = eigenvectors of AAT)

• Σ (m×n): Diagonal matrix of singular values (σ■ ≥ σ■ ≥ ... ≥ 0)
• VT (n×n): Right singular vectors — orthogonal matrix (rows = eigenvectors of ATA)

Steps to Compute SVD:

• 1. Compute ATA and find its eigenvalues λ■, λ■, ...
• 2. Singular values: σ■ = √λ■ (form diagonal matrix Σ)
• 3. V = eigenvectors of ATA (columns, normalized)
• 4. U = eigenvectors of AAT, or compute: u■ = Av■ / σ■

Applications of SVD:
• Dimensionality Reduction (PCA uses SVD internally)
• Image Compression (keep top-k singular values)
• Recommender Systems (Netflix, Amazon — Collaborative Filtering)
• Noise Reduction in signals and data
• Pseudo-inverse computation (A■ = V Σ■ UT)
• Latent Semantic Analysis (LSA) in Natural Language Processing
• Solving systems of linear equations

Q2. Define Trace of a Matrix and Explain its Properties. [★■■]

Definition: The trace of a square matrix A is the sum of its diagonal elements.
tr(A) = Σ a■■ = a■■ + a■■ + ... + a■■

Properties:
• tr(A) = Sum of eigenvalues of A
• tr(A + B) = tr(A) + tr(B) [Linearity]
• tr(cA) = c·tr(A) [Scalar multiplication]
• tr(AT) = tr(A) [Transpose invariance]
• tr(AB) = tr(BA) [Cyclic property]
• tr(ABC) = tr(BCA) = tr(CAB) [Cyclic permutation]
 ■ NUMERICALS — MODULE 2 (Matrix Operations)
NUMERICAL 1: Diagonalize Matrix A = [[4, 1], [2, 3]] (Dec 2023 PYQ ★★★)
Step 1: Find Eigenvalues — Solve det(A − λI) = 0
det([[4−λ, 1],[2, 3−λ]]) = (4−λ)(3−λ) − (1)(2) = 0
= λ² − 7λ + 12 − 2 = λ² − 7λ + 10 = 0
(λ − 5)(λ − 2) = 0 → λ■ = 5, λ■ = 2

Step 2: Find Eigenvectors

For λ■ = 5: (A − 5I)v = 0 → [[-1, 1],[2, -2]]v = 0 → v■ = [1, 1]T
For λ■ = 2: (A − 2I)v = 0 → [[2, 1],[2, 1]]v = 0 → v■ = [1, -2]T

Step 3: Form P and D

P = [[1, 1],[1, -2]] D = [[5, 0],[0, 2]] A = P·D·P■¹

NUMERICAL 2: SVD of A = [[2, 4], [1, 3]] (May 2023, May 2024 PYQ ★★★)
Step 1: Compute ATA
AT = [[2,1],[4,3]], ATA = [[2,1],[4,3]]·[[2,4],[1,3]] = [[5,11],[11,25]]

Step 2: Eigenvalues of ATA

det(ATA − λI) = (5−λ)(25−λ) − 121 = λ² − 30λ + 4 = 0
λ = (30 ± √(900−16))/2 = (30 ± √884)/2
λ■ ≈ 29.866, λ■ ≈ 0.134

Step 3: Singular values: σ■ = √29.866 ≈ 5.465, σ■ = √0.134 ≈ 0.366

Step 4: V = normalized eigenvectors of ATA
For λ■ ≈ 29.866: v■ ≈ [0.404, 0.915]T
For λ■ ≈ 0.134: v■ ≈ [0.915, −0.404]T

Step 5: U — u■ = Av■/σ■, u■ = Av■/σ■

A = U · Σ · VT where Σ = diag(5.465, 0.366)

■ Online SVD Calculator: [Link] (search: 'SVD of matrix [[2,4],[1,3]]')

NUMERICAL 3: Diagonalize A = [[1,5],[4,2]] (Dec 2023 PYQ ★★★)

det(A − λI) = (1−λ)(2−λ) − 20 = λ² − 3λ − 18 = 0
(λ − 6)(λ + 3) = 0 → λ■ = 6, λ■ = −3
For λ■ = 6: v■ = [1, 1]T; For λ■ = −3: v■ = [−5, 4]T
D = [[6, 0],[0, -3]], P = [[1, -5],[1, 4]]
 MODULE 3 — Linear Models & Regression
Q1. Explain Least-Squares Regression for Classification. [VVVIMP ★★★]
Least Squares Regression finds a line (or hyperplane) that minimizes the sum of squared differences between
observed and predicted values.
Objective: Minimize Σ(y■ − ■■)² = Σ(y■ − (ax■ + b))²

Normal Equations:
• Slope: a = [n·Σ(x■y■) − Σx■·Σy■] / [n·Σx■² − (Σx■)²]
• Intercept: b = (Σy■ − a·Σx■) / n

In Matrix Form: The solution is given by the Normal Equation:

w = (XTX)■¹ · XTy

For Classification: Least squares can be extended to classification by encoding class labels as 0/1 and fitting a
regression line, then using a threshold (e.g., 0.5) to classify. However, it is less suited than logistic regression for
classification due to sensitivity to outliers.

Q2. Compare Ridge Regression and Lasso Regression. [VIMP ★★■]

Feature Ridge Regression (L2) Lasso Regression (L1)

Regularization Term λ·Σw■² λ·Σ|w■|

Cost Function RSS + λ·Σw■² RSS + λ·Σ|w■|

Effect on Weights Shrinks weights toward zero Can make weights exactly zero

Feature Selection No — all features retained Yes — performs feature selection

Handles Multicollinearity Yes (well) Partially

Best for All features relevant Sparse models, many irrelevant features

Solution Always unique (closed form) May not be unique

Q3. Short Notes: (a) Multivariate Regression (b) Regularized Regression [VIMP ★★■]
(a) Multivariate Regression: Extension of simple linear regression to multiple independent variables.
y = w■ + w■x■ + w■x■ + ... + w■x■ + ε

• Uses matrix notation: y = Xw, solved by w = (XTX)■¹XTy

• Example: Predicting house price from size, location, number of rooms.

(b) Regularized Regression: Adds a penalty term to the loss function to prevent overfitting by discouraging large
weights.
• L2 (Ridge): Loss = RSS + λ·Σw■² → Shrinks coefficients
• L1 (Lasso): Loss = RSS + λ·Σ|w■| → Sparse coefficients (feature selection)
• Elastic Net: Combination of L1 and L2 penalties
• λ (lambda) is the regularization parameter — controls trade-off between fit and complexity
 ■ NUMERICALS — MODULE 3 (Linear Regression)
NUMERICAL 1: Fit a line Y = aX + b for X=[1,2,3], Y=[2,4,6] (May 2023, May 2024, Dec 2023 ★★★)
Setup table:

n X Y X² XY

1 1 2 1 2

2 2 4 4 8

3 3 6 9 18

Σ 6 12 14 28

n=3, ΣX=6, ΣY=12, ΣX²=14, ΣXY=28

a = (n·ΣXY − ΣX·ΣY) / (n·ΣX² − (ΣX)²)
a = (3×28 − 6×12) / (3×14 − 36) = (84 − 72) / (42 − 36) = 12/6 = 2
b = (ΣY − a·ΣX) / n = (12 − 2×6) / 3 = 0/3 = 0
Regression Line: Y = 2X + 0 → Y = 2X

NUMERICAL 2: Least Squares for X=[0,1,2,3,4], Y=[2,3,5,4,6] (May 2023 PYQ ★★★)

X Y X² XY

0 2 0 0

1 3 1 3

2 5 4 10

3 4 9 12

4 6 16 24

Σ=10 Σ=20 Σ=30 Σ=49

n=5, ΣX=10, ΣY=20, ΣX²=30, ΣXY=49

a = (5×49 − 10×20) / (5×30 − 100) = (245 − 200)/(150 − 100) = 45/50 = 0.9
b = (20 − 0.9×10)/5 = (20−9)/5 = 11/5 = 2.2
Y = 0.9X + 2.2 → When X=10: Y = 0.9×10 + 2.2 = 9+2.2 = 11.2

■ Reference: [Link]

NUMERICAL 3: Linear Regression for X=[3,5,7,9], Y=[12,18,24,30] (Dec 2023 PYQ ★★★)

X Y X² XY

3 12 9 36

5 18 25 90

7 24 49 168

9 30 81 270

Σ=24 Σ=84 Σ=164 Σ=564

n=4, ΣX=24, ΣY=84, ΣX²=164, ΣXY=564

a = (4×564 − 24×84)/(4×164 − 576) = (2256−2016)/(656−576) = 240/80 = 3
b = (84 − 3×24)/4 = (84−72)/4 = 12/4 = 3
Y = 3X + 3
 ■ NUMERICAL — Module 1: Performance Metrics
NUMERICAL: TN=300, TP=500, FP=50, FN=150 → Find Accuracy, Precision, Recall, F1 (May 2024 +
Imp ★★★)
Given: TP=500, TN=300, FP=50, FN=150
Total = TP + TN + FP + FN = 500+300+50+150 = 1000

Metric Formula Calculation Result

Accuracy (TP+TN)/Total (500+300)/1000 0.80 = 80%

Precision TP/(TP+FP) 500/(500+50) 500/550 ≈ 0.909 = 90.9%

Recall TP/(TP+FN) 500/(500+150) 500/650 ≈ 0.769 = 76.9%

F1-Score 2×P×R/(P+R) 2×0.909×0.769/(0.909+0.769) ≈ 0.833 = 83.3%

Final Answers: Accuracy=80%, Precision=90.9%, Recall=76.9%, F1=83.3%

 MODULE 4 — Clustering
Q1. Explain Hebbian Learning Rule. [★★■]
Hebbian Learning is one of the oldest and simplest learning rules in neural networks, proposed by Donald Hebb
(1949). It is based on the biological principle:
"Neurons that fire together, wire together."

Weight Update Rule:

∆w■■ = η · x■ · y■ → w■■(new) = w■■(old) + ∆w■■

• η = learning rate
• x■ = input signal
• y■ = output signal

Key Features:
• Unsupervised learning — no target output needed
• Works with bipolar (+1/-1) or binary (0/1) inputs
• Correlation-based learning
• Used to design associative memories and pattern recognition
Limitation: Weights can grow unboundedly (no forgetting mechanism).

Q2. Explain the Expectation-Maximization (EM) Algorithm for Clustering. [★★■]

EM Algorithm is an iterative optimization method for finding maximum likelihood estimates when data has missing
or latent variables. It is widely used in Gaussian Mixture Models (GMM).
Two Steps:
• E-Step (Expectation): Calculate the probability (responsibility) that each data point belongs to each cluster,
given current parameter estimates.
• M-Step (Maximization): Update the model parameters (means, covariances, mixing coefficients) to maximize
the expected log-likelihood computed in E-step.

Repeat E → M until convergence (log-likelihood stops increasing)

Algorithm Steps:
• 1. Initialize parameters (means, variances, mixing weights)
• 2. E-Step: r■■ = P(z■=k | x■, θ) — posterior probability
• 3. M-Step: Update µ■, Σ■, π■ using weighted averages
• 4. Compute log-likelihood; check convergence
• 5. Repeat steps 2-4 until convergence
Advantages: Handles overlapping clusters, soft cluster assignments, guaranteed to increase likelihood each
iteration.
 MODULE 5 — Classification Models & Neural Networks
Q1. Explain the MP Neuron Model. [★★■]
McCulloch-Pitts (MP) Neuron Model (1943) is the earliest mathematical model of a biological neuron.
Components:
• Multiple binary inputs: x■, x■, ..., x■ (0 or 1)
• Each input is either excitatory or inhibitory
• Threshold θ determines when neuron fires

Output y = 1 if Σx■ ≥ θ, else y = 0

Rules:
• If any inhibitory input = 1, neuron does NOT fire (output = 0)
• If sum of excitatory inputs ≥ θ, neuron fires (output = 1)

Q2. Explain the Perceptron Neural Network. [VIMP ★★★]

Perceptron (Rosenblatt, 1958) is the simplest feedforward neural network — a single-layer binary classifier.
Structure: Inputs (x■...x■) → Weights (w■...w■) → Summation → Activation → Output
Net input: net = Σw■x■ + bias Output: y = f(net) (step function)

Perceptron Learning Rule:

w■(new) = w■(old) + η·(target − output)·x■

Training Algorithm:
• 1. Initialize weights and bias (small random values or zero)
• 2. For each training sample: compute output = f(Σw■x■ + b)
• 3. Update weights: w■ = w■ + η(t − y)x■
• 4. Repeat until convergence (no weight changes)
Limitations: Only solves linearly separable problems (cannot solve XOR).

Q3. Explain the Perceptron Model with Bias. [★★★]

The bias is an additional weight (w■) connected to a constant input (x■ = 1).
net = w■·x■ + w■x■ + w■x■ + ... = b + Σw■x■

Role of Bias:
• Shifts the decision boundary away from origin
• Allows network to represent functions that don't pass through origin
• Increases the flexibility and expressiveness of the model
• Updated same way as other weights: b = b + η(t − y)

Q4. What are Activation Functions? Explain Binary, Bipolar, Continuous, Ramp. [★★★]
An activation function determines the output of a neuron given its input. It introduces non-linearity.

Function Formula Range Graph Shape

Binary Step f(x) = 1 if x≥θ, else 0 [0, 1] Step at x=θ

Bipolar Step f(x) = +1 if x≥θ, else -1 [-1, +1] Step at x=θ

 Sigmoid f(x) = 1/(1+e■■) (0, 1) S-curve

Ramp (Linear) [0, x>1

f(x)=0 if x<0; x if 0≤x≤1; 1 if 1] Ramp shape

Tanh f(x) = (e■−e■■)/(e■+e■■) (-1, +1) S-curve centered 0

ReLU f(x) = max(0, x) [0, ∞) Half-line from origin

Q5. Draw Block Diagram of Error Back Propagation Algorithm + Flowchart. [★★★]
Backpropagation is the most widely used training algorithm for multilayer neural networks. It uses the chain rule
of calculus to compute gradients.
Forward Pass: Input → Hidden Layer(s) → Output Layer → Compute Error
Backward Pass: Compute gradients from output back to input; update weights.

Error E = ½·Σ(t■ − o■)² where t■ = target, o■ = actual output

Weight Update for Output Layer:

δ■ = o■(1−o■)(t■−o■) ∆w■■ = η·δ■·o■

Weight Update for Hidden Layer:

δ■ = o■(1−o■)·Σ(δ■·w■■) ∆w■■ = η·δ■·x■

Algorithm Steps:
• 1. Initialize weights randomly
• 2. Forward pass: compute outputs at each layer
• 3. Compute output error: δ■ = o■(1−o■)(t■−o■)
• 4. Backpropagate error to hidden layers
• 5. Update all weights: w = w + η·δ·input
• 6. Repeat for all training patterns until error is minimized

Q6. Delta Learning Rule — LMS-Widrow Hoff. [★★★]

LMS (Least Mean Squares) or Widrow-Hoff rule minimizes mean squared error between target and actual output
using gradient descent.
∆w■ = η·(t■ − net■)·x■■ where net■ = Σw■x■■

Key Difference from Perceptron: Uses the net input (not the thresholded output) in the error signal — gives
continuous error, making convergence smoother.
Training Steps:
• 1. Initialize weights to small random values
• 2. Apply input pattern, compute net = Σw■x■
• 3. Compute error: e = t − net
• 4. Update: w■ = w■ + η·e·x■
• 5. Repeat until error < threshold

Q7. Discuss Support Vector Machines (SVM). [★★■]

SVM is a supervised learning algorithm that finds the optimal hyperplane to separate classes with maximum
margin.
Decision boundary: w·x + b = 0 Margin = 2/||w||
Key Concepts:
• Support Vectors: Data points closest to the decision boundary
• Margin: Distance between decision boundary and nearest support vectors — SVM maximizes this
• Kernel Trick: Maps data to higher dimensions to find linear separator: Linear, Polynomial, RBF kernels
• Soft Margin: Allows some misclassification (C parameter controls trade-off)
Advantages: Effective in high dimensions, works well with small datasets, robust to outliers.

Q8. Explain Multilayer Perceptron (MLP) with neat diagram. [★★■]

MLP is a feedforward neural network with one or more hidden layers between input and output.
Architecture: Input Layer → [Hidden Layer 1 → Hidden Layer 2 → ...] → Output Layer
• Each layer is fully connected to the next
• Non-linear activation functions (sigmoid, tanh, ReLU) in hidden layers
• Trained using Backpropagation algorithm
Universal Approximation Theorem: An MLP with at least one hidden layer and sufficient neurons can
approximate any continuous function.
Working:
• Forward pass computes activations layer by layer
• Loss computed at output layer
• Backpropagation computes gradients and updates weights
 ■ NUMERICALS — MODULE 5 (Neural Networks)
NUMERICAL 1: Implement XOR using McCulloch-Pitts Model (May 2024 PYQ ★★★)
XOR Truth Table:

x■ x■ x■ XOR x■

0 0 0

0 1 1

1 0 1

1 1 0

XOR = (x■ OR x■) AND NOT(x■ AND x■)

XOR requires 2 MP neurons (it's not linearly separable with one neuron):
• Neuron z■: Implements OR: weights w■=w■=1, θ=1. Output z■=1 if x■+x■≥1
• Neuron z■: Implements AND: weights w■=w■=1, θ=2. Output z■=1 if x■+x■≥2
• Neuron y: Implements AND-NOT: w(z■)=1 (excitatory), w(z■)=1 (inhibitory), θ=1
• y = z■ AND NOT(z■) = XOR
Network: y = OR(x■,x■) AND NOT(AND(x■,x■)) → This correctly implements XOR

NUMERICAL 2: Implement AND-NOT using McCulloch-Pitts (May 2024 PYQ ★★■)

AND-NOT Truth Table:

x■ x■ (inhibitory) AND-NOT (x■ AND NOT x■)

0 0 0

0 1 0

1 0 1

1 1 0

Network Design:
• x■: excitatory input, weight = 1
• x■: inhibitory input (if x■=1, neuron cannot fire)
• Threshold θ = 1
Output y = 1 only when x■=1 AND x■=0 (threshold met, no inhibition)

NUMERICAL 3: OR Gate using Single-Layer Perceptron (w1=0.6, w2=1.1, lr=0.5, θ=1) (May 2024
★★★)
OR Truth Table: (1,1)→1, (1,0)→1, (0,1)→1, (0,0)→0
Initial Weights: w■=0.6, w■=1.1, b=0 (assume), η=0.5, Threshold=1
Activation: y = 1 if net ≥ 1, else y = 0; Update: w■ = w■ + η(t−y)x■

x■ x■ t net=w■x■+w■x■ y Error(t-y) ∆w■ ∆w■ w■(new) w■(new)

1 1 1 0.6+1.1=1.7 1 0 0 0 0.6 1.1

1 0 1 0.6 0 1 0.5 0 1.1 1.1

 0 1 1 1.1 1 0 0 0 1.1 1.1

0 0 0 0 0 0 0 0 1.1 1.1

After Epoch 1: w■=1.1, w■=1.1. Continue epochs until no weight change occurs (convergence).

■ Perceptron simulator: [Link]

 MODULE 6 — Dimensionality Reduction
Q1. What is the Curse of Dimensionality? Explain PCA in detail. [VIMP ★★★]
Curse of Dimensionality: As the number of features (dimensions) increases, the data becomes increasingly
sparse in the feature space. This causes:
• Exponential increase in data needed to maintain density
• Distance measures become meaningless in high dimensions
• Models overfit easily, computation becomes expensive
• Most data points become equidistant from each other

Principal Component Analysis (PCA):

PCA is an unsupervised linear dimensionality reduction technique that transforms data to a new coordinate system
where features have maximum variance.
Steps of PCA:
• 1. Standardize data: Subtract mean, divide by std. deviation for each feature
• 2. Compute Covariance Matrix: C = (1/n) · XTX
• 3. Find Eigenvalues and Eigenvectors of C
• 4. Sort eigenvectors by decreasing eigenvalue
• 5. Select top-k eigenvectors (principal components)
• 6. Transform data: X_new = X · W (W = matrix of selected eigenvectors)

Variance explained by component i = λ■ / Σλ × 100%

Applications: Image compression, face recognition, noise reduction, visualization of high-dimensional data.

Q2. Feature Selection Methods for Dimensionality Reduction. [VIMP ★★■]

Feature selection removes irrelevant/redundant features without transforming them.
1. Filter Methods — evaluate features independently of the model:
• Correlation coefficient, Chi-squared test, Mutual information
• Fast but ignores feature interactions
2. Wrapper Methods — use model performance to select features:
• Forward Selection, Backward Elimination, Recursive Feature Elimination (RFE)
• Better accuracy but computationally expensive
3. Embedded Methods — feature selection built into the model:
• Lasso (L1) regularization, Decision Tree feature importance
• Balance between filter and wrapper methods
 ■ NUMERICAL — MODULE 6: PCA
NUMERICAL: Apply PCA on X=[2,1,0,-1], Y=[4,3,1,0.5] (May 2024, Imp Questions ★★★)
Step 1: Compute Means
X■ = (2+1+0+(−1))/4 = 2/4 = 0.5
■ = (4+3+1+0.5)/4 = 8.5/4 = 2.125

Step 2: Center the Data

Point X Y x = X−X■ y = Y−■

1 2 4 1.5 1.875

2 1 3 0.5 0.875

3 0 1 -0.5 -1.125

4 -1 0.5 -1.5 -1.625

Step 3: Compute Covariance Matrix

Cov(x,x) = Σx²/(n-1) = (1.5² + 0.5² + (−0.5)² + (−1.5)²)/3 = (2.25+0.25+0.25+2.25)/3 = 5/3 ≈ 1.667
Cov(y,y) = Σy²/(n-1) = (1.875²+0.875²+(−1.125)²+(−1.625)²)/3 = (3.516+0.766+1.266+2.641)/3 ≈ 2.729
Cov(x,y) = Σxy/(n-1) = (1.5×1.875 + 0.5×0.875 + (−0.5)(−1.125) + (−1.5)(−1.625))/3
= (2.8125 + 0.4375 + 0.5625 + 2.4375)/3 = 6.25/3 ≈ 2.083
Covariance Matrix C = [[1.667, 2.083],[2.083, 2.729]]

Step 4: Find Eigenvalues

det(C − λI) = (1.667−λ)(2.729−λ) − 2.083² = 0
λ² − 4.396λ + (1.667×2.729 − 4.339) = 0
λ² − 4.396λ + (4.549 − 4.339) = 0
λ² − 4.396λ + 0.21 = 0
λ = (4.396 ± √(19.324 − 0.84))/2 = (4.396 ± √18.484)/2 = (4.396 ± 4.299)/2
λ■ ≈ 4.348, λ■ ≈ 0.048

Step 5: Find Principal Component (Eigenvector for λ■ ≈ 4.348)

(C − λ■I)v = 0 → [[-2.681, 2.083],[2.083, -1.619]]v = 0
Solving: v■ ≈ [0.618, 0.786]T (normalized)
First Principal Component ≈ [0.618, 0.786] (captures ~98.9% variance)

■ PCA tool: [Link] (search: 'eigenvalues of [[1.667,2.083],[2.083,2.729]]')

■ PCA calculator: [Link]
■ QUICK REVISION CHEAT SHEET — Last Day Study
Topic Key Formula / Key Point

ML Definition System that learns from experience E, task T, performance P

Accuracy (TP+TN)/(TP+TN+FP+FN)

Precision TP/(TP+FP) — 'When positive predicted, how often correct?'

Recall TP/(TP+FN) — 'Of all positives, how many found?'

F1-Score 2×P×R/(P+R) — Harmonic mean

SVD A = UΣV<super>T</super>; σ■ = √(eigenvalue of A<super>T</super>A)

Diagonalization A = PDP■¹; D = diagonal eigenvalue matrix

Least Squares a = (nΣXY−ΣXΣY)/(nΣX²−(ΣX)²), b = (ΣY−aΣX)/n

Ridge Regression Loss = RSS + λΣw■² (L2 regularization)

Lasso Regression Loss = RSS + λΣ|w■| (L1 regularization, feature selection)

Perceptron Update w■ = w■ + η(t−y)x■

Hebbian Rule ∆w■■ = η·x■·y■

LMS/Widrow-Hoff ∆w■ = η·(t−net)·x■ (uses net input, not output)

Backpropagation δ■=o■(1−o■)(t■−o■); δ■=o■(1−o■)·Σδ■w■■

PCA Steps Mean subtract → Covariance Matrix → Eigendecompose → Select top-k

Bias-Variance Total Error = Bias² + Variance + Noise

K-Fold CV Split into K folds; train on K-1, test on 1; average K results

MP Neuron y=1 if Σx■≥θ AND no inhibitory input active

SVM Margin Maximize margin = 2/||w||; support vectors on margin boundary

EM Algorithm E-Step: compute responsibilities; M-Step: update parameters

Curse of Dim. Data becomes sparse; distance measures fail in high dimensions

■ USEFUL ONLINE RESOURCES FOR NUMERICALS: • Matrix operations: [Link] •

SVD/Eigenvalues: [Link] • Linear Regression:
[Link] • PCA Calculator:
[Link] • Neural Network viz: [Link] •
Matrix calculator: [Link]

Machine Learning Study Guide — T.E. CSE (AI&ML;) Sem VI | Mumbai University | Paper Code: 37474 | Based on PYQ Analysis