ANN_Module2

– Part A
Q1) Derive the Least Mean Square (LMS) algorithm from first
principles.
The LMS algorithm is a stochastic gradient descent method used to minimize the
Mean Square Error (MSE) between desired and actual outputs.

J(w) = (1/2) * (d - w^T x)^2

1. Gradient:

∂J/∂w = -e * x

1. Update rule:

w_new = w_old + η * e * x

ANN_Module2 1
 This is the LMS weight update rule.

Q2) Apply the LMS algorithm to solve a basic adaptive filtering

problem with two inputs.
Suppose inputs are x=[x1,x2]x = [x_1, x_2], weights w=[w1,w2]w = [w_1, w_2],
target dd.

1. Compute output:

y = w1*x1 + w2*x2

1. Error:

e=d-y

1. Update weights:

w1_new = w1_old + η * e * x1
w2_new = w2_old + η * e * x2

This updates weights iteratively until convergence.

Q3) Design a single-layer perceptron to implement a logical

NAND gate. Show weight updates using learning rule.
NAND truth table:

x1 x2 → y
0 0 →1
0 1 →1
1 0 →1
1 1 →0

Weight Update Formula:

ANN_Module2 2
 w_new = w_old + η * (t - y) * x

Possible solution after training:

w1 = -1, w2 = -1, b = 1.5

Decision function:

y = 1 if (w1*x1 + w2*x2 + b >= 0) else 0

Q4) Analyze the effect of varying learning rate on the

convergence of perceptron training.
Small η (learning rate): Converges slowly, requires many iterations.

Large η: Converges faster initially, but may overshoot and oscillate, preventing
convergence.

Optimal η: Balanced value ensures stability and speed. Often chosen

experimentally or with annealing schedules.

Q5) Explain the significance of the Perceptron Convergence

Theorem.
The Perceptron Convergence Theorem states:

If the training data is linearly separable, the perceptron algorithm will

converge to a solution (set of weights) in a finite number of steps.

If the data is not linearly separable (e.g., XOR), the perceptron will never
converge.

Significance: Provides a theoretical guarantee of perceptron success under

certain conditions.

Q6) Compare the performance of LMS and Linear Least Square

(LLS) filters for noisy signal data.

ANN_Module2 3
 LLS (Analytical):

Exact solution using normal equations.

Computationally expensive (matrix inversion).

Not adaptive; requires stationary data.

LMS (Iterative):

Approximate solution using gradient descent.

Efficient and adaptive in real-time.

Handles noisy, time-varying environments.

Conclusion: LMS is preferred for noisy, dynamic signals.

Q7) Solve the XOR problem and explain why a single-layer

perceptron fails to model it.
XOR truth table:

x1 x2 → y
0 0 →0
0 1 →1
1 0 →1
1 1 →0

XOR is not linearly separable.

No single straight line can separate outputs correctly in the (x1, x2) plane.

Therefore, a single-layer perceptron fails.

Solution: Use multilayer perceptron with non-linear activation + hidden layer.

Q8) Describe the relation between perceptron output and Bayes

classifier under Gaussian assumptions.
Bayes Classifier: Chooses class with maximum posterior probability.

ANN_Module2 4
 P(Ck | x) = ( P(x | Ck) * P(Ck) ) / P(x)

If classes have Gaussian distribution with equal covariance, the decision

boundary is linear.

In this case, the Bayes classifier ≡ Perceptron (both give same linear
separation).

Thus, perceptron can approximate Bayes decision in Gaussian environment.

Q9) Implement a perceptron learning algorithm using a small

dataset and plot the learning curve.
Algorithm Steps:

1. Initialize weights randomly.

2. For each training sample:

y = step( w^T x + b )
w_new = w_old + η * (t - y) * x
b_new = b_old + η * (t - y)

3. Repeat until all samples are correctly classified.

Learning Curve: Plot of error (misclassifications) vs epochs. Typically decreases

until convergence.

Q10) Propose modifications to improve the classification

performance of a single-layer perceptron for a non-linear
dataset.
1. Use Multilayer Perceptron (MLP): Add hidden layers with nonlinear
activations.

2. Kernel Methods: Map inputs to higher-dimensional feature space.

3. Feature Engineering: Create polynomial or interaction terms.

4. Adaptive Learning Rate: Improve training stability.

ANN_Module2 5
 Module 2 –
Q1) Explain the concept and architecture of a single-layer
perceptron.
A single-layer perceptron is the simplest type of artificial neural network,
introduced by Rosenblatt in 1958. It consists of an input layer directly connected
to an output layer, without any hidden layers. The perceptron is a binary linear
classifier that can separate data that is linearly separable.

Architecture:

Inputs (x₁, x₂, …, xₙ): Represent features of the data.

Weights (w₁, w₂, …, wₙ): Each input has an associated weight that determines
its influence.

Summation Function: Computes a weighted sum:

u=∑i=1nwixi+bu = \sum_{i=1}^n w_i x_i + b

Activation Function (step function): Produces output based on threshold:

y=f(u)={1if u≥00otherwisey = f(u) =

\begin{cases}
1 & \text{if } u \geq 0 \\
0 & \text{otherwise}
\end{cases}

Output (y): Final classification decision.

Diagram (Single-Layer Perceptron):

x1 ---- w1 \
x2 ---- w2 >----( Σ + b )----[Step f()]----> y
x3 ---- w3 /

Working:

ANN_Module2 6
 If the weighted sum exceeds the threshold, the perceptron outputs 1 (class A),
otherwise 0 (class B). It thus implements linear decision boundaries like lines (2D)
or hyperplanes (nD).

Q2) Derive the LMS algorithm used in adaptive filtering

problems.
The Least Mean Square (LMS) algorithm is a gradient descent-based method
used for adaptive filtering, introduced by Widrow and Hoff. It minimizes the Mean
Squared Error (MSE) between the desired output and the actual output.

Explanation:

At each iteration, the weights are adjusted in the direction opposite to the gradient
of the error. The learning rate η\eta controls step size.

Q3) Describe the perceptron learning rule and the convergence

theorem.
Perceptron Learning Rule:

ANN_Module2 7
 Proposed by Rosenblatt, it updates weights whenever the perceptron misclassifies
an input.

If classification is correct → no update.

If classification is wrong → update weights:

wnew=wold+η(t−y)xw_{new} = w_{old} + \eta (t - y)x

where tt = target output, yy = actual output.

Convergence Theorem:
The Perceptron Convergence Theorem states that if the training data is linearly
separable, the perceptron learning algorithm is guaranteed to converge to a set of
weights that correctly classifies all training samples in finite steps.
If the data is not linearly separable (e.g., XOR problem), the perceptron will never
converge.

Q4) Implement a perceptron for AND logic function with all

calculations.
The AND logic outputs 1 only when both inputs are 1.
Truth Table:

x1 x2 → y (Target)
0 0 → 0
0 1 → 0
1 0 → 0
1 1 → 1

ANN_Module2 8
 This correctly models AND logic.

Q5) Explain Linear Least Square Filters and compare with LMS
algorithm.
Linear Least Squares (LLS) Filters:
They minimize the squared error between desired and actual outputs. The optimal
weights woptw_{opt} are obtained analytically using the normal equation:
wopt=(XTX)−1XTdw_{opt} = (X^T X)^{-1} X^T d
where XX = input matrix, dd = target vector.

Comparison with LMS:

LLS: Exact solution, but computationally expensive (O(n3)O(n^3) for

inversion).

LMS: Iterative approximation using gradient descent; computationally efficient,

suitable for real-time adaptive filtering.

LLS is deterministic; LMS adapts dynamically to changing environments.

Q6) What are Learning Curves? Explain their use in

performance evaluation.
A Learning Curve is a graphical representation showing how a model’s
performance improves with experience (training data or epochs).

ANN_Module2 9
 Types of Learning Curves:

1. Training Error Curve: Error on training data decreases as epochs increase.

2. Validation Error Curve: Error on unseen data may first decrease, then
increase due to overfitting.

Use in Evaluation:

Helps identify underfitting (high error, no improvement).

Helps detect overfitting (training error low, validation error high).

Guides early stopping to prevent over-training.

Useful for comparing models and tuning hyperparameters.

Q7) Discuss Learning Rate Annealing Techniques and their

impact.
Learning Rate (η): Controls step size in weight updates. A constant high η can
cause divergence, while a very low η slows convergence.
Learning Rate Annealing: Gradually decreases the learning rate during training.

Impact:

Ensures fast initial learning and fine convergence later.

Reduces oscillations near minimum.

Improves generalization.

ANN_Module2 10
 Q8) Solve the XOR problem using multilayer perceptron and
backpropagation.
The XOR problem is not linearly separable; a single-layer perceptron fails. A
multilayer perceptron (MLP) with one hidden layer solves it.
Architecture:

Input layer: 2 neurons (x1, x2).

Hidden layer: 2 neurons with non-linear activation (sigmoid/tanh).

Output layer: 1 neuron.

Training:

1. Forward pass: Compute activations using weighted sums + sigmoid.

2. Error calculation: Compare output with target.

3. Backpropagation: Adjust weights using gradient descent.

Result:
MLP can learn XOR mapping:

x1 x2 → Output
0 0 →0
0 1 →1
1 0 →1
1 1 →0

Q9) Compare Bayes classifier with perceptron in a Gaussian

environment.
Bayes Classifier:

Probabilistic model based on posterior probabilities:

ANN_Module2 11
 Minimizes classification error by assigning x to the class with highest
posterior.

In Gaussian environment (normal distribution), decision boundaries are

quadratic or linear.

Perceptron:

Linear classifier based on weighted sum and threshold.

Ignores probability distribution, only separates classes linearly.

Comparison:

Bayes is optimal under correct assumptions (Gaussian distributions).

Perceptron is simpler, needs no prior probability estimation, but only works

for linearly separable problems.

Bayes is statistical, perceptron is deterministic.

Q10) Explain feature detection in multilayer perceptrons with

examples.
Feature Detection in MLPs:

Hidden layers act as automatic feature extractors.

Each neuron detects specific patterns in inputs.

Lower layers learn simple features; higher layers combine them into complex
representations.

Example – Image Processing (digit recognition):

First hidden layer: Detects edges and lines.

Second hidden layer: Detects shapes (circles, corners).

Output layer: Recognizes digits 0–9.

Example – NLP (word embeddings):

Lower layers: Capture character patterns.

Middle layers: Learn word semantics.

ANN_Module2 12
 Output layer: Classifies sentences.

Thus, MLPs transform raw input into meaningful feature hierarchies.

— By K. Shashank

ANN_Module2 13