UNIT I: INTRODUCTION TO ARTIFICIAL NEURAL NETWORKS

Course Outcome: CO1 | Duration: 07 Hours

1. Introduction to Artificial Neural Networks (ANN)

An Artificial Neural Network (ANN) is a computational model inspired by the structure and functioning of
the human brain. It is a system of interconnected processing elements (neurons/nodes) that work
together to process information by responding to external inputs and relaying information between each
unit.

Definition
An ANN is a massively parallel distributed processor made up of simple processing units (artificial
neurons) that has a natural propensity for storing experimental knowledge and making it available for
use.

Why ANN?
• Traditional computers follow explicit programmed rules and struggle with noisy or incomplete
data.
• ANNs can learn from examples without being explicitly programmed.
• ANNs can generalize - apply learned patterns to new unseen data.
• ANNs are tolerant to noise and fault in the input data.
• ANNs can solve problems that are too complex for conventional techniques.

Key Insight
ANNs acquire knowledge through learning (training on data) and store this knowledge within
inter-neuron connection strengths called synaptic weights. This is analogous to how the
human brain learns from experience.

2. History of Neural Networks

The development of neural networks spans several decades, marked by periods of enthusiasm and
setbacks:

Year / Era Milestone Contributors

1943 First mathematical model of a neuron - introduced McCulloch & Pitts
the concept of threshold logic
 1949 Hebbian Learning Rule - 'Neurons that fire together, Donald Hebb
wire together'

1957 Perceptron - first trainable single-layer neural Frank Rosenblatt

network

1960 Adaline (Adaptive Linear Neuron) - used delta Widrow & Hoff
learning rule

1969 Limitations of Perceptron shown (XOR problem) - Minsky & Papert

caused first AI winter

1974-1986 Backpropagation algorithm developed - revived Werbos, Rumelhart et al.

interest in ANNs

1980s Hopfield Networks, Boltzmann Machines, CNN Hopfield, Hinton

concepts emerged

1990s Support Vector Machines challenge ANNs; second Vapnik et al.

AI winter for deep nets

2006 Deep Learning revival - efficient training of deep Hinton, Bengio, LeCun
networks via pre-training

2012+ Deep Learning dominates - AlexNet, GPT, ResNet, Multiple researchers

Transformers

3. Structure and Working of Biological Neural Network

(BNN)

3.1 Structure of a Biological Neuron

The human brain contains approximately 100 billion neurons, each connected to thousands of others. A
neuron consists of:

Component Description & Function

Cell Body (Soma) Contains the nucleus. Processes incoming signals and generates
output. Acts as the main processing unit.
Dendrites Tree-like branches extending from the soma. Receive signals
(inputs) from other neurons via synaptic connections.
Axon A long fiber extending from the soma. Transmits the output signal
(action potential) to other neurons.
Axon Terminals (Synaptic Tips of the axon that form synaptic connections with dendrites of
Knobs) other neurons.
Synapse The junction between neurons. Chemical neurotransmitters pass the
signal from one neuron to the next.
Myelin Sheath Insulates the axon and speeds up signal transmission.
3.2 Working Mechanism
A neuron works through the following steps:
• Step 1 - Reception: Dendrites receive electrical/chemical signals from multiple other neurons.
• Step 2 - Integration: The soma integrates (sums up) all incoming signals. Excitatory signals
increase firing probability; inhibitory signals decrease it.
• Step 3 - Threshold: If the integrated signal exceeds a threshold potential (~-55mV), the neuron
fires (generates an action potential).
• Step 4 - Transmission: The action potential travels down the axon to its terminals.
• Step 5 - Output: Neurotransmitters are released at synapses and received by the next neuron's
dendrites.

Biological Learning
Learning in the brain occurs by changing the strength of synaptic connections (synaptic
plasticity). Frequently used connections grow stronger. This is the biological basis for
Hebbian Learning in ANNs.

4. Neural Net Architecture

4.1 Artificial Neuron (Node)

An artificial neuron mimics the biological neuron. It performs these operations:
• Receives multiple input signals: x1, x2, ..., xn
• Each input is multiplied by a corresponding weight: w1, w2, ..., wn (synaptic weights)
• A bias term (b or w0) is added to shift the activation
• Computes the net input (weighted sum): net = sum(wi * xi) + b
• Applies an activation function f(net) to produce the output y

Mathematical Model of a Neuron

net = w1*x1 + w2*x2 + ... + wn*xn + b Output y = f(net) Where f() is the activation function,
wi are weights, xi are inputs, and b is bias.

4.2 Components of ANN Architecture

Component Role
Input Layer Receives raw input data. No computation; just passes data forward.
Number of neurons = number of features in input.
 Hidden Layer(s) Intermediate layers that extract features and learn complex patterns.
Number and size are hyperparameters chosen by the designer.
Output Layer Produces the final result (class probabilities, values, etc.). Number of
neurons = number of output classes/values.
Weights (W) Numerical values on connections that determine the strength of
influence between neurons. These are learned during training.
Bias (b) Extra parameter for each neuron that allows shifting the activation
function, providing flexibility in learning.
Activation Function Introduces non-linearity allowing the network to learn complex
patterns. Applied at each neuron.

5. Topology of Neural Network Architecture

5.1 Based on Connection Patterns

Feedforward Networks: Information flows in one direction only - from input to output. No cycles or
loops. Examples: Single-layer Perceptron, Multilayer Perceptron (MLP), CNN.
Feedback (Recurrent) Networks: Outputs can feed back as inputs. Contains cycles/loops. Allows
memory and temporal processing. Examples: Hopfield Network, RNN, LSTM.

5.2 Based on Number of Layers

Single Layer Network: Has only one computational layer (output layer). The input layer is not counted.
Can solve only linearly separable problems. Example: Simple Perceptron.
Multilayer Network: Has one or more hidden layers between input and output. Can solve non-linearly
separable problems. The Universal Approximation Theorem states that an MLP with one hidden layer
can approximate any continuous function.

5.3 Based on Connectivity

• Fully Connected (Dense): Every neuron in a layer connects to every neuron in the next layer.
• Partially Connected: Only some connections exist between layers.
• Locally Connected: Neurons connect only to a subset of neurons in adjacent layers (e.g., CNN
filters).

6. Features and Characteristics of ANN

6.1 Key Features

 Feature Description
Massive Parallelism Many neurons perform computations simultaneously, making ANNs
fast and efficient.
Distributed Representation Knowledge is stored across many weights, not in a single location.
This makes ANNs fault-tolerant.
Learning Ability ANNs learn from training data by adjusting weights. They can learn
complex nonlinear relationships.
Adaptability Weights can be retrained when the environment changes without
redesigning the system.
Generalization After training, ANNs can correctly process new unseen inputs - not
just memorized patterns.
Fault Tolerance Even if some neurons fail, the network can still produce reasonable
outputs due to distributed storage.
Non-linearity Activation functions introduce non-linearity allowing modeling of
complex real-world problems.

6.2 Characteristics
• Inspired by biological nervous systems
• Composed of many simple computational units operating in parallel
• Knowledge is encoded in connection weights (not in explicit rules)
• Can handle noisy, incomplete, or fuzzy input data
• Performance improves with more data and training
• Black-box nature: difficult to interpret internal representations

7. Types of Neural Networks

7.1 By Feedback Mechanism

Feedforward Neural Network (FNN): No cycles. Data flows input to output. Used for classification and
regression (MLP, CNN).
Recurrent Neural Network (RNN): Has directed cycles. Maintains internal state (memory). Used for
sequential data: time series, NLP.

7.2 By Learning Method

Supervised Learning Networks: Trained with labeled data. Adjusts weights to minimize error between
predicted and actual output. Examples: MLP with backpropagation, Perceptron.
Unsupervised Learning Networks: No labels. Network finds patterns and clusters on its own.
Examples: Kohonen Self-Organizing Maps (SOM), Autoencoders.
Reinforcement Learning Networks: Learn by interacting with environment. Receive reward/penalty
for actions. Examples: Deep Q-Networks (DQN).

7.3 Notable Types

Network Type Key Characteristics & Applications
Perceptron Simplest ANN. Single layer. Binary classification of linearly separable
data.
Multilayer Perceptron (MLP) Multiple hidden layers. Universal approximator. Classification,
regression.
Convolutional Neural Network Uses convolutional layers for spatial feature extraction. Image
(CNN) recognition, computer vision.
Recurrent Neural Network Sequential data processing with memory. NLP, time series
(RNN) prediction.
Long Short-Term Memory Special RNN that solves vanishing gradient problem. Long-term
(LSTM) dependencies.
Autoencoder Encoder-decoder architecture. Learns compressed representation.
Dimensionality reduction, anomaly detection.
Generative Adversarial Generator vs Discriminator network. Generates realistic synthetic
Network (GAN) data, images.
Radial Basis Function Network Uses radial basis activation functions. Function approximation,
(RBFN) pattern recognition.

8. Activation Functions
An activation function determines the output of a neuron given its net input. It introduces non-linearity,
which is essential for learning complex patterns.

8.1 Why Activation Functions?

• Without activation functions, the output would be a linear combination of inputs regardless of
depth.
• Non-linear activation allows stacking layers to approximate complex functions.
• Controls neuron firing: determines if/how much a neuron activates.

8.2 Common Activation Functions

Function Formula Range Properties & Use Cases
Step (Threshold) f(x) = 1 if x >= 0, else 0 {0, 1} Binary output. Used in McCulloch-Pitts
neuron and Perceptron. Cannot handle
multi-class.

Sigmoid (Logistic) f(x) = 1 / (1 + e^(-x)) (0, 1) Smooth S-curve. Good for binary
classification output. Suffers from
vanishing gradients.

Hyperbolic Tangent f(x) = (e^x - e^(-x)) / (e^x (-1, 1) Zero-centered. Stronger gradients than
(tanh) + e^(-x)) sigmoid. Still suffers vanishing
gradients.

ReLU (Rectified f(x) = max(0, x) [0, infinity) Most popular in deep nets. Solves
Linear Unit) vanishing gradient. Sparse activation.
Dying ReLU problem.

Leaky ReLU f(x) = x if x>0, else 0.01x (-inf, inf) Fixes dying ReLU. Small negative slope
for x<0.

Softmax f(xi) = e^xi / sum(e^xj) (0, 1), Used in output layer for multi-class
sums=1 classification. Outputs probability
distribution.

Linear (Identity) f(x) = x (-inf, inf) Used in regression output layer. No

non-linearity.

Vanishing Gradient Problem

Sigmoid and tanh functions saturate at extremes where the gradient approaches zero.
During backpropagation, these near-zero gradients are multiplied through layers, causing
earlier layers to learn very slowly or not at all. ReLU largely solves this by having a constant
gradient of 1 for positive inputs.

9. Models of Neuron

9.1 McCulloch & Pitts (M-P) Model (1943)

The first mathematical model of a neuron, proposed by Warren McCulloch (neurophysiologist) and
Walter Pitts (logician). It was a simplified binary threshold model.

Structure
• Receives multiple binary inputs: x1, x2, ..., xn (each 0 or 1)
• Each input has an excitatory (+1) or inhibitory (-1) connection
• Computes a weighted sum of inputs
• Fires (output=1) if the sum meets or exceeds a threshold theta; else output=0

Mathematical Formulation
 M-P Neuron Formula
net = sum(xi * wi), for i = 1 to n Output y = 1 if net >= theta Output y = 0 if net < theta
Where: xi = binary inputs (0 or 1) wi = weights (+1 excitatory, -1 inhibitory) theta =
threshold value

Properties of M-P Model

• Time is discrete - neurons process in synchronized time steps
• Inputs and outputs are binary (0 or 1)
• Weights are fixed - no learning; threshold is set manually
• Can implement basic logical functions (AND, OR, NOT)

Implementing Logic Gates with M-P Neurons

Gate Inputs Weights Threshold Operation

AND x1, x2 w1=1, w2=1 theta=2 Fires only if both inputs

are 1 (net=2 >= 2)
OR x1, x2 w1=1, w2=1 theta=1 Fires if at least one
input is 1 (net>=1)

NOT x1 w1=-1 theta=0 Fires if input is 0;

inhibited if input is 1
NAND x1, x2 w1=-1, w2=-1 theta=-1 Fires unless both
inputs are 1

Limitations of M-P Model

• Weights are fixed (not learnable) - cannot adapt from data
• Only binary inputs/outputs - cannot handle continuous values
• Cannot implement XOR gate (non-linearly separable)
• Threshold must be set manually by the user

9.2 Perceptron Model (Rosenblatt, 1957)

Frank Rosenblatt introduced the Perceptron as an extension of the M-P model, adding the crucial ability
to learn weights from training data automatically.

Structure
• Input layer: n input nodes (x1, x2, ..., xn)
• Weights: w1, w2, ..., wn associated with each connection
• Bias: w0 (or b) - an extra input with fixed value +1
• Net input computation: net = sum(wi * xi) + b
 • Activation: Step function (output 1 or 0 / +1 or -1)

Perceptron Learning Algorithm

Learning Steps
1. Initialize weights randomly (small values near 0) 2. For each training sample (x, d) -
x=input, d=desired output: a. Compute actual output: y = f(W*X + b) b. Compute error: e
= d - y c. Update each weight: wi(new) = wi(old) + learning_rate * e * xi d. Update bias:
b(new) = b(old) + learning_rate * e 3. Repeat until all samples classified correctly or max
iterations reached

Perceptron Convergence Theorem

If the training data is linearly separable, the Perceptron Learning Algorithm is guaranteed to converge
to a solution (find correct weights) in a finite number of steps. This was the first formal learning theorem
for neural networks.

Limitations
• Can only classify linearly separable problems
• Cannot solve XOR - famously shown by Minsky & Papert (1969)
• Only a single layer - no hidden layers
• Does not guarantee convergence for non-linearly separable data

9.3 ADALINE Model (Widrow & Hoff, 1960)

ADALINE stands for Adaptive Linear Neuron (or ADAptive LInear NEuron). It was proposed by Bernard
Widrow and Ted Hoff at Stanford as an improvement over the Perceptron.

Key Differences from Perceptron

Aspect Perceptron ADALINE

Training Signal Uses thresholded (binary) output for Uses continuous (pre-threshold) net
weight update input for weight update

Learning Rule Perceptron learning rule Widrow-Hoff Delta Rule (Least Mean
Squares - LMS)

Error Signal d - y (where y is 0 or 1) d - net (where net is the continuous

weighted sum)

Convergence Guaranteed only for linearly separable Minimizes mean squared error, more
stable convergence

Output Binary after step function Continuous (before threshold) or

binary (after threshold)
Delta Learning Rule (Widrow-Hoff / LMS Rule)

Delta Rule Formula

Weight Update: delta_wi = learning_rate * (d - net) * xi New weight: wi(new) = wi(old) +
delta_wi Where: d = desired (target) output net = actual net input (continuous, before
activation) xi = input value learning_rate (eta) = step size (0 < eta < 1) Objective:
Minimize E = (1/2) * (d - net)^2

MADALINE
MADALINE (Multiple ADALINE) extended the concept to multiple adaptive elements arranged in a
layer, representing one of the first multi-layer networks. It used a more complex learning rule (MRI -
MADALINE Rule I).

10. Basic Learning Laws

10.1 Hebbian Learning Rule

Proposed by Donald Hebb (1949) in 'The Organization of Behavior'. The core principle: 'Cells that fire
together, wire together.'

Hebbian Rule
If two interconnected neurons are both active simultaneously, strengthen the connection
between them. Delta_wij = learning_rate * yi * xj Where yi is the output of neuron i, xj is the
input from neuron j Result: Correlation-based learning - captures statistical regularities.

10.2 Perceptron Learning Rule

Update weights only when the network makes an error. The error signal drives weight adjustment:

Perceptron Rule
If output is correct: no weight change If output should be 1 but got 0: add input to weights
(delta_w = +learning_rate * x) If output should be 0 but got 1: subtract input from weights
(delta_w = -learning_rate * x)

10.3 Delta Rule (Widrow-Hoff / LMS)

Minimizes the sum of squared errors using gradient descent. Used in ADALINE. Adjusts weights
proportional to the error in the linear output (before activation):

Delta Rule
 delta_wi = eta * (d - net) * xi This is a form of gradient descent on the error surface E =
0.5*(d-net)^2

10.4 Competitive Learning Rule

Neurons compete to be activated. Only the winner (most active neuron) updates its weights. Used in
Self-Organizing Maps and clustering:

Competitive Learning
Winner neuron w*: the one closest to input x Update: delta_w* = learning_rate * (x - w*)
Loser neurons: no update

10.5 Widrow-Hoff Learning (Extended)

A generalization of the delta rule. The goal is to find weights that minimize the mean squared error
between desired and actual network output across all training samples. This forms the basis for
gradient descent optimization.

10.6 Boltzmann Learning

Used in stochastic networks. Adjusts weights based on correlations between neuron activations when
the network runs freely versus when it is clamped to training data. Used in Boltzmann Machines and
Restricted Boltzmann Machines (RBMs).

11. Applications of Neural Networks

Domain Application ANN Type Used

Image Recognition Face detection, object classification, medical CNN

imaging (tumor detection)
Natural Language Machine translation, sentiment analysis, RNN, LSTM, Transformer
Processing chatbots, text summarization
Speech Recognition Voice assistants (Siri, Alexa), speech-to-text, RNN, CNN
speaker recognition

Medical Diagnosis Predicting cancer from images, ECG MLP, CNN

classification, drug discovery
Financial Systems Stock price prediction, credit scoring, fraud MLP, RNN, LSTM
detection, risk assessment

Robotics & Control Autonomous vehicles, industrial robot Reinforcement Learning

control, path planning
Recommendation Netflix, Amazon product recommendations, Collaborative filtering ANN
Systems personalized ads

Game Playing AlphaGo, Chess engines, video game AI Deep Reinforcement Learning
agents

Agriculture Crop disease detection, yield prediction, CNN, MLP

precision irrigation

Water Management Controlling water reservoirs (CO1 Case MLP, RNN

Study), flood prediction

12. Comparison of BNN and ANN

Aspect Biological Neural Network (BNN) Artificial Neural Network (ANN)

Basic Unit Biological neuron (cell body, Artificial neuron (node/perceptron)

dendrites, axon)

Processing Element ~100 billion neurons in human brain Typically thousands to millions of
nodes

Processing Speed ~100 Hz (slow individual neurons, GHz processors; fast sequential or
fast parallel) GPU parallel

Energy Consumption ~20 Watts (entire brain) Hundreds of watts to kilowatts for
large networks

Fault Tolerance Highly fault tolerant - works with Partially fault tolerant due to
dead neurons distributed storage

Learning Continuous lifelong learning from Batch training on fixed datasets; can
experience be retrained
Memory Memory and processing are Memory and processing are
integrated (associative) separate

Representation Electrochemical signals; complex Mathematical weights in matrix form

3D structure
Reliability Can function with neuron Robust but sensitive to weight
death/damage corruption

Adaptability Adapts in real-time to new stimuli Requires retraining; some online

learning possible

Complexity Incredibly complex; not fully Simplified model; mathematically

understood tractable

Speed of Learning Years of experience needed for Can learn in hours/days on

complex skills appropriate data

Size Compact biological structure Can be distributed across many

machines
Connectivity Each neuron ~1000-10000 Typically fully connected layers;
connections varies by architecture
Exemplar / Case Study: Controlling Water Reservoirs
Objective
Use an ANN to optimize the operation of water reservoirs by predicting optimal water
release schedules based on rainfall, water level, demand, and seasonal patterns.

Problem Statement
Water reservoir management involves complex decisions about when and how much water to release
to satisfy downstream demand while preventing floods and maintaining minimum storage. Traditional
rule-based systems struggle with the complexity and variability of inputs.

ANN Solution Design

• Network Type: Multilayer Perceptron (MLP) or RNN (for temporal patterns)
• Inputs: Current reservoir level, inflow rate, rainfall forecast, downstream demand,
season/month, temperature
• Outputs: Recommended water release rate (cubic meters/second)
• Training Data: Historical data of reservoir levels, releases, and outcomes over many years
• Learning: Supervised learning using backpropagation minimizing prediction error

Benefits
• Handles non-linear relationships between rainfall, inflow, and demand
• Learns seasonal patterns automatically from historical data
• Provides real-time recommendations as new sensor data arrives
• Can generalize to unusual weather patterns not seen in training data

Rule Extraction from ANN

After training, rules can be extracted from the learned weights to make the system interpretable:
• If rainfall forecast > X mm AND reservoir level > Y% AND demand < Z, then release W m3/s
• Methods: CART (Classification and Regression Trees) trained on ANN outputs, sensitivity
analysis of weights, DeepLIFT
• Benefit: Engineers can validate and understand the ANN's decision logic

CO1: Understand fundamentals of ANN including its structure, history, neuron models, learning
laws, and applications.
 UNIT II: LEARNING ALGORITHMS
Course Outcome: CO2 | Duration: 07 Hours

1. Learning and Memory in ANN

1.1 What is Learning in ANN?

Learning in an ANN is the process by which the network modifies its synaptic weights (and biases) in
response to external stimuli (training data) so that it can perform a desired task. After learning, the
network's behavior is determined by the adjusted weights.

Formal Definition
Learning = a process by which the free parameters (weights and biases) of a neural network
are adapted through a continuous process of stimulation by the environment.

1.2 Types of Learning

Supervised Learning: Network learns from labeled training data {(x1,d1), (x2,d2), ..., (xN,dN)}. An
external teacher provides desired outputs. Network minimizes the difference between actual and
desired output. Example: MLP trained with backpropagation.
Unsupervised Learning (Self-Organized): No labels provided. Network discovers structure, patterns,
and clusters in data on its own. Uses statistical properties of input. Examples: SOM, K-means
clustering ANN, autoencoders.
Reinforcement Learning: Network learns through interaction with an environment. Receives a reward
signal (not exact desired output). Maximizes cumulative reward over time. Example: DQN playing Atari
games.

1.3 Memory in ANN

Memory Type Description Example

Long-Term Memory Stored in synaptic weights. Represents learned Trained weights of a deep
knowledge that persists indefinitely. CNN model

Short-Term Memory Temporary activation states in recurrent Hidden state in LSTM during
networks. Represents recent context. a sentence

Associative Memory Can recall complete patterns from partial or Hopfield Network recalling a
noisy cues. Content-addressable. stored pattern
2. Learning Algorithms

2.1 Classification of Learning Algorithms

Algorithm Category Description & Examples
Error Correction Learning Adjust weights based on error between actual and desired output.
Backpropagation, Delta Rule, LMS.
Hebbian Learning Strengthen connections between co-active neurons. Auto-
associative memories.
Competitive Learning Winner-take-all dynamics. SOM, k-WTA networks, Vector
Quantization.
Boltzmann Learning Stochastic/probabilistic learning. Minimize energy function.
Boltzmann Machines, RBMs.
Gradient Descent Methods Minimize cost function by computing gradients. Batch GD, Stochastic
GD, Mini-batch GD, Adam.

3. Number of Hidden Nodes

3.1 Why It Matters

The number of hidden layers and the number of neurons in each hidden layer are critical
hyperparameters. Too few neurons: underfitting (cannot learn complex patterns). Too many neurons:
overfitting (memorizes training data, poor generalization).

3.2 Rules of Thumb

• Number of hidden neurons should be between the size of the input and output layers.
• Common heuristic: h = sqrt(n * m), where n = input nodes, m = output nodes.
• Another rule: h = (2/3) * n + m
• Cascade-Correlation method: Start with no hidden nodes and add them one at a time.
• Cross-validation: Try different architectures and choose the one with best validation
performance.

Universal Approximation Theorem

A feedforward network with a single hidden layer containing a finite number of neurons can
approximate any continuous function on compact subsets of R^n, given enough neurons.
This justifies the use of MLP for a wide range of problems.
3.3 Practical Guidelines
• Start with a simple network and increase complexity only if needed
• Use regularization techniques (dropout, L2) to reduce overfitting in large networks
• Use validation set performance to guide architecture selection
• Modern practice often uses deeper networks (more layers) with fewer neurons per layer

4. Error Correction and Gradient Descent Rules

4.1 Error Correction Learning

The fundamental principle: if the network output differs from the desired output, the weights are
adjusted to reduce the error. This is the basis for nearly all supervised learning in ANNs.

Error Correction Principle

Error signal: e(n) = d(n) - y(n) Weight update: delta_w(n) = eta * e(n) * x(n) Where: d(n) =
desired/target output y(n) = actual output of network x(n) = input signal eta = learning
rate Goal: Minimize J = E[e^2(n)] = Mean Squared Error

4.2 Gradient Descent Rule

Gradient descent is an optimization algorithm that minimizes the cost function (typically Mean Squared
Error) by iteratively moving in the direction of steepest descent (negative gradient):

Gradient Descent Update Rule

Cost Function: J(W) = (1/2N) * sum[(d_i - y_i)^2] (MSE over N samples) Gradient: dJ/dw =
partial derivative of J with respect to weight w Update: w_new = w_old - eta * (dJ/dw) The
negative gradient points in the direction of steepest decrease in J.

4.3 Variants of Gradient Descent

Variant Description Pros Cons

Batch GD Compute gradient using Stable convergence, Very slow for large datasets
entire dataset accurate gradient

Stochastic GD Update weights after each Fast, can escape Noisy updates, unstable
(SGD) sample local minima convergence

Mini-Batch GD Update after each mini-batch Balance of speed Need to tune batch size
(e.g., 32 samples) and stability

Momentum SGD Add fraction of previous Faster convergence, Extra hyperparameter

update less oscillation
 Adam Optimizer Adaptive learning rates per Efficient, works well Computationally heavier
parameter in practice

5. Perceptron Learning Algorithm (Detailed)

5.1 Algorithm Steps

Perceptron Learning Algorithm
Input: Training set {(x1,d1), (x2,d2), ..., (xN,dN)} Parameters: learning_rate (eta),
max_epochs Step 1: Initialize weights w = [0,...,0] or random small values Step 2: Set
epoch = 0 Step 3: Repeat until convergence or max_epochs: For each training sample (xi,
di): a. Compute net = sum(wj * xij) + b b. Apply activation: yi = f(net) [step function: 1 if
net>=0, else 0] c. Compute error: ei = di - yi d. Update weights: wj = wj + eta * ei * xij
e. Update bias: b = b + eta * ei epoch = epoch + 1 Step 4: Return final weights

5.2 Decision Boundary

The Perceptron finds a linear decision boundary (hyperplane) that separates two classes. In 2D input
space, this is a straight line. The equation of this boundary is: w1*x1 + w2*x2 + b = 0. Points above the
line get output 1; points below get output 0.

5.3 Convergence Theorem

The Perceptron Convergence Theorem guarantees: if the training data is linearly separable, the
algorithm will find a separating hyperplane in a finite number of updates. The number of updates is
bounded by (R/gamma)^2, where R is the maximum norm of training vectors and gamma is the margin
of separation.

6. Supervised Learning: Backpropagation

6.1 Overview
Backpropagation (Backprop) is the most widely used learning algorithm for multilayer feedforward
neural networks. It was popularized by Rumelhart, Hinton, and Williams in 1986. The key insight is the
chain rule of calculus used to efficiently compute gradients through multiple layers.

Core Idea
 Two-phase process: 1. Forward Pass: Input propagates through the network to produce an
output. Compute loss/error. 2. Backward Pass: Error propagates backwards from output to
input layer. Compute gradient of loss with respect to each weight using the chain rule.
Update each weight.

6.2 Mathematical Foundation: Chain Rule

For a network with layers L1, L2, ..., LM, the gradient of the error E with respect to weight wij in layer k
is computed using the chain rule:

Chain Rule Application

dE/dwij = dE/d(net_j) * d(net_j)/dwij For output layer neuron j: delta_j = dE/d(net_j) = (d_j -
y_j) * f'(net_j) [error signal * derivative of activation] For hidden layer neuron j: delta_j =
f'(net_j) * sum[delta_k * w_jk] [derivative of activation * weighted sum of deltas from next
layer] Weight update: w_ij(new) = w_ij(old) + eta * delta_j * x_i

7. Multilayer Network Architectures

7.1 Structure of MLP

• Input Layer: Passes raw features. No computation.
• Hidden Layers: One or more layers with non-linear activation (ReLU, sigmoid, tanh). Learn
feature representations.
• Output Layer: Produces prediction. Activation depends on task (sigmoid for binary, softmax for
multi-class, linear for regression).

7.2 Universal Approximation

An MLP with at least one hidden layer and a non-linear activation function can approximate any
continuous function to arbitrary accuracy given enough neurons. This makes MLP a general-purpose
function approximator.

7.3 Depth vs Width

Approach Description Trade-offs

Wider Network (more neurons Single hidden layer with many Easier to train; may need
per layer) neurons exponentially more neurons for
complex tasks
Deeper Network (more hidden Multiple hidden layers with fewer More efficient; hierarchical feature
layers) neurons each learning; harder to train (vanishing
gradients)
Deep Learning Best Practice Multiple layers with batch Best of both; modern standard for
normalization, skip connections complex tasks
(ResNet)

8. Backpropagation Learning Algorithm (Complete)

8.1 Complete Algorithm

Full Backpropagation Algorithm
Initialize: All weights w randomly (e.g., small Gaussian values N(0, 0.01)) For each epoch
(until convergence or max_epochs): Shuffle training data For each mini-batch B:
=== FORWARD PASS === For each layer l = 1 to L: net_l = W_l * a_{l-1} + b_l
a_l = f(net_l) [apply activation function] Output: a_L Compute Loss: J = (1/|B|) *
sum L(d_i, a_L_i) [e.g., MSE: J = (1/2)|d-a_L|^2] === BACKWARD PASS ===
Output layer: delta_L = (a_L - d) * f'(net_L) For each layer l = L-1 down to 1: delta_l
= (W_{l+1}^T * delta_{l+1}) * f'(net_l) === WEIGHT UPDATE === For each layer l:
W_l = W_l - (eta/|B|) * delta_l * a_{l-1}^T b_l = b_l - (eta/|B|) * sum(delta_l)

8.2 Key Hyperparameters

Hyperparameter Description & Typical Values
Learning Rate (eta) Controls step size in gradient descent. Too large: diverges. Too
small: slow convergence. Typical: 0.001 to 0.1.
Batch Size Number of samples per weight update. Powers of 2 common: 32, 64,
128, 256.
Number of Epochs Number of complete passes through training data. Monitor validation
loss to decide when to stop.
Momentum Fraction of previous weight change added to current. Speeds
convergence. Typical: 0.9.
Regularization (L2/Dropout) L2 adds penalty on weight magnitudes. Dropout randomly
deactivates neurons during training. Prevents overfitting.
Weight Initialization Xavier/Glorot initialization for sigmoid/tanh. He initialization for ReLU.
Prevents vanishing/exploding gradients.

8.3 Issues in Backpropagation Training

Problem Description Solution
Vanishing Gradient Gradients become extremely small in Use ReLU activation, Batch
early layers (sigmoid/tanh), preventing Normalization, skip connections
learning
 Exploding Gradient Gradients grow exponentially, causing Gradient clipping, weight initialization,
NaN weights Batch Normalization

Local Minima Gradient descent gets stuck in Momentum, Adam optimizer, random
suboptimal solutions restarts, stochastic GD

Overfitting Network memorizes training data, poor Dropout, L1/L2 regularization, early
generalization stopping, more data

Slow Convergence Training takes too long Adaptive learning rates (Adam,
RMSProp), learning rate schedules

Saddle Points Gradient is zero but not a minimum Stochastic noise in SGD helps escape;
second-order methods

9. Feedforward and Feedback (Recurrent) Neural Networks

9.1 Feedforward Neural Networks (FNN)

In FNNs, information flows strictly from input to output with no cycles. There is no memory of past
inputs.

Characteristics
• Acyclic directed graph of neurons
• Each layer only receives input from the previous layer
• No loops or cycles - no temporal dynamics
• Output depends only on current input
• Suitable for: image classification, tabular data regression, pattern recognition

Examples
• Single Layer Perceptron: Input -> Output layer
• Multilayer Perceptron: Input -> Hidden Layer(s) -> Output
• Convolutional Neural Network: Input -> Conv layers -> Fully Connected -> Output

9.2 Feedback (Recurrent) Neural Networks (RNN)

RNNs have directed cycles - outputs can feed back as inputs to the same or earlier layers. This creates
an internal state (memory) that allows processing of sequences.

Characteristics
• Contains feedback connections (directed cycles)
• Maintains hidden state h(t) that persists across time steps
 • Output at time t depends on current input AND previous hidden state
• Can model temporal dependencies in sequences

RNN Equations
Hidden state: h(t) = tanh(W_h * h(t-1) + W_x * x(t) + b_h) Output: y(t) = softmax(W_y * h(t) +
b_y) Where x(t) is input at time t, h(t-1) is previous hidden state Backpropagation Through
Time (BPTT) is used to train RNNs.

Problem: Vanishing Gradients in RNNs

During BPTT, gradients are multiplied through many time steps, causing them to vanish for long
sequences. This makes RNNs unable to learn long-term dependencies.

Solution: LSTM (Long Short-Term Memory)

LSTM was introduced by Hochreiter & Schmidhuber (1997). It uses a gating mechanism to control
information flow:
• Forget Gate: Decides what information to throw away from cell state
• Input Gate: Decides what new information to store in cell state
• Output Gate: Decides what to output based on cell state
• Cell State: Carries long-term information across time steps with minimal modification

Feature Simple RNN LSTM GRU (Simplified

LSTM)
Memory Short-term only Long and short-term Better than RNN

Gates None Forget, Input, Output (3 Reset, Update (2

gates) gates)

Vanishing Gradient Severe problem Largely solved Better than RNN

Complexity Simple Most complex Simpler than LSTM

Parameters Fewest Most Fewer than LSTM

Use Case Simple sequences Long sequences, NLP Faster training on

sequences

9.3 Comparison: Feedforward vs Feedback

Property Feedforward (FNN) Feedback/Recurrent (RNN)

Data Flow One direction: input to output Bidirectional with feedback loops

Memory No internal memory Has internal state (memory)

Input Type Fixed-size input vectors Sequential data of variable length

Temporal Modeling Cannot model time dependencies Naturally models sequences and time
Training Standard backpropagation Backpropagation Through Time
(BPTT)

Applications Image classification, tabular data NLP, speech, time series, video

Stability Always stable Can be unstable; gradient issues

Exemplar / Case Studies - Unit II

Case Study 1: Medical Diagnosis using ANN

Scenario
Develop an ANN system to diagnose heart disease based on patient clinical data. The
system should classify patients as having heart disease or not.

Problem Formulation
• Task: Binary Classification (heart disease: yes/no)
• Dataset: Cleveland Heart Disease dataset (303 samples, 13 features)
• Features: age, sex, chest pain type, resting blood pressure, cholesterol, fasting blood sugar,
ECG results, max heart rate, exercise-induced angina, ST depression, slope, vessels,
thalassemia
• Target: 1 = heart disease present, 0 = absent

ANN Architecture
• Input Layer: 13 neurons (one per feature)
• Hidden Layer 1: 16 neurons, ReLU activation
• Hidden Layer 2: 8 neurons, ReLU activation
• Output Layer: 1 neuron, Sigmoid activation (probability of disease)
• Loss Function: Binary Cross-Entropy
• Optimizer: Adam with learning rate 0.001

Training Process
• Data preprocessing: normalize features to [0,1], handle missing values
• Train/Validation/Test split: 70% / 15% / 15%
• Train using mini-batch backpropagation (batch size = 32)
• Early stopping: stop if validation loss doesn't improve for 10 epochs
• Typical results: 80-85% accuracy on test set
Significance
Medical diagnosis ANN systems can assist doctors by: providing a second opinion, processing multiple
features simultaneously, detecting subtle patterns in patient data, and being applicable in areas with
limited specialist access.

Case Study 2: Automated Trading Systems

Scenario
Build an LSTM-based trading system that predicts stock price movements and generates
buy/sell signals for automated trading.

Problem Formulation
• Task: Predict whether stock price will go up or down tomorrow (binary classification) or predict
exact price (regression)
• Input Features: Historical prices (OHLCV), technical indicators (RSI, MACD, Bollinger Bands),
volume, market sentiment
• Time Window: Use last 60 days of data to predict next day

ANN Architecture (LSTM-based)

• Input Shape: (60 time steps, N features)
• LSTM Layer 1: 128 units with return_sequences=True
• LSTM Layer 2: 64 units
• Dense Hidden Layer: 32 neurons, ReLU
• Dropout Layer: 0.2 rate (prevents overfitting)
• Output Layer: 1 neuron, Sigmoid for direction prediction

Training and Evaluation

• Training on historical data with walk-forward validation
• Loss: Binary cross-entropy; Optimizer: Adam
• Evaluation metrics: Accuracy, Sharpe Ratio, Maximum Drawdown, Profit Factor
• Risk Management: Position sizing based on prediction confidence

Key Considerations
• Non-stationarity of financial data requires re-training regularly
• Overfitting is a major risk - extensive regularization needed
• Transaction costs must be factored into profit calculations
 • Cannot predict black swan events (COVID crash, flash crashes)

CO2: Apply learning algorithms including backpropagation to train multilayer neural networks
for real-world applications.

Quick Reference: Key Formulas and Concepts

Important Equations Summary

Concept Formula / Key Point

Neuron Output y = f(sum(wi*xi) + b), where f is activation function

Sigmoid f(x) = 1/(1+e^(-x)), range (0,1)

ReLU f(x) = max(0,x), derivative = 1 if x>0, else 0

MSE Loss J = (1/N)*sum[(d_i - y_i)^2]

Delta Rule (Output) delta_j = (d_j - y_j) * f'(net_j)

Delta Rule (Hidden) delta_j = f'(net_j) * sum[delta_k * w_jk]

Weight Update w_ij(new) = w_ij(old) + eta * delta_j * x_i

Gradient Descent w = w - eta * dJ/dw

Perceptron Update w = w + eta * (d-y) * x

Hebbian Rule delta_w = eta * y * x

LSTM Cell State c(t) = f(t)*c(t-1) + i(t)*g(t)

Softmax f(xi) = e^xi / sum(e^xj)

- END OF STUDY NOTES -

Artificial Neural Networks | Unit I (CO1) & Unit II (CO2)