Course Material
Subject Name: Computer Vision
Faculty: Dr. P. Shunmuga Perumal (SP)
Deep networks:
●A deep learning model is a type of artificial neural network
(ANN) that is composed of multiple layers of processing
units (neurons) that learn to extract increasingly abstract
features from raw input data.
●It's called "deep" because it has many hidden layers,
unlike traditional machine learning models which are
typically shallow.
●Deep learning is a subset of machine learning that excels
at learning complex patterns from large datasets, especially
for image recognition, speech processing, natural
language understanding, and more.
Figure: Architecture of a typical deep-neural network
Components of a Deep Learning Model
A typical deep neural network consists of:
1. Input Layer
●Purpose: Receives the raw input data.
○e.g., pixel values of an image,
○words in text,
○sensor data, etc.
Example:
○For an image of size 28×28 (like in MNIST), the input
layer has 784 neurons (28×28 = 784).
○Each neuron corresponds to one feature (e.g., pixel
intensity).
2. Hidden Layers
●Purpose: Perform computations on input data using:
○weights,
○biases,
○and activation functions.
●Deep models can have many such layers (dozens to
hundreds).
Each hidden layer includes:
a. Neurons
●Processing units that apply a weighted sum and activation
function to their inputs.
b. Weights and Biases
●These parameters are what we update when we talk about
"training" a model.
●They are initialized to some random value or set to 0 and updated
as the training progresses.
●The bias is analogous to a weight independent of any input node.
●Basically, it makes the model more flexible, since you can "move" the
activation function around.
●Weights: Influence how important an input feature is.
●Biases: Shift the activation to better fit the data.
c. Activation Function
●The activation function can be defined as a mathematical
function that introduces non-linearity to the neural network.
●This enables the models to learn complex patterns and helps
them make accurate predictions.
●In simple words, think of it like a switch, that decides
whether a neuron should be activated or not.
○If the input is above a certain threshold or meets certain
criteria, the neuron will activate/fire, or else it remains
inactive.
●By infusing non-linear transformations in the network,
activation functions help in the creation of more advanced
and sophisticated decision boundaries.
●This leads to better accuracy and predictive capabilities.
●Classical example: XOR problem
Figure: Decision Boundary created by Network with and
without Activation Function
Common activation functions:
Example:
● During forward propagation, the weighted sum will be calculated as:
● As this is a binary classification task, we will use the Sigmoid activation function.
● This is known as the activation value(neuron’s level of activation) for the
given inputs and weights.
More Information about Activation function:
●The human brain has approximately 100 billion neurons,
each connected to thousands of other neurons through
trillions of synapses.
●This complex network forms the basis for:
○cognitive abilities,
○sensory perception,
○and motor functions.
●At the core of neuron firing is the action potential. This is an
electrochemical signal that travels along the length of a
neuron’s axon.
Neural Networks Replicate Biological Activation
●Activation functions play a pivotal role in enabling Artificial
Neural Networks (ANNs) to learn from data and adapt to
new patterns.
●By adjusting the weights of connections between neurons,
ANNs can refine their responses to inputs.
●This gradually improves their ability to perform tasks such
as:
○image recognition,
○natural language processing (NLP),
○speech recognition, etc.
●Inspired by the functioning of the human brain, ANNs
leverage neuron activation to:
○process information,
○make decisions,
○and learn from data.
●Activation functions, mathematical operations within
neurons, introduce non-linearities to the network, enabling it
to capture intricate patterns and relationships in complex
datasets.
●This non-linearity is crucial for the network’s ability to learn
and adapt.
3. Output Layer
●Purpose: Produces the final prediction based on the
computations in the hidden layers.
●The number of output neurons = number of target classes
(for classification).
●Uses an appropriate activation function:
○Softmax for multi-class classification
○Sigmoid for binary classification
○Linear for regression tasks
Forward Propagation
●The input data moves from input → hidden layers → output
layer.
●Each layer transforms the data using matrix operations and
activations.
●The result is the model’s prediction.
Backpropagation and Training
●The model compares its prediction with the true output
using a loss function (e.g., cross-entropy, MSE).
●Backpropagation computes the gradient of the loss w.r.t.
each weight.
●Optimizer (e.g., SGD, Adam) updates the weights to
minimize the loss.
Problem with linear functions:
●Single-layer perceptrons cannot solve non-linearity based
problems.
●Because when the data is not linearly separable we cannot
draw a straight line to separate the output classes.
Architecture of a single layer perceptron:
●A linear model cannot capture the "exclusive or" pattern.
●Linear functions (without activation) are just weighted sums
– they can’t form curved decision boundaries.
XOR- Problem:
●The XOR operation is a binary operation that takes two
binary inputs and produces a binary output.
●The output of the operation is 1 only when the inputs are
different.
●XOR problem cannot be solved by linear models.
Truth Table for XOR
Solution: