LECTURE NOTES

Programme Name : [Link]

Course Name: DEEP LEARNING

Course code : 23CSM362T

Year / Branch: III / CSE(AI)

Regulation: R23

Prepared By: M E Palanivel

 SYLLABUS
UNIT-I: LINEAR ALGEBRA, PROBABILITY AND NUMERICAL COMPUTATION (9)
Linear Algebra: Scalars, Vectors, Matrices and Tensors, Matrix operations, types of matrices, Norms, Eigen
decomposition, Singular Value Decomposition, Principal Components Analysis.
Probability and Information Theory: Random Variables, Probability Distributions, Marginal Probability,
Conditional Probability, Expectation, Variance and Covariance, Bayes’ Rule, Information Theory.
Numerical Computation: Overflow and Underflow, Gradient-Based Optimization, Constrained
Optimization, Linear Least Squares.

UNIT-II: MACHINE LEARNING AND DEEP FEED FORWARD NETWORKS (9)

Machine Learning: Basics and Under fitting, Hyper parameters and Validation Sets, Estimators, Bias and
Variance, Maximum Likelihood, Bayesian Statistics, Supervised and Unsupervised Learning, Stochastic
Gradient Descent, Challenges Motivating Deep Learning.
Deep Feed forward Networks: Learning XOR, Gradient-Based Learning, Hidden Units,
Architecture Design, Back-Propagation and other Differentiation Algorithms.

UNIT-III: REGULARIZATION AND OPTIMIZATION OF DL MODELS (9)

Regularization for Deep Learning: Parameter Norm Penalties, Norm Penalties as Constrained
Optimization, Regularization and Under-Constrained Problems, Dataset Augmentation, Noise
Robustness, Semi-Supervised Learning, Multi-Task Learning, Early Stopping, Parameter Tying
and Parameter Sharing, Sparse Representations, Bagging and Other Ensemble Methods, Dropout,
Adversarial Training, Tangent Distance, Tangent Prop and Manifold Tangent Classifier.

Optimization for Training Deep Models: Pure Optimization, Challenges in Neural Network

Optimization, Basic Algorithms, Parameter Initialization Strategies, Algorithms with Adaptive

Learning Rates, Approximate Second-Order Methods, Optimization Strategies and Meta-

Algorithms.
UNIT-IV: CONVOLUTIONAL NETWORKS (9)

Convolutional Networks: The Convolution Operation, Pooling, Convolution, Basic

Convolution Functions, Structured Outputs, Data Types, Efficient Convolution
Algorithms, Random or Unsupervised Features, Basis for Convolutional Networks.

UNIT-V: SEQUENCE MODELING (9)

Sequence Modeling: Recurrent and Recursive Nets: Unfolding Computational Graphs,

Recurrent Neural Networks, Bidirectional RNNs, Encoder-Decoder Sequence-to-

Sequence Architectures, Deep Recurrent Networks, Recursive Neural Networks, Echo
State Networks, LSTM, Gated RNNs, Optimization for Long-Term Dependencies, Auto
encoders, Deep Generative Models.
TEXT BOOKS
1. Ian Goodfellow, YoshuaBengio, Aaron Courville, “Deep Learning”, MIT Press,2016.
2. Josh Patterson and Adam Gibson, “Deep learning: A practitioner's approach”, O'Reilly
Media, First Edition,2017.
REFERENCE BOOKS:
1. Nikhil Buduma, Fundamentals of Deep Learning, Designing next-generation machine
intelligence algorithms, O’Reilly, Shroff Publishers, 2019.
2. Douwe Osinga, Deep learning Cook Book, Practical recipes to get started Quickly,
O’Reilly, Shroff Publishers, 2019.
ONLINE LEARNING RESOURCES
1. www. [Link]/blog

2. [Link]

3. [Link]

4. [Link]
 COURSE OBJECTIVES
 The main objective of the course is to develop a system to solve the real world
problems using Deep learning models.

 To Demonstrate the major technology trends driving Deep Learning

 To Build, train, and apply fully connected deep neural networks

 To build efficient neural networks models by Reducing Generalization error using

Optimization techniques.

 To Demonstrate the Convolutional Neural Network model

 To Demonstrate the sequence modeling and Networks of Deep Learning

LINEAR ALGEBRA
INTRODUCTION
 Linear algebra is a branch of mathematics that is widely used throughout
science and engineering.

 A good understanding of linear algebra is essential for understanding

and working with many machine learning algorithms, especially deep
learning algorithms.
SCALARS, VECTORS, MATRICES AND TENSORS

 Scalars: A scalar is just a single number, in contrast to most of the

other objects studied in linear algebra, which are usually arrays of
multiple numbers.

 Example, we might say “Let s ∈ R be the slope of the line,” while

defining a real-valued scalar, or “Let n ∈ N be the number of
units,” while defining a natural number scalar.
 VECTORS
 Vectors: A vector is an array of numbers. The numbers are arranged in order. We can
identify each individual number by its index in that ordering.

 Typically we give vectors lower case names written in bold typeface, such as x.

 The elements of the vector are identified by writing its name in italic typeface, with a
subscript.

 The first element of x is x1 , the second element is x2 and so on. We also need to say
what kind of numbers are stored in the vector.

 If each element is in R, and the vector has n elements, then the vector lies in the set
formed by taking the Cartesian product of R n times, denoted as Rn. When we need to
explicitly identify the elements of a vector, we write them as a column enclosed in square
brackets:
• Sometimes we need to index a set of elements of a vector. In this case, we define a
set containing the indices and write the set as a subscript.
• For example, to access x1 , x3 and x6, we define the set S = {1, 3, 6} and write xS.
• We use the − sign to index the complement of a set.
• For example x−1 is the vector containing all elements of x except for x1 , and x−S is the
vector containing all of the elements of x except for x1, x3 and x6 .
 MATRICES
 A matrix is a 2-D array of numbers, so each element is identified by

two indices instead of just one.

o We usually give matrices upper-case variable names with bold typeface, such as
A.

o If a real-valued matrix A has a height of m and a width of n, then we say that A ∈

Rm×n.

o We usually identify the elements of a matrix using its name in italic but not bold
font, and the indices are listed with separating commas.

o For example, A1,1 is the upper left entry of A and A m,n is the bottom right entry.
TENSORS

 Tensors: In some cases we will need an array with more than two axes.

 In the general case, an array of numbers arranged on a regular grid with a

variable number of axes is known as a tensor.

 We denote a tensor named “A” with this typeface: A.

 We identify the element of A at coordinates (i, j, k ) by writing A i,j,k.

LINEAR DEPENDENCE AND SPAN
 In order for A−1 to exist, Eq. Ax=b must have exactly one solution for every value of b.

 However, it is also possible for the system of equations to have no solutions or infinitely
many solutions for some values of b.

 If both x and y are solutions then z = αx + (1 − α)y is also a solution for any real α.

 Ax = ∑ xiA:,i.

 In general, this kind of operation is called a linear combination.

 Formally, a linear combination of some set of vectors {v (1) , . . . , v (n)} is given by

multiplying each vector v (i) by a corresponding scalar coefficient and adding the results:

∑ civ (i)

 The span of a set of vectors is the set of all points obtainable by linear combination of the
original vectors.
 CONT.

 Determining whether Ax = b has a solution thus amounts to testing whether b is in the

span of the columns of A. This particular span is known as the column space or the range
of A.
 In order for the system Ax = b to have a solution for all values of b ∈ Rm, we therefore
require that the column space of A be all of Rm .
 If any point in Rm is excluded from the column space, that point is a potential value of b
that has no solution.
 The requirement that the column space of A be all of Rm implies immediately that A must
have at least m columns, i.e., n ≥ m. Otherwise, the dimensionality of the column space
would be less than m.

 NORMS
 In machine learning, we usually measure the size of vectors using a function called a norm.
Formally, the L p norm is given by
||x||p = [ ∑ |xi|p ] (1/ p) for p ∈ R, p ≥ 1.
 Norms, including the L p norm, are functions mapping vectors to non-negative values.
 A norm is any function f that satisfies the following properties:
 f (x) = 0 ⇒ x = 0
 f (x + y) ≤ f(x) + f(y) (the triangle inequality)
 ∀α ∈ R, f(αx) = |α|f(x)
 The L2 norm, with p = 2, is known as the Euclidean norm.
 It is simply the Euclidean distance from the origin to the point identified by x. The L 2 norm is
used so frequently in machine learning that it is often denoted simply as ||x||, with the
subscript 2 omitted.
 CONT.
 It is also common to measure the size of a vector using the squared L 2 norm, which can be calculated
simply as xTx.

 The L1 norm may be simplified to ||x||1 = ∑|xi|

 One other norm that commonly arises in machine learning is the L∞ norm, also known as the max
norm.

 This norm simplifies to the absolute value of the element with the largest magnitude in the vector,
||x||∞ = max |xi|.

 In the context of deep learning, the most common way to do this is with the otherwise obscure
Frobenius norm ||A||F = √ ∑ Ai,j2, which is analogous to the L 2 norm of a vector.

 The dot product of two vectors can be rewritten in terms of norms.

x Ty = ||x||2||y||2 cos θ , where θ is the angle between x and y.

 SPECIAL KINDS OF MATRICES AND VECTORS

 A symmetric matrix is any matrix that is equal to its own transpose:

 A = AT

 A unit vector is a vector with unit norm: ||x||2 = 1.

 A vector x and a vector y are orthogonal to each other if xTy = 0

 If the vectors are not only orthogonal but also have unit norm, we call them orthonormal.

 An orthogonal matrix is a square matrix whose rows are mutually orthonormal and whose
columns are mutually orthonormal:

 ATA = AAT = I

 This implies that A−1 = AT

 EIGEN DECOMPOSITION
 One of the most widely used kinds of matrix decomposition is called eigen

decomposition, in which we decompose a matrix into a set of eigenvectors and

eigenvalues.

 An eigenvector of a square matrix A is a non-zero vector v such that multiplication by

A alters only the scale of v: Av = λv.

 The scalar λ is known as the eigenvalue corresponding to this eigenvector. (One can

also find a left eigenvector such that vTA = λvT , but we are usually concerned with

right eigenvectors).
EXAMPLE : EIGEN DECOMPOSITION OF A 2×2 MATRIX

 Problem
Find the eigenvalues and eigenvectors of
4 1
𝐴=
2 3
Step 1: Characteristic Equation
4−𝜆 1
= 4−𝜆 3−𝜆 −2
2 3−𝜆
= 𝜆2 − 7𝜆 + 10 = 0

Step 2: Eigenvalues
𝜆−5 𝜆−2 =0
𝜆1 = 5, 𝜆2 = 2
Step 3: Eigenvectors
For 𝜆 = 5
−1 1
𝐴 − 5𝐼 𝑥 = 0 ⇒
2 −2
1
𝑥=
1
 For 𝜆 = 2
2 1
𝐴 − 2𝐼 𝑥 = 0 ⇒
2 1
1
𝑥=
−2
 Final Eigen Decomposition

𝐴 = 𝑄Λ𝑄−1
1 1 5 0
𝑄= ,Λ =
1 −2 0 2
Apply Formula
1 −2 −1
𝑄 −1 =
−3 −1 1
 CONT.
 If v is an eigenvector of A, then so is any rescaled vector sv for s ∈ R, s ≠ 0.

Moreover, sv still has the same eigenvalue. For this reason, we usually only look for

unit eigenvectors.

 Suppose that a matrix A has n linearly independent eigenvectors, {v (1) , . . . , v (n)},

with corresponding eigenvalues {λ1, . . . , λn}.

 We may concatenate all eigenvectors to form a matrix V with one eigenvector per

column:

V = [v (1) , . . . , v (n) ]. Likewise, we can concatenate the eigenvalues to form a

vector λ = [λ1 , . . . , λn ]

 A = V diag(λ)V −1
 CONT.

 We have seen that constructing matrices with specific eigenvalues and eigenvectors
allows us to stretch space in desired directions.
 However, we often want to decompose matrices into their eigenvalues and eigenvectors.
 Doing so can help us to analyze certain properties of the matrix, much as decomposing
an integer into its prime factors can help us understand the behavior of that integer.
 Every real symmetric matrix can be decomposed into an expression using only real-
valued eigenvectors and eigenvalues:
A = Q Λ QT
where Q is an orthogonal matrix composed of eigenvectors of A, and Λ is a diagonal
matrix.
 CONT.
 The eigenvalue Λi,i is associated with the eigenvector in column i of Q, denoted as
Q:,i. Because Q is an orthogonal matrix, we can think of A as scaling space by λi in
direction v(i).

 The eigen decomposition of a matrix tells us many useful facts about the matrix.

 The matrix is singular if and only if any of the eigenvalues are zero.

 The eigen decomposition of a real symmetric matrix can also be used to optimize
quadratic expressions of the form f (x) = x TAx subject to ||x||2 = 1. Whenever x is
equal to an eigenvector of A, f takes on the value of the corresponding eigenvalue.

 The maximum value of f within the constraint region is the maximum eigenvalue and
its minimum value within the constraint region is the minimum eigenvalue.
 SINGULAR VALUE DECOMPOSITION
 we saw how to decompose a matrix into eigenvectors and eigenvalues.

 The singular value decomposition (SVD) provides another way to factorize a matrix, into
singular vectors and singular values.

 The SVD allows us to discover some of the same kind of information as the eigen
decomposition.

 However, the SVD more generally applicable.

 Every real matrix has a singular value decomposition, but the same is not true of the
eigenvalue decomposition.

 For example, if a matrix is not square, the eigen decomposition is not defined, and we must
use a singular value decomposition instead.
EXAMPLE FOR SINGULAR
3 0
VALUE DECOMPOSITION
 Find the SVD of 𝐴 =
0 4
 step 1: Compute 𝐴𝑇 𝐴
3 0 3 0 9 0
𝐴𝑇 𝐴 = =
0 4 0 4 0 16
 Step 2: Eigenvalues of 𝐴𝑇 𝐴
𝜆1 = 16, 𝜆2 = 9
 Step 3: Singular Values
𝜎1 = 16 = 4, 𝜎2 = 9 = 3
 Step 4: Right Singular Vectors (𝑉)
Eigenvectors of 𝐴𝑇 𝐴:
𝑉=𝐼
 Step 5: Left Singular Vectors (𝑈)
 𝑈 = 𝐴𝑉Σ−1 = 𝐼
 Final SVD
1 0 4 0 1 0
 𝐴 = 𝑈Σ𝑉 𝑇 =
0 1 0 3 0 1
 Given Matrix
1 2
𝐴=
2 1
 Step 1: Compute 𝐴𝑇 𝐴
5 4
𝐴𝑇 𝐴 =
4 5
 Step 2: Eigenvalues & Singular Values
𝜆1 = 9, 𝜆2 = 1
𝜎1 = 3, 𝜎2 = 1
 Step 3: Right Singular Vectors (𝑉)
Solve 𝐴𝑇 𝐴 − 9𝐼 𝑣 = 0:
1 1
⇒ 𝑣1 =
2 1

Solve 𝐴𝑇 𝐴 − 𝐼 𝑣 = 0:
11
⇒ 𝑣2 =
2 −1
1 1
2 2
𝑉=
1 1
−
2 2
Step 4: Left Singular Vectors (𝑈)
For 𝜎1 = 3
1 1 1
𝑢1 = 𝐴𝑣1 =
3 2 1
For 𝜎2 = 1
1 1
𝑢2 = 𝐴𝑣2 =
2 −1
1 1
2 2
𝑈=
1 1
−
2 2
 Final SVD
𝐴 = 𝑈Σ𝑉 𝑇
 CONT.
 Recall that the eigen decomposition involves analyzing a matrix A to discover a matrix V
of eigenvectors and a vector of eigenvalues λ such that we can rewrite A as
A = V diag(λ)V −1 .
 The singular value decomposition is similar, except this time we will write A as a product
of three matrices:

A = UDV T.

 Suppose that A is an m ×n matrix. Then U is defined to be an m ×m matrix, D to be an m

× n matrix, and V to be an n × n matrix.

 The elements along the diagonal of D are known as the singular values of the matrix A.

 The columns of U are known as the left-singular vectors.

 The columns of V are known as the right-singular vectors.

 We can actually interpret the singular value decomposition of A in terms of the eigen
decomposition of functions of A .

 The left-singular vectors of A are the eigenvectors of AAT.

 The right-singular vectors of A are the eigenvectors of ATA.

 The non-zero singular values of A are the square roots of the eigenvalues of AT A.

 The same is true for AAT.

 THE TRACE OPERATOR
 The trace operator gives the sum of all of the diagonal entries of a matrix:

 Tr(A) = ∑ Ai,i.

 The trace operator is useful for a variety of reasons.

 Some operations that are difficult to specify without resorting to summation notation
can be specified using matrix products and the trace operator.

 example, the trace operator provides an alternative way of writing the Frobenius
norm of a matrix: ||A||F = √Tr(AAT).

 For example, the trace operator is invariant to the transpose operator: Tr(A) = Tr(AT).

 , if the shapes of the corresponding matrices allow the resulting product to be

defined:
 CONT.
 Tr(ABC) = Tr(CAB) = Tr(BCA) or more generally,

 Tr(ℿ F (i)) = Tr(F (n) ℿ F (i) ).

This invariance to cyclic permutation holds even if the resulting product has a
different shape.

 For example, for A ∈ Rm×n and B ∈ Rn×m, we have Tr(AB) = Tr(BA) even
though AB ∈ Rm×m and BA ∈ Rn×n .
 THE DETERMINANT
 The determinant of a square matrix, denoted det(A), is a function mapping matrices to real
scalars.

 The determinant is equal to the product of all the eigenvalues of the matrix. The absolute
value of the determinant can be thought of as a measure of how much multiplication by the
matrix expands or contracts space.

 If the determinant is 0, then space is contracted completely along at least one dimension,
causing it to lose all of its volume. If the determinant is 1, then the transformation is
volume-preserving.
 PROBABILITY AND INFORMATION THEORY

 Probability theory is a mathematical framework for representing uncertain statements.

 In artificial intelligence applications, we use probability theory in two major ways.

1. The laws of probability tell us how AI systems should reason, so we design our algorithms
to compute or approximate various expressions derived using probability theory.

2. we can use probability and statistics to theoretically analyze the behavior of proposed AI
systems.

 Probability theory is a fundamental tool of many disciplines of science and engineering.

 Why Probability?
 CONT.
 Many branches of computer science deal mostly with entities that are entirely
deterministic and certain. A programmer can usually safely assume that a CPU will
execute each machine instruction flawlessly.

 It can be surprising that machine learning makes heavy use of probability theory.

 This is because machine learning must always deal with uncertain quantities, and
sometimes may also need to deal with stochastic (non-deterministic) quantities.

 Uncertainty and stochasticity can arise from many sources.

 CONT.
 There are three possible sources of uncertainty:

 [Link] stochasticity in the system being modeled.

 For example, most interpretations of quantum mechanics describe the dynamics of subatomic
particles as being probabilistic. We can also create theoretical scenarios that we postulate to
have random dynamics, such as a hypothetical card game where we assume that the cards are
truly shuffled into a random order.

 2. Incomplete observability. Even deterministic systems can appear stochastic when we

cannot observe all of the variables that drive the behavior of the system.
 CONT.

 For example, in the Monty Hall problem, a game show contestant is asked to choose
between three doors and wins a prize held behind the chosen door. Two doors lead to a
goat while a third leads to a car. The outcome given the contestant’s choice is
deterministic, but from the contestant’s point of view, the outcome is uncertain.

 3. Incomplete modeling. When we use a model that must discard some of the
information we have observed, the discarded information results in uncertainty in the
model’s predictions. For example, suppose we build a robot that can exactly observe the
location of every object around it.
 CONT.
 If the robot discretizes space when predicting the future location of these objects, then
the discretization makes the robot immediately become uncertain about the precise
position of objects: each object could be anywhere within the discrete cell that it was
observed to occupy.
 RANDOM VARIABLES
 A random variable is a variable that can take on different values randomly.

 x1 and x2 are both possible values that the random variable x can take on.

 For vector-valued variables, we would write the random variable as x and one of its values as x.

 A random variable is just a description of the states that are possible; it must be coupled with a
probability distribution that specifies how likely each of these states are.

 Random variables may be discrete or continuous. A discrete random variable is one that has a
finite or countably infinite number of states.

 Note that these states are not necessarily the integers; they can also just be named states that are
not considered to have any numerical value.

 A continuous random variable is associated with a real value.

 PROBABILITY DISTRIBUTIONS
 A probability distribution is a description of how likely a random variable or set of random variables is to take
on each of its possible states. The way we describe probability distributions depends on whether the variables
are discrete or continuous.
 Discrete Variables and Probability Mass Functions

 A probability distribution over discrete variables may be described using a probability mass function (PMF).
We typically denote probability mass functions with a capital P.
 Often we associate each random variable with a different probability mass function and the reader must infer
which probability mass function to use based on the identity of the random variable, rather than the name of the
function; P(x) is usually not the same as P(y).
 To be a probability mass function on a random variable x, a function P must satisfy the following properties:

• The domain of P must be the set of all possible states of x.

• ∀x ∈ x,0 ≤ P(x) ≤ 1. An impossible event has probability 0 and no state can be less probable than that. Likewise,
an event that is guaranteed to happen has probability 1, and no state can have a greater chance of occurring.
 CONT.
• ∑ P(x) = 1. We refer to this property as being normalized. Without this property, we could
obtain probabilities greater than one by computing the probability of one of many events
occurring.

Example: consider a single discrete random variable x with k different states. We can place a
uniform distribution on x make each of its states equally likely by setting its probability mass
function to P(x = xi) = 1 /k for all i.

We can see that this fits the requirements for a probability mass function.

The value 1/ k is positive ,because k is a positive integer.

∑ P(x = xi) = ∑ 1/ k = k/ k = 1,

so the distribution is properly normalized.

CONTINUOUS VARIABLES AND PROBABILITY DENSITY
FUNCTIONS
 When working with continuous random variables, we describe probability distributions using
a probability density function (PDF) rather than a probability mass function.

 To be a probability density function, a function p must satisfy the following properties:

• The domain of p must be the set of all possible states of x.

• ∀x ∈ x, p(x) ≥ 0. Note that we do not require p(x) ≤ 1.

• ∫ p(x)dx = 1.

 A probability density function p(x) does not give the probability of a specific state directly,
instead the probability of landing inside an infinitesimal region with volume δx is given by
p(x)δx.
 MARGINAL PROBABILITY
 we know the probability distribution over a set of variables and we want to know the
probability distribution over just a subset of them. The probability distribution over the
subset is known as the marginal probability distribution.

 Example: suppose we have discrete random variables x and y, and we know

 P(x, y).

 We can find P(x) with the sum rule: ∀x ∈ x, P(x = x) = ∑ P(x = x, y = y).

 For continuous variables, we need to use integration instead of summation: p(x) = ∫ p(x, y)dy.
 CONT.

Conditional Probability
 we are interested in the probability of some event, given that some
other event has happened. This is called a conditional probability.
We denote the conditional probability that y = y given x = x as P(y
= y | x = x). This conditional probability can be computed with the
formula
 P(y = y | x = x) = P(y = y, x = x)/ P(x = x) .
 The conditional probability is only defined when P(x = x) >0
 THE CHAIN RULE OF CONDITIONAL PROBABILITIES
 Any joint probability distribution over many random variables may be decomposed into
conditional distributions over only one variable:

P(x (1) , . . . , x (n)) = P(x (1))Π P(x (i) | x (1) , . . . , x (i−1)).

This is known as the chain rule or product rule of probability.

P(a, b, c) = P(a | b, c)P(b, c)

P(b, c) = P(b | c)P(c)

P(a, b, c) = P(a | b, c)P(b | c)P(c).

 INDEPENDENCE AND CONDITIONAL INDEPENDENCE
 Two random variables x and y are independent if their probability distribution can be
expressed as a product of two factors, one involving only x and one involving only y:

∀x ∈ x, y ∈

 Two random variables x and y are conditionally independent given a random variable z
if the conditional probability distribution over x and y factorizes in this way for every
value of z:

∀x ∈ x, y ∈ y, z ∈ z, p(x = x, y = y | z = z) = p(x = x | z = z)p(y = y | z = z)

 EXPECTATION, VARIANCE AND COVARIANCE
 The expectation or expected value of some function f(x) with respect to a probability
distribution P(x) is the average or mean value that f takes on when x is drawn from P. For
discrete variables this can be computed with a summation: Ex∼P [f(x)] =∑ P(x)f(x),

 while for continuous variables, it is computed with an integral:

 Ex∼p[f(x)] = ∫ p(x)f(x)dx.

 Expectations are linear, for example, Ex[αf(x) + βg(x)] = αEx[f(x)] + βEx[g(x)], when α and
β are not dependent on x. The variance gives a measure of how much the values of a function
of a random variable x vary as we sample different values of x from its probability
distribution:
 CONT.

 Var(f(x)) = E [ (f(x) − E[f(x)])2].

 Expectations are linear, for example,

 Ex[αf(x) + βg(x)] = αEx[f(x)] + βEx[g(x)],

 when α and β are not dependent on x. The variance gives a measure of how
much the values of a function of a random variable x vary as we sample
different values of x from its probability distribution:

 Var(f(x)) = E [ (f(x) − E[f(x)])2].

 COMMON PROBABILITY DISTRIBUTIONS
 Bernoulli Distribution

 The Bernoulli distribution is a distribution over a single binary random variable. It is

controlled by a single parameter φ ∈ [0, 1], which gives the probability of the random
variable being equal to .

 properties:

 P(x = 1) = φ

 P(x = 0) = 1 − φ

 P(x = x) = φ x (1 − φ) (1−x)
 Bernoulli Random Variable in ML
 In ML, we often model a binary label as a Bernoulli random variable:

 Y∈{0,1},Y∼Bernoulli(p)

 Where:

 𝑌 = 1→ positive class (spam, disease present, fraud)

 𝑌 = 0→ negative class (not spam, healthy, normal)

 𝑝 = 𝑃 𝑌 = 1 ∣ 𝑋 →model’s predicted probability

 Bernoulli in Logistic Regression

 Model

 Logistic regression directly models the Bernoulli parameter 𝑝:

1
 𝑝=𝜎 𝑧 = ,𝑧 = 𝑤𝑇𝑥 + 𝑏
1+𝑒 −𝑧
 Thus:
 𝑌 ∣ 𝑋 ∼ Bernoulli 𝑝
 Bernoulli in Generative Models
 Bernoulli Likelihood

 Used when data is binary:

 Pixel on/off (e.g., MNIST binarized images)

 Presence/absence features

 Example: Variational Autoencoders (VAEs)

 𝑝 𝑥 ∣ 𝑧 = ς𝑖 Bernoulli 𝑥𝑖 ∣ 𝑝𝑖

 Each pixel is modeled as a Bernoulli variable.

 Bernoulli in Naive Bayes (Bernoulli NB)

 Used when features are binary:

 𝑃 𝑥𝑗 ∣ 𝑦 ∼ Bernoulli 𝑝𝑗𝑦
 Applications:
 Text classification (word present / not present)

 Spam filtering
 Bernoulli in Regularization & Dropout
 Dropout

 Dropout uses Bernoulli random variables:

 𝑑𝑖 ∼ Bernoulli 𝑝

 𝑑𝑖 = 1 →neuron kept

 𝑑𝑖 = 0 →neuron dropped

Bernoulli vs Gaussian in ML
Aspect Bernoulli Gaussian
Data type Binary Continuous
Output activation Sigmoid Linear
Loss Binary Cross-Entropy MSE
Use case Classification Regression
Real-World ML Examples

Task Bernoulli Usage

Spam detection Email is spam / not spam

Medical diagnosis Disease present / absent

Fraud detection Fraud / not fraud

Click prediction Click / no-click

 CONT.

 Ex[x] = φ

 Varx(x) = φ(1 −φ)

 Multinoulli Distribution

 The multinoulli or categorical distribution is a distribution over a single

discrete variable with k different states, where k is finite. The multinoulli
distribution is parametrized by a vector p ∈ [0, 1] k−1 , where pi gives the
probability of the i-th state.
 GAUSSIAN DISTRIBUTION
 The most commonly used distribution over real numbers is the normal distribution, also known as the
Gaussian distribution:
N (x; µ, σ 2) = √ 1/ 2πσ2 exp (− 1/ 2σ2 )(x − µ) 2 .
 The two parameters µ ∈ R and σ ∈ (0,∞) control the normal distribution. The parameter µ gives the
coordinate of the central peak. This is also the mean of the distribution: E[x] = µ. The standard
deviation of the distribution is given by σ, and the variance by σ 2 .
 The normal distribution generalizes to Rn , in which case it is known as the multivariate normal
distribution. It may be parametrized with a positive definite symmetric matrix Σ:
N (x; µ, Σ) = √ (1 /(2π)ndet(Σ)) exp ((− ½) (x − µ)TΣ−1(x − µ)) .
 We can instead use a precision matrix β:
N (x; µ, β −1) = √ (det(β) / (2π)n ) exp ((− 1/ 2) (x − µ)Tβ(x − µ)) .
 EXPONENTIAL AND LAPLACE DISTRIBUTIONS
 In the context of deep learning, we often want to have a probability distribution with a sharp
point at x = 0.

 To accomplish this, we can use the exponential distribution:

 p(x; λ) = λ1x≥0 exp (−λx).

 The exponential distribution uses the indicator function 1x≥0 to assign probability zero to all
negative values of x.

 A closely related probability distribution that allows us to place a sharp peak of probability
mass at an arbitrary point µ is the Laplace distribution

 Laplace (x;µ, γ) – (1/ 2γ) exp (− |x − µ| / γ ) .

 THE DIRAC DISTRIBUTION AND EMPIRICAL DISTRIBUTION
 we wish to specify that all of the mass in a probability distribution clusters around a single point. This can be
accomplished by defining a PDF using the Dirac delta function , δ(x):
p(x) = δ(x − µ).
 A common use of the Dirac delta distribution is as a component of an empirical distribution,
pˆ(x) = (1/m) ∑ δ(x − x (i))
 which puts probability mass 1/m on each of the m points x (1) , . . . ,x (m) forming a given data set or collection
of samples.
 Mixtures of Distributions
 One common way of combining distributions is to construct a mixture distribution. A mixture
distribution is made up of several component distributions. On each trial, the choice of which component
distribution generates the sample is determined by sampling a component identity from a multinoulli
distribution:
P(x) = P(c = i)P(x | c = i)
 where P(c) is the multinoulli distribution over component identities.
 USEFUL PROPERTIES OF COMMON FUNCTIONS
 Certain functions arise often while working with probability distributions, especially the
probability distributions used in deep learning models.
 One of these functions is the logistic sigmoid:
 σ(x) = 1/ (1 + exp(−x)) .
 The logistic sigmoid is commonly used to produce the φ parameter of a Bernoulli distribution
because its range is (0,1), which lies within the valid range of values for the φ parameter.
 Another commonly encountered function is the softplus function (Dugas et al., 2001):
 ζ(x) = log (1 + exp(x)).
 The softplus function can be useful for producing the β or σ parameter of a normal distribution
because its range is (0,∞).
 The name of the softplus function comes from the fact that it is a smoothed or “softened” version
of x+ = max(0, x).
 CONT.
 The following properties are all useful enough that you may wish to memorize them:
σ(x) = exp(x) / (exp(x) + exp(0))
d /dx (σ(x)) = σ(x)(1 − σ(x))
1 − σ(x) = σ(−x)
log σ(x) = −ζ(−x)
d/ dx ζ(x) = σ(x)
∀x ∈ (0, 1), σ−1 (x) = log (x /1 − x)
∀x > 0, ζ−1(x) = log (exp(x) − 1)
ζ(x) = ∫ σ(y)dy
ζ(x) − ζ(−x) = x
 The function σ−1(x) is called the logit in statistics, but this term is more rarely used in machine learning.
 BAYES’ RULE
 We often find ourselves in a situation where we know P(y | x) and need to know P(x | y).
Fortunately, if we also know P(x),

 we can compute the desired quantity using Bayes’ rule: P(x | y) = P(x)P(y | x) P(y) .

 Note that while P (y) appears in the formula, it is usually feasible to compute

 P(y) = ∑ P(y | x)P(x), so we do not need to begin with knowledge of P(y).

 CASE STUDIES
Case Study 1: Face Recognition using Principal Component Analysis (PCA)
Problem: A security system needs to automatically identify individuals entering a building using their
facial images.
Concepts Used: Linear Algebra , Eigen Decomposition , Principal Component Analysis (PCA)
Explanation: A large dataset of facial images is collected. Each image is converted into a vector of pixel
values. Since images contain thousands of pixels, the dimensionality is very high.
Using PCA, the system reduces the dimensionality of the dataset by identifying the principal components
(Eigenfaces). These components represent the most important variations in facial images.
Process: 1. Convert each image into a vector. 2. Compute the covariance matrix.
Perform Eigen decomposition.
Select top eigenvectors to reduce dimensions.
Compare the new face image with stored face representations.
Result:
The system can identify faces accurately while reducing computation time.
Real-world applications:
Smartphone face unlock
Airport security systems
Criminal identification
Case Study 2: Movie Recommendation System using Probability
Problem: An online streaming platform wants to recommend movies to users based on their
viewing history.
Concepts Used: Random variables , Conditional probability , Bayes’ rule
Explanation: The system treats user preferences as random variables. For example:
A user likes Action movies
A user likes Science Fiction movies
Using Bayesian probability, the system calculates the probability that a user will like a particular
movie based on past preferences.
Example:
P(User likes movie | Genre = Action)
Using Bayes’ Rule, the recommendation engine updates probabilities as new user data arrives.
Result:
Users receive personalized movie suggestions.
Real-world applications:
Netflix recommendations
Amazon product suggestions
Spotify music recommendations
Case Study 3: Image Compression using Singular Value Decomposition (SVD)
Problem:
A company wants to store large numbers of images while reducing storage space.
Concepts Used: Matrices , Singular Value Decomposition (SVD)
Explanation: An image can be represented as a matrix of pixel values. Using SVD, the
matrix is decomposed into three smaller matrices.
Only the largest singular values are retained, while smaller values are discarded.
Process:
Represent image as matrix 𝐴
Apply SVD:
𝐴 = 𝑈Σ𝑉 𝑇
Keep only the top k singular values
Reconstruct the image
Result:
The image size is reduced while preserving most visual information.
Applications:
Image compression
Video streaming optimization
Data storage systems
Case Study 4: Predicting House Prices using Linear Least Squares
Problem:
A real estate company wants to predict house prices based on several factors.
Concepts Used: Linear algebra, Linear least squares , Gradient-based optimization
Features Used: House size, Number of bedrooms , Location, Age of the building
A linear model is created:
Price = w₁ × Size + w₂ × Bedrooms + w₃ × Location + b
The model parameters are estimated using least squares, which minimizes the
error between predicted and actual prices.
Result:
The company can estimate house prices for new properties.
Applications:
Real estate pricing tools
Economic forecasting
Market analysis
Case Study 5: Spam Email Detection using Probability and Bayes’ Rule
Problem:
Email providers need to detect whether an incoming email is spam or legitimate.
Concepts Used: Conditional probability, Bayes’ rule, Random variables
Each word in an email is treated as a random variable.
Example:
Word "Free"
Word "Offer"
Word "Prize"
Using Naive Bayes classifier, the system calculates:
Probability(Spam | Words in Email)
If the probability exceeds a threshold, the email is classified as spam.
Result:
Automatic spam filtering with high accuracy.
Applications:
Gmail spam detection
Email security systems
Fraud detection
Case Study 6: Training Neural Networks using Gradient-Based Optimization
Problem: A company develops a neural network to recognize handwritten digits.
Concepts Used: Gradient-based optimization, Numerical computation,Overflow and underflow
handling
The neural network minimizes a loss function using gradient descent.
However, during computation:
Very large numbers cause overflow
Very small numbers cause underflow
Numerical techniques such as normalization and log transformations are used to maintain
stable calculations.
Result:
The neural network successfully learns to recognize digits.
Applications:
Handwritten digit recognition
Optical character recognition (OCR)
Bank cheque processing
QUESTIONS
PART –A 2 Mark Questions
1. Define Tensors.
2. Distinguish between linear dependence and linear independent.
3. What is the use of a function called norm?
4. Define Eigen decomposition.
5. Define single value decomposition.
6. Define Variance and Co-variance
7. State Bayes’ theorem.
8. What is entropy?
9. What is gradient descent?
10. Define numerical overflow.
 PART-B

1. Define scalars, vectors, matrices, and tensors with examples

2. Explain norms and their importance in deep learning.
3. Define different types of matrix operations (addition, multiplication, transpose,
inverse) and their significance
4. Discuss different types of probability distributions. Explain any three with
examples.
3 0
5. Apply Singular Value Decomposition (SVD) to the matrix A=
0 4
6. Explain Eigen decomposition and its role in dimensionality reduction
4 1
7. Find the eigenvalues and eigenvectors of 𝑨 =
2 3
8. Examine overflow and underflow problems in numerical computation
9. Discuss about gradient-based optimization
10. Discuss the role of constrained optimization techniques in ML problems.
11. State and Explain Bayes’ Rule. Derive the formula using conditional probability.
12. Define Expectation and Variance of a random variable. Explain their
significance in probability and machine learning.
References
 Ian Goodfellow, Yoshua Bengio, Aaron Courville, “Deep Learning”, MIT Press,2016.

 Gilbert Strang, Introduction to Linear Algebra, Wellesley-Cambridge Press, 2016.

 [Link]

Expected Outcome
The lecture notes should:

a) Promote conceptual understanding

b) Encourage analytical and critical thinking

c) Align with Bloom’s Taxonomy

d) Support semester end examination preparation

e) Integrate theory with practical relevance

Standard Quote
“Linear algebra is the language of machine learning.” - Gilbert Strang

“Data in the real world is high-dimensional, and linear algebra provides the
tools to understand and manipulate it.”

Probability theory is nothing but common sense reduced to calculation.” -

Pierre-Simon Laplace

“Statistics is the grammar of science.” - Karl Pearson

Without data, you’re just another person with an opinion.” - W. Edwards

Deming
 COURSE OUTCOMES
 Understand the fundamental concepts of Linear Algebra, including scalars, vectors,
matrices, tensors, matrix operations, and different types of matrices.
 Apply mathematical techniques such as matrix norms, eigen decomposition, singular
value decomposition (SVD), and principal component analysis (PCA) for data
representation and dimensionality reduction.
 Explain the concepts of probability theory, including random variables, probability
distributions, marginal and conditional probability and Compute statistical measures
such as expectation, variance, covariance, and apply Bayes’ rule and information
theory concepts in data analysis and machine learning.
 Understand the principles of numerical computation, including overflow and
underflow issues in computer arithmetic.
 Apply optimization techniques, including gradient-based optimization, constrained
optimization, and linear least squares methods for solving numerical and machine
learning problems.
Machine Learning
Basics
M E PALANIVEL
Professor
 Introduction
• Deep learning is a specific kind of machine learning.

• In order to understand deep learning well, one must have a solid understanding of the
basic principles of machine learning.

• A machine learning algorithm is an algorithm that is able to learn from data.

• what do we mean by learning? Mitchell (1997) provides the definition

• There are multiple ways to define machine learning. But the one which is perhaps
most relevant, concise and accepted universally is the one stated by Tom M. Mitchell,
Professor of Machine Learning Department, School of Computer Science, Carnegie
Mellon University. Tom M. Mitchell has defined machine learning as:
 Definition

• “A computer program is said to learn from experience E with respect to some

class of tasks T and performance measure P, if its performance at tasks in T, as
measured by P, improves with experience E.”

• A formal definition of the word “task,” the process of learning itself is not the
task. Learning is our means of attaining the ability to perform the task. For
example, if we want a robot to be able to walk, then walking is the task.

• We could program the robot to learn to walk

• In order to evaluate the abilities of a machine learning algorithm, we must design a
quantitative measure of its performance.

• Usually this performance measure P is specific to the task T being carried out by the
system.

• Machine learning algorithms can be broadly categorized as unsupervised or supervised

by what kind of experience they are allowed to have during the learning process.

• Most of the learning algorithms in this book can be understood as being allowed to
experience an entire dataset.

• It means a machine can be considered to learn if it is able to gather experience by doing a

certain task and improve its performance in doing the similar tasks in the future.
Traditional programming VS machine learning.
Data science models
 THE MACHINE LEARNING PROCESS
• Data Collection and Preparation : To solve a problem we have to collect the from
scratch.
• If the problem is completely new, so that appropriate data can be chosen, then this process
should be merged with the next step of feature selection, so that only the required data is
collected.
• For supervised learning, target data is also needed, which can require the involvement of
experts in the relevant field and significant investments of time.
• Finally, the quantity of data needs to be considered. Machine learning algorithms need
significant amounts of data, preferably without too much noise, but with increased dataset
size comes increased computational costs, and the sweet spot at which there is enough data
without excessive computational overhead is generally impossible to predict.
 Cont.
• Feature Selection :when we looked at possible features that might be useful for
coin recognition.
• It consists of identifying the features that are most useful for the problem under
examination.
• This invariably requires prior knowledge of the problem and the data; our common
sense was used in the coins example above to identify some potentially useful
features and to exclude others.
• As well as the identification of features that are useful for the learner, it is also
necessary that the features can be collected without significant expense or time, and
that they are robust to noise and other corruption of the data that may arise in the
collection process.
 Cont.
• Algorithm Choice : Given the dataset, the choice of an appropriate algorithm (or
algorithms) is what this book should be able to prepare you for, in that the knowledge
of the underlying principles of each algorithm and examples of their use is precisely what is
required for this.

• Parameter and Model Selection : For many of the algorithms there are parameters that
have to be set manually, or that require experimentation to identify appropriate values.

• Training: Given the dataset, algorithm, and parameters, training should be simply the use
of computational resources in order to build a model of the data in order to predict the
outputs on new data.
 Cont.
• Evaluation :Before a system can be deployed it needs to be tested and
evaluated for accuracy on data that it was not trained on.

• This can often include a comparison with human experts in the field, and the
selection of appropriate metrics for this comparison.
 MODEL REPRESENTATION AND INTERPRETABILITY

• The goal of supervised machine learning is to learn or derive a target function

which can best determine the target variable from the set of input variables.
• It is an extent of generalization.

• The input data is just a limited, specific view and the new, unknown data in the test
data set may be differing quite a bit from the training data.

• Fitness of a target function approximated by a learning algorithm determines how

correctly it is able to classify a set of data it has never seen.
 Fitting: Overfitting and Underfitting
in ML
• Overfitting and Underfitting are the two main problems that occur in machine learning and
degrade the performance of the machine learning models.

• The main goal of each machine learning model is to generalize well.

Here generalization defines the ability of an ML model to provide a suitable output by
adapting the given set of unknown input.

• It means after providing training on the dataset, it can produce reliable and accurate output.

• Hence, the underfitting and overfitting are the two terms that need to be checked for the
performance of the model and whether the model is generalizing well or not.
 Overfittin
g
• Overfitting occurs when our machine learning model tries to cover all the data
points or more than the required data points present in the given dataset.

• Because of this, the model starts caching noise and inaccurate values present in
the dataset, and all these factors reduce the efficiency and accuracy of the model.

• The overfitted model has low bias and high variance.

• The chances of occurrence of overfitting increase as much we provide training to our

model. It means the more we train our model, the more chances of occurring the
overfitted model.

• Overfitting is the main problem that occurs in S-L.

Ex: Linear Regression O/P
As we can see from the above graph,
the model tries to cover all the data points
present in the scatter plot. It may look
efficient, but in reality, it is not so.

Because the goal of the regression model

to find the best fit line,

but here we have not got any best fit,

so, it will generate the prediction errors
 How to avoid the overfitting in
model
• Both overfitting and underfitting cause the degraded performance of the machine learning
model. But the main cause is overfitting, so there are some ways by which we can reduce
the occurrence of overfitting in our model.

• Cross-Validation

• Training with more data

• Removing features

• Early stopping the training

• Regularization

• Ensembling
 Underfitting

• A typical case of underfitting may occur when trying to represent a non-linear data
with a linear model

• Many times underfitting happens due to unavailability of sufficient training data.

• Underfitting results in both poor performance with training data as well as poor
generalization to test data.
• Underfitting can be avoided by
1. using more training data
2. reducing features by effective feature selection
Underfitting and Overfitting
of models
 Cont.

• Overfitting refers to a situation where the model has been designed in such a way that it
emulates the training data too closely.

• any specific deviation in the training data, like noise or outliers, gets embedded in the model.
• It adversely impacts the performance of the model on the test data.

• Overfitting, in many cases, occur as a result of trying to fit an excessively complex

model to closely match the training data.

• The target function, in these cases, tries to make sure all training data points are correctly
partitioned by the decision boundary.
• This exact nature is not replicated in the unknown test data set.
• The target function results in wrong classification in the test data set.
What is Underfitting?
• When a model has not learned the patterns in the training data well and is unable to
generalize well on the new data, it is known as underfitting.

• An underfit model has poor performance on the training data and will result in unreliable
predictions.

• Underfitting occurs due to high bias and low variance.

 Reasons for Underfitting
• Data used for training is not cleaned and contains noise (garbage values) in it

• The model has a high bias

• The size of the training dataset used is not enough

• The model is too simple

Ways to Tackle Underfitting

• Increase the number of features in the dataset

• Increase model complexity

• Reduce noise in the data

• Increase the duration of training the data

 What Is a Good Fit In Machine
Learning?
• To find the good fit model, you need to look at the performance of a machine learning model
over time with the training data.

• As the algorithm learns over time, the error for the model on the training data reduces, as
well as the error on the test dataset.

• If you train the model for too long, the model may learn the unnecessary details and the
noise in the training set and hence lead to overfitting.

• In order to achieve a good fit, you need to stop training at a point where the error starts to
increase.
 Overfitting
• Overfitting can be avoided by
1. using re-sampling techniques like k-fold cross validation
2. hold back of a validation data set

3. remove the nodes which have little or no predictive power for the given machine
learning problem.

• Both underfitting and overfitting result in poor classification quality which is reflected
by low classification accuracy.
Estimat
ors
• An estimator is a function or algorithm that tries to estimate a target variable from
given input features.

• Point Estimation is the attempt to provide the single best prediction of some quantity
of interest.

• Quantity of interest can be:

• A single parameter

• A vector of parameters — e.g., weights in linear regression

• A whole function
Point estimator
• A point estimator is just a formula or function that gives you a best guess
for an unknown value (parameter) based on data.

• We write this guess as:

q = true value
q̂ (read as "q hat") = estimated value
(from data)
Example:
• You want to estimate the average exam score of all students in a
school.
But you only collect the scores of 5 students:
• Scores: 78, 85, 90, 82, 75
• Point estimate of the average score (mean):

• 78+85+90+82+75
μ^= -------------------------- =82
• 5
• So, μ̂ = 82 is your point estimate of the true average score.
 Example 2:
 Out of the past 30 days, it rained on 12 days.
 Point estimate of the probability of rain:
 p^=12/30=0.4

 So, p̂ = 0.4 is your point estimate of the true probability of rain

on a given day.
 Bias and
Variance
• These are one of the ways to evaluate a machine-learning model.

• There are two types of error in machine learning:

• [Link] error and [Link] error.

• Bias and Variance come under reducible error.

• Bias is the error occurring between the model’s predicted value and the actual value.

• Let Y be the true value of a parameter, and

• let Y^ be an estimator of Y based on a sample of data.

• Then, the bias of the estimator Y^ is given by: Bias(Y^)=E(Y^)-Y

• where E(Y^) is the expected value of the estimator Y^.

• It is the measurement of the model that how well it fits the data.

• Low Bias: Low bias value means fewer assumptions are taken to build the
target function. In this case, the model will closely match the training
dataset.

• High Bias: High bias value means more assumptions are taken to build the
target function. In this case, the model will not match the training dataset
closely.
 To Reduce High
Bias
• Use a more complex model

• Polynomial Regression, CNN, RNN & Image Processing

• Increase the number of features

• adding more features to train the dataset will increase the complexity of the model.

• Reduce Regularization of the model

• To prevent overfitting problem and to increase generalization ability of the model

• Increase the size of the training data

• where E[Y^] is the expected value of the predicted values. Here expected value is averaged
over all the training data.
• Variance errors are either low or high-variance errors
• Low variance: Low variance means that the model is less sensitive to changes in the
training data. This is the case of underfitting when the model fails to generalize on both
training and test data.
• High variance: High variance means that the model is very sensitive to changes in the
training data. This is the case of overfitting when the model performs well on the training
data but poorly on test data

Type Meaning Result

Model doesn’t change much Underfitting — poor on

Low Variance with different training both training and test
data data

Model changes too much Overfitting — good on

High Variance with different training training, bad on test
data data
Maximum Likelihood
• Maximum Likelihood Estimation (MLE) is a method for estimating model
parameters by choosing the values that make the observed data most probable under
the assumed statistical model.

• Likelihood vs Probability
• Probability: data is random, parameters fixed
• Likelihood: data is fixed, parameters vary

• 𝑝 𝑥 ∣ 𝜃 viewed as a function of 𝜃
• Why MLE is Central in ML
• MLE underlies:

• Linear regression

• Logistic regression

• Neural networks

• Naive Bayes

• Gaussian Mixture Models

• Hidden Markov Models

• Training a model = maximizing likelihood

• Likelihood for i.i.d. Data

• Assume 𝑛independent samples:

• ℒ 𝜃 = ς𝑛𝑖=1 𝑝 𝑥𝑖 ∣ 𝜃

• Taking logs (for numerical stability):

• log ℒ 𝜃 = σ𝑛𝑖=1 log 𝑝 𝑥𝑖 ∣ 𝜃

• MLE in Deep Learning

• Neural networks are trained by:

• 𝜃: = arg min σ𝑛𝑖=1 − log 𝑝 𝑦𝑖 ∣ 𝑥𝑖 𝜃

𝜃

• SGD ≈ stochastic MLE optimization

• Properties of MLE

• Advantages

• Consistent (converges to true parameters)

Efficient (low variance asymptotically)
Simple and general

• Limitations

• Overfits with small data

Sensitive to outliers
No uncertainty estimates
 Bayesian Statistics
• Why Bayesian Statistics in ML

• Classical (frequentist) machine learning:

• Learns point estimates of parameters

• Treats parameters as fixed but unknown

• Often gives no uncertainty measure

• Bayesian machine learning:

• Treats parameters as random variables

• Learns full probability distributions

• Naturally models uncertainty, noise, and prior knowledge

• Bayes’ Theorem
𝑃(𝐷∣𝜃)𝑃(𝜃)
• P(θ∣D)=
𝑃(𝐷)

•θ: model parameters

•D: observed data
•p(θ) : prior
•p(D∣θ): likelihood
•p(θ∣D): posterior
•p(D) : evidence (marginal likelihood)
• Bayesian Learning Framework
• Choose a prior 𝑝 𝜃
• Define a likelihood 𝑝 𝐷 ∣ 𝜃
• Compute the posterior
• Make predictions by integrating over parameters
• Bayesian Prediction
• Instead of using a single parameter value:
• p(y∗∣x∗,D)=∫p(y∗∣x∗,θ)p(θ∣D)dθ
• Bayesian Neural Networks
• Weights are probability distributions, not scalars
• 𝑤∼𝑝 𝑤
• Predictive distribution
• 𝑝 𝑦 ∣ 𝑥, 𝐷 = ‫𝑥 ∣ 𝑦 𝑝׬‬. 𝑤 𝑝 𝑤 ∣ 𝐷 𝑑𝑤
• Strengths of Bayesian ML

• Handles uncertainty
Incorporates prior knowledge
Prevents overconfidence
Works well with small data

• Limitations

• Computational cost
Approximation errors
Difficult posterior design •Data is scarce
•Interpretability is needed
• When to Use Bayesian Methods •Decisions are high-risk
•Uncertainty matters
 Bias – variance trade-off

• In supervised learning, the class value assigned by the learning model built based on
the training data may differ from the actual class value.

• This error in learning can be of two types – errors due to ‘bias’ and error due to
‘variance’.
Errors due to ‘Bias’:
• While making predictions, a difference occurs between prediction values made by
the model and actual values/expected values, and this difference is known as bias
errors or Errors due to bias.
 Cont.

• Errors due to bias arise from simplifying assumptions made by the model to make the
target function less complex or easier to learn.
• It is due to underfitting of the model.
• Parametric models generally have high bias

• These algorithms have a poor performance on data sets which are complex in
nature
• Underfitting results in high bias.
 Errors due to ‘Variance’

• variance tells that how much a random variable is different from its expected value.

• Ideally the difference in the data sets should not be significant and the model trained
using different training data sets should not be too different.

• in case of overfitting, since the model closely matches the training data, even a small
difference in training data gets magnified in the model.
• A high variance model leads to overfitting.
• It Increase model complexities.
Combinations of
Bias-Variance
 Combinations of Bias-Variance

[Link]-Bias,Low-Variance:The combination of low bias and low variance

shows an ideal machine learning model. However, it is not possible
practically.

2. Low-Bias, High-Variance: With low bias and high variance, model

predictions are inconsistent and accurate on average. This case occurs when the
model learns with a large number of parameters and hence leads to an overfitting.
 Cont.
3. High-Bias, Low-Variance: With High bias and low variance, predictions are consistent
but inaccurate on average. This case occurs when a model does not learn well with the
training dataset or uses few numbers of the parameter. It leads to underfitting problems
in the model.

4. High-Bias, High-Variance: With high bias and high variance, predictions are inconsistent
and also inaccurate on average.

Ways to reduce High Bias:

• Increase the input features as the model is underfitted.

• Decrease the regularization term.

• Use more complex models, such as including some polynomial features.

 Cont.

Ways to Reduce High Variance:

• Reduce the input features or number of parameters as a model is overfitted.

• Do not use a much complex model.
• Increase the training data.
• Increase the Regularization term.
 Bias-Variance

• If the model is very simple with fewer parameters, it may have low variance and
high bias.

• If the model has a large number of parameters, it will have high variance and
low bias.

• It is required to make a balance between bias and variance errors, and this balance
between the bias error and variance error is known as the Bias-Variance trade-
off.
 TYPES OF MACHINE LEARNING
• Machine learning can be classified into three broad categories.

[Link] learning - Also called predictive learning. A machine

predicts the class of unknown objects based on prior class related
information of similar objects.

[Link] learning – Also called descriptive learning. A machine

finds patterns in unknown objects by grouping similar objects together.

[Link] learning – A machine learns to act on its own to achieve

the given goals.
Cont.
 Supervised learning
• Supervised learning is the machine learning task of learning a function
that maps an input to an output based on example input-output pairs.

• f(X) = Y

• X= { (X1,Y1), (X2,Y2), ……, (Xn,Yn)}

• Where X1, X2,….,Xn are inputs and Y1,Y2….,Yn are target outputs

• In supervised learning, each example in the training set is a pair

consisting of an input object (typically a vector) and an output value.
 Cont.
• A supervised learning algorithm analyzes the training data and produces a

function, which can be used for mapping new examples.

• In the optimal case, the function will correctly determine the class labels for

unseen instances.

• Both classification and regression problems are supervised learning problems.

• A wide range of supervised learning algorithms are available, each with its

strengths and weaknesses. There is no single learning algorithm that works best

on all supervised learning problems.

 Cont.
• There is a set of data (the training data) that consists of a set of input data that has target

data, which is the answer that the algorithm should produce, attached.

• This is usually written as a set of data (xi, ti), where the inputs are xi, the targets are ti, and

the i index suggests that we have lots of pieces of data, indexed by i running from 1 to

some upper limit N .

• If we had examples of every possible piece of input data, then we could put them together

into a big look-up table, and there would be no need for machine learning at all.

• The thing that makes machine learning better than that is generalization: the algorithm

should produce sensible outputs for inputs that weren’t encountered during learning.
 Cont.
Given:
– a set of input features X 1, … , X 𝑛
– A target feature 𝑌

– a set of training examples where the values for the input features and the
target features are given for each example

– a new example, where only the values for the input features are given

Predict the values for the target features for the new example.
– classification when Y is discrete
– regression when Y is continuous
Cont.
 Supervised
Learning

X y
Input1 Output1 New Input x
Input2 Output2
Input3 Output3 Learning
Algorithm
Model
Input-n Output-n Output y
Example
Consider the following data regarding patients entering a clinic. The
data consists of the gender and age of the patients and each patient is
labeled as “healthy” or “sick”.
 Cont.
• Based on this data, when a new patient enters the clinic, how can one predict whether
he/she is healthy or sick?
Classification

• The Classification algorithm is a Supervised Learning technique that is used to

identify the category of new observations on the basis of training data.

• In Classification, a program learns from the given dataset or observations and then
classifies new observation into a number of classes or groups Such as, Yes or No, 0 or
1, Spam or Not Spam, cat or dog, etc. Classes can be called as targets/labels or
categories.
 Cont.
• There are mainly four types of classification tasks that one may come across, these are:

• Binary Classification

• Multi-Class Classification

• Multi-Label Classification

• Imbalanced Classification

• Binary Classification- This type of classification involves separating the dataset into two categories. It means
that the output variable can only take two values.

• The task of labeling an e-mail as "spam" or "not spam." The input variable here will be the content of the e-mail
that we are trying to classify. The output variable is represented by 0 for "not spam" and 1 for "spam”.

• Multi-Class Classification- In multi-class classification, the output variable can have more than two possible
values.

• Example-Email received from friend, relative, unknown contact etc.,

Linear Classification
• Binary Classification problem
• The data above the red line
belongs to class ‘x’
x • The data below red line
x x
x x belongs to class ‘o’
x • Examples: SVM, Perceptron,
x x x o
o Probabilistic Classifiers
x o o
ooo
o o
o o o o

59
Cont.
 Cont.
• If the image is of a round object, it is put under one category, while if the image
is of a triangular object, it is put under another category.
• In which category the machine should put an image of unknown category, also
called a test data in machine learning parlance, depends on the information it
gets from the past data, which we have called as training data.

• The whole problem revolves around assigning a label or category or class to a

test data based on the label or category or class information that is imparted by
the training data.
 CONT.
• There are number of popular machine learning algorithms which help in
solving classification problems.

• To name a few, Naïve Bayes, Decision tree, and k-Nearest Neighbour

algorithms are adopted by many machine learning practitioners.

• Some typical classification problems include: Image classification,

Prediction of disease, Win– loss prediction of games, Prediction of
natural calamity like earthquake, flood, etc., Recognition of handwriting.
 Unsupervised learning
• Def: Unsupervised learning is a type of machine learning in which
models are trained using unlabeled dataset and are allowed to act
on that data without any supervision.

• Unsupervised learning is a type of machine learning algorithm used to

draw inferences from datasets consisting of input data without labeled
responses.

• In unsupervised learning algorithms, a classification or categorization is

not included in the observations.
 Cont.
• There are no output values and so there is no estimation of functions. Since the
examples given to the learner are unlabeled, the accuracy of the structure that is output by
the algorithm cannot be evaluated.

• The most common unsupervised learning method is cluster analysis, which is used for
exploratory data analysis to find hidden patterns or grouping in data.
• In unsupervised learning, the objective is to take a dataset as input and try to find natural
groupings or patterns within the data elements or records.
• Therefore, unsupervised learning is often termed as descriptive model and the process of
unsupervised learning is referred as pattern discovery or knowledge discovery.
• One critical application of unsupervised learning is customer segmentation.
 Cont
.
• Reasons for unsupervised learning:

 Unsupervised learning is helpful for finding useful insights from the data.
 Unsupervised learning is much similar as a human learns to think by
their own experiences, which makes it closer to the real AI.
 Unsupervised learning works on unlabeled and uncategorized data which
make unsupervised learning more important.
 In real-world, we do not always have input data with the corresponding
output so to solve such cases, we need unsupervised learning.
Types of Unsupervised Learning Algorithm
 Unsupervised
Learning
X Clusters
Input1
Input2
Input3 Learning
Algorithm
Input-n
 Clustering
• Different measures of similarity can be applied for clustering. One of
the most commonly adopted similarity measure is distance.

• Two data items are considered as a part of the same cluster if the
distance between them is less.

• In the same way, if the distance between the data items is high, the
items do not generally belong to the same cluster. This is also known as
distance-based clustering.
Cont
.
cont.
 Cont.
• Unsupervised Learning algorithms:

• K-means clustering

• Hierarchal clustering

• Anomaly detection

• Neural Networks

• Principle Component Analysis

• Independent Component Analysis

• Apriori algorithm

• Singular value decomposition

 Cont.
• Advantages

• Unsupervised learning is used for more complex tasks as compared to supervised

learning because, in unsupervised learning, we don't have labeled input data.

• Unsupervised learning is preferable as it is easy to get unlabeled data in comparison to

labeled data.

• Disadvantages

• Unsupervised learning is intrinsically more difficult than supervised learning as it does

not have corresponding output.

• The result of the unsupervised learning algorithm might be less accurate as input data is
not labeled, and algorithms do not know the exact output in advance.
 Reinforcement learning

• It tries to improve its performance of doing the task.

• When a sub-task is accomplished successfully, a reward is given.

• When a sub-task is not executed correctly, obviously no reward is given.

• This continues till the machine is able to complete execution of the whole
task.

• This process of learning is known as reinforcement learning.

Cont.
 Reinforcement
Learning
Action at

State st St+1
Agent Environment
Reward rt rt+1
 Reinforcement
Learning
Action at
State st St+1
RLearner Environment
Reward rt rt+1

values
upda
Stat

Q-
te
e,

Policy
Action at
stat
acti
on
Be

e
st

State st St+1
User Environment
Reward rt rt+1
 Building a Machine Learning
Algorithm
• Most ML algorithms follow a simple recipe:
Dataset + Model + Cost Function + Optimization Procedure

• Example: Linear regression is one such instance.

• Understanding this recipe allows creating a wide variety of algorithms.

Flexibility of Components
• Dataset, model, cost, and optimizer are mostly independent.

• Changing any component leads to new algorithms.

• This modularity supports a wide variety of machine learning methods.

 Cost Functions and Statistical
Estimation
• Cost functions often encode statistical objectives.

• Most common: negative log-likelihood → equivalent to maximum

likelihood estimation (MLE).

• Additional terms may be added for regularization:

• Example: Weight decay in linear regression

• 𝐽𝜆 𝑤 𝑏 =∣∣ 𝑤 ∣∣22 −𝔼𝑥,𝑦∼𝑝 log 𝑝model 𝑦 ∣ 𝑥

data
Nonlinear Models and Numerical
Optimization

•Nonlinear models usually cannot be optimized in closed form.

•Require iterative numerical optimization methods:
•Gradient Descent
•Stochastic Gradient Descent
•Variants like Adam, RMSProp
• PCA Example
• Loss function for first PCA vector:

• 𝐽 𝑤 = 𝔼𝑥∼𝑝 ∣∣ 𝑥 − 𝑟 𝑥 𝑤 ∣∣22
data
• Model constraints: ∣∣ 𝑤 ∣∣= 1, 𝑟 𝑥 = 𝑤 ⊤ 𝑥 𝑤

• Optimization finds principal components without labels.

• Cost Functions That Are Hard to Evaluate

• Some cost functions cannot be evaluated exactly.

• Approximate gradients can still be used for iterative optimization.

• Allows flexible learning even in computationally difficult scenarios.

• Special-Case Optimizers
• Some algorithms require specialized optimization:
• Decision Trees
• k-Means
• Reason: Their cost functions have flat regions
unsuitable for gradient-based methods.
• Understanding ML Algorithms as a Recipe
• Recognizing the common recipe helps:
• See ML algorithms as a taxonomy of methods.
• Understand why similar methods work for similar tasks.
• Avoid thinking of algorithms as unrelated or ad hoc.
Stochastic Gradient Descent
• Deep learning is powered by one very important algorithm: stochastic gradient descent or
SGD.

• Stochastic gradient descent is an extension of the gradient descent algorithm.

• Stochastic Gradient Descent (SGD) is an optimization algorithm used to minimize a loss

function by iteratively updating model parameters using gradients computed from one
training example (or a small batch) at a time.

• A recurring problem in machine learning is that large training sets are necessary for good
generalization, but large training sets are also more computationally expensive.

• The cost function used by a machine learning algorithm often decomposes as a sum over
training examples of some per-example loss function.
• Idea of Stochastic Gradient Descent

• Instead of using the full dataset, SGD:

• Uses one randomly chosen data point (or a mini-batch)

• Makes frequent noisy updates

• Reaches good solutions much faster

• SGD Update Rule

• For a training example 𝑥𝑖 𝑦𝑖 :

•θ: model parameters
• 𝜃: = 𝜃 − 𝜂∇𝜃 ℓ 𝜃 𝑥𝑖 𝑦𝑖
•η: learning rate
•ℓ: loss for one data point
• Variants
Data set
Type
Batch GD Entire data
SGD One data point
Mini-batch SGD Small batch
Challenges Motivating Deep Learning
• The simple machine learning algorithms work very well on a wide variety of
important problems.

• However, they have not succeeded in solving the central problems in AI, such
as recognizing speech or recognizing objects.

• The development of deep learning was motivated in part by the failure of

traditional algorithms to generalize well on such AI tasks.
The Curse of Dimensionality
• Many machine learning problems become exceedingly difficult when the number of
dimensions in the data is high.

• This phenomenon is known as the curse of dimensionality.

• Of particular concern is that the number of possible distinct configurations of a set of

variables increases exponentially as the number of variables increases.

• The curse of dimensionality arises in many places in computer science, and especially so in
machine learning
Local Constancy and Smoothness
Regularization
• The most widely used of these implicit “priors” is the smoothness prior or local constancy prior.

• This prior states that the function we learn should not change very much within a small region.

• Many simpler algorithms rely exclusively on this prior to generalize well, and as a result they fail to
scale to the statistical challenges involved in solving Ailevel tasks.

• we explain why the smoothness prior alone is insufficient for these tasks.

• There are many different ways to implicitly or explicitly express a prior belief that the learned function
should be smooth or locally constant.

• All of these different methods are designed to encourage the learning process to learn a function f∗ that
satisfies the condition

• f∗(x) ≈ f ∗(x + ε)) for most configurations x and small change ε .

 Manifold Learning
• An important concept underlying many ideas in machine learning is that of a manifold.

• A is a connected region. Mathematically, it is a set manifold of points, associated with a

neighborhood around each point.

• From any given point, the manifold locally appears to be a Euclidean space. In everyday life,
we experience the surface of the world as a 2-D plane, but it is in fact a spherical manifold in
3-D space.

• The definition of a neighborhood surrounding each point implies the existence of

transformations that can be applied to move on the manifold from one position to a
neighboring one.

• In the example of the world’s surface as a manifold, one can walk north, south, east, or west.
• Although there is a formal mathematical meaning to the term “manifold,” in machine
learning it tends to be used more loosely to designate a connected set of points that can be
approximated well by considering only a small number of degrees of freedom, or
dimensions, embedded in a higher-dimensional space.

• Each dimension corresponds to a local direction of variation.

• In the context of machine learning, we allow the dimensionality of the manifold to vary
from one point to another.

• This often happens when a manifold intersects itself.

• For example, a figure eight is a manifold that has a single dimension in most places but two
dimensions at the intersection at the center.
DEEP FEED FORWARD NEURAL NTWORK

M E PALANIVEL
Professor
SITAMS
 Introduction
• In the last chapter we saw that while linear models are easy to understand and they can only
identify straight lines, planes, or hyperplanes.

• This is not usually enough, because the majority of interesting problems are not linearly
separable.

• We have pretty much decided that the learning in the neural network happens in the weights.
So, to perform more computation it seems sensible to add more weights.

• There are two things that we can do: add some backwards connections, so that the output
neurons connect to the inputs again, or add more neurons.

• The first approach leads into recurrent networks.

• We will instead consider the second approach.
• We can add neurons between the input nodes and the outputs, and this will make more
complex neural networks, such as the one shown in Figure
Learning XOR Problem
• we can check that a prepared network can solve the two-dimensional XOR problem,
something that we have seen is not possible for a linear model like the Perceptron.

• A suitable network is shown in Figure

•
 Deep Feed forward Neural Network model
▪The input to the network is an n-dimensional vector
▪ The network contains L − 1 hidden layers (2, in this case) having n neurons each
Finally, there is one output layer containing k neurons (say, corresponding to k
classes)
▪ Each neuron in the hidden layer and output layer can be split into two parts : pre-
activation and activation (ai and h i are vectors)
▪ The input layer can be called the 0-th layer and the output layer can be called the (L)-
th layer Wi ∈ Rn×n and bi ∈ Rn are the weight and bias between layers i − 1 and i (0 < i <
L ) WL ∈ Rn×k and bL ∈ Rk are the weight and bias between the last hidden layer and
the output layer (L = 3 in this case)
• A DFFNN Model consists of three layers. They are

• 1. Input layer

• 2. Hidden layer

• 3. Output layer

• Each layer is made of units or neurons.

• The inputs to the model corresponds to the features measured to each training tuple.

• The input are feed simultaneously into the units making up the input layer.

• They are then weighted and fed simultaneously to a second layer of hidden neurons
known as hidden layer.

• The output of hidden layer units can be input to another hidden layer and so on.
• The number of hidden layers is arbitrary.

• The weighted outputs of the last hidden layer are input to units making up the output layer, which
emits the DFFNN MODEL prediction output.

• The units in the input layer are called input units.

• The units in the hidden layer are called hidden units.

• The units in the output layer are called output units.

• The number of hidden layers in the above model is L-1 and one output layer, therefore we call it as L
layered DFFNN Model.

• The input layer is not counted because it serves only to pass the input values to the next layer.

• The network is feed-forward in that none of the weights cycles back to an input unit or to an output
unit of a previous layer
• The training the DFFNN Model consists of two phases:
• 1. Forward phase
• 2. Backward phase
Forward Phase

The pre-activation at layer i is given by

a i (x) = bi + W i h i − 1 (x)

The activation at layer i is given by

h i (x) = g(a i (x))
12/
9

where g is called the activation function example, logistic, tanh, linear, etc.
The activation at the output layer is given by

f (x ) = h L (x ) = O (aL (x ))

where O is the output activation function (for example, softmax, linear, etc.)
 N
Data: { x i , y i }i = 1
Model:

ŷ i = f (x i ) = O (W 3 g(W 2 g(W 1 x + b1 ) + b2 ) + b3 )

Parameters:
13/
9 θ = W 1 , .., W L , b1 , b2 , ..., bL (L = 3)
Algorithm: Gradient Descent with Back-
propagation (we will see soon)
O b j e c t i ve / L o ss/ E r ro r function: Say,
1 ΣN Σk
min (ŷ i j −y i j ) 2
N
i=1 j=1

In general, min L (θ)

where L (θ) is some function of the parameters 7/9

• Backward phase
• Backpropagation allows us to readjust our weights to reduce output error.
• The error is propagated backward during backpropagation from the output to the input layer.
• This error is then used to calculate the gradient of the cost function with respect to each
weight.
• Essentially, backpropagation aims to calculate the negative gradient of the cost function.
• This negative gradient is what helps in adjusting of the weights.
• It gives us an idea of how we need to change the weights so that we can reduce the cost
function.
❖ To find this weight, we must navigate down the cost function until we find its
minimum point.
Gradient Descent

❖ The weights are adjusted using a process called gradient descent.

❖ Gradient descent is an optimization algorithm that is used to find the weights that
minimize the cost function.

❖ Minimizing the cost function means getting to the minimum point of the cost
function.

❖ So, gradient descent aims to find a weight corresponding to the cost function’s
minimum point.
 Navigating towards minimum cost function depends on 2 things
Direction - Gradient Descent or Gradient Ascent
Direction is determined by calculating the Gradients.
Specifically, we aim to find the negative gradient.
This is because a negative gradient indicates a decreasing slope.
A decreasing slope means that moving downward will lead us to the minimum point.
Step Size – Learning Rate
Step Size is determined by Learning Rate.
The learning rate is a tuning parameter that determines the step size at each iteration of gradient
descent.
1t determines the speed at which we move down the slope.
❖ Let’s say you are playing a game where the players are at the top of a mountain, and they are
asked to reach the lowest point of the mountain. Additionally, they are blindfolded. So, what
approach do you think would make you reach the lake?

❖ The best way is to observe the ground and find where the land descends. From that position,
take a step in the descending direction and iterate this process until we reach the lowest point.
Using the initial weight and the gradient and learning rate, we can determine the subsequent
weights.
 From the graph of the cost function, we can see that:

1. To start descending the cost function, we first initialize a random weight.

2. Then, we take a step down and obtain a new weight using the gradient and learning
rate. With the gradient, we can know which direction to navigate. We can know the
step size for navigating the cost function using the learning rate.

3. We are then able to obtain a new weight using the gradient descent formula.

4. We repeat this process until we reach the minimum point of the cost function.

5. Once we’ve reached the minimum point, we find the weights that correspond to the
minimum of the cost function.
• Algorithm: gradient descent()

t ← 0;

max iterations ← 1000;

I nitializ e w0 , b0 ;
while t++ < max iterations do

wt+ 1 ← wt −η∇wt ;

bt+ 1 ← bt −η∇bt ;

end
The DFFNN Model in Practice
• We apply these ideas to using the MLP to find solutions to four different types of
problem: regression, classification, time-series prediction, and data compression.

• For the MLP with one hidden layer there are (L + 1) X M + (M + 1)X N weights,
where L, M, N are the number of nodes in the input, hidden, and output layers,
respectively.

• The extra +1s come from the bias nodes, which also have adjustable weights.

• This is a potentially huge number of adjustable parameters that we need to set during
the training phase.

• the more training data there is, the better for learning, although the time that the
algorithm takes to learn increases.
• Number of Hidden Layers

• There are two other considerations, which is the choice of the number of hidden nodes, and the
number of hidden layers.

• We can use the back-propagation algorithm for a network with as many layers as we like, although it
gets progressively harder to keep track of which weights are being updated at any given time.

• When to Stop Learning

• The training of the MLP requires that the algorithm runs over the entire dataset many times, with the
weights changing as the network makes errors in each iteration.

• Setting some predefined number N of iterations, and running until that is reached runs the risk that
the network has overfitted by then, or not learnt sufficiently, and only stopping when some
predefined minimum error is reached might mean the algorithm never terminates, or that it overfits.
• Using both of these options together can help, as can terminating the learning once the error stops
decreasing.

• If we plot the sum-of-squares error during training, it typically reduces fairly quickly during the first
few training iterations, and then the reduction slows down as the learning algorithm performs small
changes to find the exact local minimum.

• We don’t want to stop training until the local minimum has been found, but, as we’ve just
discussed, keeping on training too long leads to overfitting of the network.

• This is where the validation set comes in useful. We train the network for some predetermined
amount of time, and then use the validation set to estimate how well the network is generalizing.
At some stage the error on the validation set will start increasing again, because the network has
stopped learning about the function that generated the data, and started to learn about the noise that is
in the data itself as shown in Figure .
At this stage we stop the training. This technique is called early stopping.
Back Propagation Algorithm
• Back Propagation algorithm is a learning or Training algorithm works on Deep
Feed Forward Neural Network model to solve Classification or Prediction
problems.

• Input : a) A training data set D, consisting of Training tuples and their

associated target values.

b) the learning rate η

c) A Deep Feed Forward Neural Network model

• Output : A trained Deep Feed Forward Neural Network model

Method :
1. Initialization : Initialize all Weights and Biases in the DFFNN Model to small
random values ( i.e [0,1] or [-1,+1])

2. Training phase :
i) Forward phase :

For each training tuple or input vector X in D , propagate the inputs in the forward

Input layer : for each input layer unit or node j

Oj = Ij i.e output of an input node is its actual input value

Hidden layers : for each Hidden layer unit or node j , Compute the net input of neuron j
with respect to the previous layer
 𝐼𝑗 = ෍ 𝑤𝑖𝑗 𝑂𝑖 + 𝜃𝑗
𝑖

• Compute the activation or output of each neuron j in the hidden layers

using the sigmoid activation function
1
• Oj = f (Ij) =
1 + 𝑒 −𝐼𝑗

• Output layer : For Output layer unit or node j , Compute the net input of
neuron j with respect to the previous layer

𝐼𝑗 = ෍ 𝑤𝑖𝑗 𝑂𝑖 + 𝜃𝑗
𝑖
• Compute the activation or output of each neuron j in the Output layer using the
sigmoid activation function

1
Oj = f (Ij) =
1 + 𝑒 −𝐼𝑗

• Backward Phase:
• Compute Error at each neuron j in the output layer

Errj = Oj(1-Oj) (Tj - Oj)

• For each neuron j in the Hidden layer

Errk = Oj(1-Oj) ∑ Errk wjk

• For each weight wij in the Neural Network
• The weight increment equation is
∆wij = η Errj Oj
• The weight update equation
wij = wij + ∆wij
• For each bias θj in the Neural Network
• The bias increment equation is
∆ θj = η Errj
• The bias update equation
θj = θj + ∆ θj
 EXAMPLES OF USING THE DFFNN Model
• A Regression Problem

• We will take a set of samples generated by a simple mathematical function, and try to learn
the generating function (that describes how the data was made) so that we can find the values
of any inputs, not just the ones we have training data for.

• We can now train an MLP on the data. There is one input value, x and one output value t,
so the neural network will have one input and one output. Also, because we want the output
to be the value of the function, rather than 0 or 1, we will use linear neurons at the output.
We don’t know how many hidden neurons we will need yet, so we’ll have to experiment to
see what works.

• Before getting started, we need to normalise the data using the method and then separate the
data into training, testing, and validation sets.
 Classification with the DFFNN Model
• Using the DFFNNM for classification problems is not radically different once the output
encoding has been worked out.

• The inputs are easy: they are just the values of the feature measurements (suitably
normalized). There are a couple of choices for the outputs. The first is to use a single linear
node for the output, y, and put some thresholds on the activation value of that node.

• For example, for a four-class problem

• However, this gets impractical as the number of classes gets large, and the boundaries are
artificial; what about an example that is very close to a boundary, say y = 0.5?

• We arbitrarily guess that it belongs to class C3, but the neural network doesn’t give us any
information about how close it was to the boundary in the output, so we don’t know that this
 Cont.

• A more suitable output encoding is called 1-of-N encoding. A separate node is used to
represent each possible class, and the target vectors consist of zeros everywhere except for
in the one element that corresponds to the correct class, e.g., (0, 0, 0, 1, 0, 0) means that
the correct result is the 4th class out of 6. We are therefore using binary output values (we
want each output to be either 0 or 1).

• Once the network has been trained, performing the classification is easy:

• simply choose the element yk of the output vector that is the largest element of y (in
mathematical notation, pick the yk for which yk > yj j = k; means for all, so this
statement says pick the yk that is bigger than all other possible values yj).

• This generates an unambiguous decision, since it is very unlikely that two output neurons
will have identical largest output values.
• This is known as the hard-max activation function (since the neuron with the highest
activation is chosen to fire and the rest are ignored).

• An alternative is the soft-max function, which and which has the effect of scaling the output
of each neuron according to how large it is in comparison to the others, and making the total
output sum to 1.

• So if there is one clear winner, it will have a value near 1, while if there are several P values
that are close to each other, they will each have a value of about 1 , where p is the number of
output neurons that have similar values.
Case study
• A research team is developing a neural network model named LeafYield, designed to
predict the crop yield (in kg) from leaf-level features extracted via sensors and imaging.
Each sample contains 5 normalized numerical features: Leaf length, Leaf width, Color
intensity, Water content, Light absorption level. The Deep Feed forward Neural Network as
follows and the neurons in the hidden layer 1, Hidden layer 2 and output layer are having
sigmoid activation function
•
• Given the input vector x= [0.1,0.5,0.3,0.7,0.2]T and

W1 = 1 0101
0 1010
1 -1 0 1 1
01110
• Find the pre-activation vector for hidden layer 1, Apply sigmoid activation to the above
vector obtained find the output from the hidden layer1 , Find the pre-activation vector for
hidden layer 2, Apply sigmoid activation to the above vector obtained find the output from
the hidden layer 2 and Find the pre-activation vector for output layer , Apply sigmoid
activation to the above vector obtained find the output from the Output layer.
 Case Studies
Case Study 1: Predicting Student Performance using Supervised Learning
Problem: A university wants to predict whether a student will pass or fail based on previous academic records.
Concepts Used: Supervised Learning, Bias and Variance, Hyperparameters, Validation Set
Explanation:
The university collects historical student data including: Attendance, Assignment scores ,Internal exam marks ,Study hours
A supervised learning model is trained using labeled data (Pass/Fail).
The dataset is divided into:
Training set – to train the model
Validation set – to tune hyperparameters
Test set – to evaluate performance
During training:
If the model is too simple → Underfitting
If the model memorizes training data → Overfitting
Hyperparameters such as learning rate and model complexity are adjusted using the validation set.
Outcome:
The system predicts students who may need academic support.
Case Study 2: Customer Segmentation using Unsupervised Learning
Problem: A retail company wants to group customers based on their buying behavior.
Concepts Used: Unsupervised Learning, Feature analysis, Data clustering
Explanation:
Customer data includes: Purchase frequency, Amount spent, Product categories
Since there are no predefined labels, the company uses unsupervised learning algorithms
such as clustering.
The algorithm automatically identifies groups like:
High-value customers
Occasional buyers
Budget shoppers
Outcome:
The company can design targeted marketing strategies.
Applications:
Marketing personalization
Customer analytics
E-commerce recommendations
• Case Study 3: Spam Email Detection using Bayesian Statistics
• Problem: Email providers must automatically identify spam emails.
• Concepts Used: Bayesian Statistics, Maximum Likelihood, Probability distributions
• Explanation:
Each word in an email is treated as a feature.
• The system calculates:
• Probability(Spam | Email Content)
• Using Bayes’ Rule, the classifier evaluates whether the email is spam or legitimate.
• Training involves estimating probabilities using Maximum Likelihood Estimation (MLE)
from historical email data.
• Outcome:
Emails are automatically filtered into Spam or Inbox.
• Applications:
• Gmail spam filtering
• Email security systems
Case Study 4: Product Recommendation using Stochastic Gradient Descent
Problem: An online shopping platform wants to recommend products to users.
Concepts Used: Stochastic Gradient Descent (SGD) , Gradient-based optimization,
Hyperparameter tuning
Explanation:
The system learns user preferences based on: Previous purchases , Product ratings , Browsing
history
A recommendation model is trained using SGD, which updates model parameters using small
batches of data.
Advantages:
Faster training
Efficient for large datasets
Hyperparameters like: Learning rate, Batch size are tuned using validation data.
Outcome:
Users receive personalized product suggestions.
Applications:
Amazon product recommendations
E-commerce personalization
• Case Study 5: Solving the XOR Problem using Deep Feedforward
Networks
• Problem: Early neural networks failed to learn the XOR logical function.

• Concepts Used:Deep Feedforward Networks

• Hidden Units, Non-linear activation functions

• Explanation:
The XOR function:
Input 1 Input 2 Output
0 0 0
0 1 1
1 0 1
1 1 0
• A single-layer perceptron cannot solve XOR because it is not linearly
separable.

• By introducing a hidden layer, a deep feedforward network learns nonlinear

decision boundaries.

• Outcome:
The neural network correctly models the XOR function.

• Importance:
This example demonstrated the need for multi-layer neural networks, which
led to modern deep learning.
Case Study 6: Handwritten Digit Recognition using Backpropagation
Problem: Banks need to automatically recognize handwritten digits on cheques.
Concepts Used: Deep feedforward networks , Backpropagation ,Gradient-based learning
Explanation: A neural network is trained on thousands of digit images.
Training process:
Input image passes through layers (forward propagation).
The model predicts the digit.
The error between predicted and actual value is computed.
Backpropagation calculates gradients.
Weights are updated using gradient descent.
Hidden units learn features like: Edges ,Curves, Digit shapes
Outcome:
The model accurately recognizes handwritten numbers.
Applications:
Bank cheque processing
Postal code recognition
Document digitization
Case Study 7: Image Recognition Challenges Motivating Deep Learning
Problem: Traditional machine learning struggled with complex tasks such as image
recognition.
Challenges: High-dimensional data, Feature engineering difficulty, Large datasets
Solution:
Deep learning models automatically learn hierarchical features.
Example:
In image recognition:
Layer 1 → detects edges
Layer 2 → detects shapes
Layer 3 → detects objects
This hierarchical learning makes deep learning effective for complex tasks.
Applications:
Self-driving cars
Medical image analysis
Face recognition
Questions
PART –A 2 Mark Questions
1. Define Machine Learning.

2. Distinguish between Over fitting and Underfitting.

3. Give two examples of hyperparameters.

4. What is the bias–variance tradeoff.

5. State Bayes’ theorem.

6. Give two examples of supervised learning algorithms.

7. How many hidden layers are typically needed to solve XOR

8. Draw a simple Deep Feed Forward Network graph.

9. What is gradient-based learning

10. Write the equation of weight updating in Backpropagation algorithm.

 Part-B
1. Explain the concepts of overfitting and underfitting with suitable examples.
2. Discuss the importance of Bayesian statistics in machine learning.
3. Differentiate between supervised and unsupervised learning with examples.
4. Discuss about learning XOR problem
5. Describe the architecture and working principle of a Deep Feedforward Neural
Network (DFNN) with a neat diagram.
6. Explain the backpropagation algorithm in deep learning. Illustrate how errors are
propagated and weights are updated during training.
7. Explain Stochastic Gradient Descent (SGD) and describe the challenges that motivate
deep learning.
8. Write short notes on Bias and Variance trade-off.
9. Explain how Batch, Mini-batch, and Stochastic Gradient Descent update model
parameters during training.
10. Compare the advantages and disadvantages of Batch, Mini-batch, and Stochastic
Gradient Descent
11. Why is Mini-batch Gradient Descent commonly used in deep learning?
References
• Ian Goodfellow, Yoshua Bengio, Aaron Courville, “Deep Learning”, MIT
Press, 2016.

• Stephen Marsland, “Machine learning an Algorithmic Perspective”, CRC

Press, 2nd Edition 2015

• Anuradha Srinivasa Raghavan and Vincy Joseph, “Machine learning”, Wiley

Publisher, 2019.
• [Link]
• [Link]
Expected Outcome
The lecture notes should:

a) Promote conceptual understanding

b) Encourage analytical and critical thinking

c) Align with Bloom’s Taxonomy

d) Support semester end examination preparation

e) Integrate theory with practical relevance

Course Outcomes
• Understand the fundamental concepts of Machine Learning, including underfitting,
hyperparameters, validation sets, estimators, bias–variance tradeoff, and the challenges motivating
deep learning.

• Differentiate between Supervised Learning and Unsupervised Learning techniques and apply
appropriate algorithms to different types of datasets.

• Analyze and implement optimization techniques, particularly Stochastic Gradient Descent and
gradient-based learning, for training machine learning models.

• Design and understand the structure of Deep Feedforward Neural Networks, including hidden
units, architecture design, and learning nonlinear functions such as XOR.

• Apply Backpropagation and other differentiation algorithms to train neural networks and
improve model performance.
 Standard Quote
• “Artificial intelligence is the new electricity transforming every industry.”

- Andrew Ng

• “Data is the new oil, but machine learning is the engine that makes it valuable.”

• “Every dataset tells a story, and machine learning helps us understand it.”

• “Deep learning is a powerful set of techniques for learning representations from

large amounts of data.”

• - Ian Goodfellow

• “Deep learning turns data into knowledge.”