M.

Sc (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

DEEP LEARNING
Module 1 — Applied Math & ML Basics + Numerical Computation
[Link] (IT) | Year II / Semester IV | SVKM's UPG College | 2024-25
Duration: 15 Lectures · Exam Weightage: ~25%

Section A — Applied Math & ML Basics: Linear Algebra — Scalars, Vectors, Matrices, Tensors,
Matrix Multiplication, Identity & Inverse Matrices, Linear Dependence & Span, Norms, Special
Matrices & Vectors, Eigendecomposition
Section B — Numerical Computation: Overflow & Underflow, Poor Conditioning, Gradient-Based
Optimization, Constraint Optimization

Page 1 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

SECTION A — APPLIED MATH & MACHINE LEARNING

BASICS

1. Why Mathematics for Deep Learning?

Deep learning is fundamentally a mathematical discipline. Understanding the math behind neural
networks allows you to debug failing models, design better architectures, and understand research
papers. The three pillars are Linear Algebra (data representation), Calculus (learning via
gradients), and Probability (uncertainty modelling).
• Linear Algebra: How data is represented (vectors, matrices) and how transformations work
(matrix multiplication). Every neural network layer is a linear algebra operation.
• Calculus & Optimization: How models learn — computing gradients and descending the
loss surface.
• Probability & Statistics: How uncertainty is modelled and how distributions are learned.
• Numerical Computation: How computers handle real numbers — floating-point precision,
stability issues, and efficient algorithms.
This module covers Linear Algebra and Numerical Computation — the mathematical foundation for
everything in deep learning.

2. Scalars
📌 Scalar: A single real number. The simplest mathematical object. Denoted by lowercase italic
letters: x, y, n, λ.
Scalars are the fundamental unit of numerical information. In deep learning, scalars appear as:
• Learning rate: α = 0.001 — a scalar controlling step size in gradient descent.
• Loss value: L = 0.324 — a single number measuring model error.
• Bias term: b = 0.5 — a scalar added to a neuron's weighted sum.
• Temperature: τ = 1.0 in softmax — controls prediction sharpness.
• Regularisation parameter: λ = 0.01 in L2 regularisation.
x ∈ ℝ (x is a real-valued scalar)
n ∈ ℕ (n is a natural number scalar, e.g., number of training
examples)

✎ Example: Learning rate α = 0.001, dropout rate p = 0.5, number of classes K = 10 — all scalars.

💡 EXAM TIP: Scalars: know the notation (lowercase italic), that they are single numbers, and
3 deep learning examples (learning rate, loss value, regularisation param).

3. Vectors
📌 Vector: An ordered array of n scalars. Denoted by lowercase bold letters: x, y, w. A vector in ℝⁿ
has n elements called components or entries.

Page 2 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

Vectors are one-dimensional arrays. In deep learning, they are the most common data structure —
a single training example, a weight array, an embedding.

3.1 Vector Notation and Components

x = [x₁, x₂, x₃, ..., xₙ]ᵀ (column vector, n-dimensional)
• Column vector: n×1 matrix. Default convention in linear algebra and deep learning.
• Row vector: 1×n matrix. Transpose of a column vector: xᵀ.
• Indexing: x₁ is the first component (1-indexed in math), x[0] in Python (0-indexed).
✎ Example: x = [1.2, -0.5, 3.7, 2.1]ᵀ is a 4-dimensional feature vector representing one training
example with 4 features (e.g., area, bedrooms, age, price per sq ft of a house).

3.2 Vectors in Deep Learning

• Input vector: x ∈ ℝᵈ — d features describing one data sample. E.g., a 784-dim vector for a
28×28 MNIST image (each pixel is one feature).
• Weight vector: w ∈ ℝᵈ — weights of one neuron. The dot product w·x + b gives the
neuron's pre-activation.
• Bias vector: b ∈ ℝⁿ — one bias per neuron in a layer of n neurons.
• Output vector: y ∈ ℝᴷ — K class probabilities from softmax layer.
• Embedding vector: e ∈ ℝᴰ — dense representation of a word/token in D-dimensional
space.
• Gradient vector: ∇L ∈ ℝᵈ — partial derivatives of loss w.r.t. all parameters.

3.3 Vector Operations

Vector Addition:
z = x + y where zᵢ = xᵢ + yᵢ (element-wise)
• Add corresponding elements. Vectors must have the same dimension.
✎ Example: x = [1, 2, 3], y = [4, 5, 6] → x+y = [5, 7, 9]
Scalar Multiplication:
z = αx where zᵢ = α·xᵢ (scale each element by scalar α)

✎ Example: 2 × [1, 2, 3] = [2, 4, 6]

Dot Product (Inner Product):
x·y = xᵀy = Σᵢ xᵢyᵢ = x₁y₁ + x₂y₂ + ... + xₙyₙ
• Result is a SCALAR. Fundamental operation in neural network forward pass.
• Geometric meaning: x·y = ||x|| ||y|| cos(θ) where θ is angle between vectors.
• If x·y = 0: vectors are orthogonal (perpendicular). Used in attention mechanisms.
✎ Example: w = [2, 3], x = [4, 1] → w·x = 2×4 + 3×1 = 8 + 3 = 11 (neuron output before activation)

💡 EXAM TIP: Vectors: know notation (bold lowercase), column vector default, dot product
formula and result is scalar. Explain 3 deep learning uses: input vector, weight vector,
gradient vector.

Page 3 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

4. Matrices
📌 Matrix: A 2-dimensional array of scalars arranged in rows and columns. An m×n matrix has m
rows and n columns. Denoted by uppercase bold letters: A, W, X.
A ∈ ℝᵐˣⁿ (m rows, n columns)

4.1 Matrix Notation

• Element notation: Aᵢⱼ or A[i,j] — element at row i, column j.
• Row i of A: Aᵢ,: (all columns of row i).
• Column j of A: A:,ⱼ (all rows of column j).
✎ Example: A = [[1, 2, 3], [4, 5, 6]] is a 2×3 matrix. A₁₂ = 2 (row 1, col 2). A₂₁ = 4 (row 2, col 1).

4.2 Matrices in Deep Learning

• Weight matrix: W ∈ ℝⁿˣᵈ — weights connecting d input neurons to n output neurons. W[i,j]
is the weight from input j to neuron i.
• Data matrix: X ∈ ℝᴺˣᵈ — N training examples, d features each. Each row is one example.
• Activation matrix: A ∈ ℝᴺˣⁿ — activations of n neurons for N examples.
• Jacobian matrix: J ∈ ℝᵐˣⁿ — all first-order partial derivatives (used in backpropagation).
• Covariance matrix: Σ ∈ ℝᵈˣᵈ — encodes relationships between features (PCA, Gaussian
models).
• Attention matrix: A ∈ ℝᵀˣᵀ — attention weights between T tokens (Transformer).

4.3 Matrix Transpose

(Aᵀ)ᵢⱼ = Aⱼᵢ (swap rows and columns)
• m×n matrix → n×m matrix after transpose.
• Column vector x ∈ ℝⁿ transposed: xᵀ ∈ ℝ¹ˣⁿ (row vector).
• Properties: (Aᵀ)ᵀ = A, (AB)ᵀ = BᵀAᵀ, (A+B)ᵀ = Aᵀ+Bᵀ.
✎ Example: A = [[1,2],[3,4],[5,6]] (3×2) → Aᵀ = [[1,3,5],[2,4,6]] (2×3)

5. Tensors
📌 Tensor: A generalisation of scalars, vectors, and matrices to an arbitrary number of dimensions
(axes). An n-dimensional tensor has n axes.
Tensors are the universal data container in deep learning. PyTorch and TensorFlow are named
after this concept — all data flows through the network as tensors.

5.1 Tensor Dimensions (Ranks)

Rank (Axes) Name Deep Learning Example

0 Scalar Loss value L = 0.324
1 Vector Input features x ∈ ℝ⁷⁸⁴,
weight vector w
2 Matrix Weight matrix W ∈ ℝⁿˣᵈ, data
matrix X ∈ ℝᴺˣᵈ

Page 4 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

Rank (Axes) Name Deep Learning Example

3 3D Tensor Batch of images: (batch_size,
height, width) = (32, 28, 28)
4 4D Tensor Batch of colour images:
(batch_size, channels, height,
width) = (32, 3, 224, 224)
5 5D Tensor Batch of videos: (batch_size,
frames, channels, height,
width)

5.2 Tensors in Deep Learning Practice

• Image data: Shape (N, C, H, W) — N samples, C colour channels (1 grayscale, 3 RGB), H
height, W width.
• Text data: Shape (N, T) — N sequences of length T. Or (N, T, D) with D-dim embeddings.
• CNN feature maps: Shape (N, C_out, H_out, W_out) after a convolution.
• RNN hidden state: Shape (layers, N, hidden_size) — num_layers × batch × features.
• Transformer attention: Shape (N, heads, T, T) — batch × attention heads × sequence ×
sequence.
✎ Example: import torch; x = [Link](32, 3, 224, 224) creates a 4D tensor for a batch of 32 RGB
images of size 224×224. [Link] = [Link]([32, 3, 224, 224]).

💡 EXAM TIP: Tensors: explain ranks 0-5 with DL examples. A 28×28 grayscale image = rank-2
tensor. A batch of 32 such images = rank-3 tensor (32,28,28). RGB images batch = rank-4
(32,3,224,224).

6. Multiplying Matrices and Vectors

Matrix multiplication is the core operation of neural networks. Every forward pass through a layer is
fundamentally a matrix multiplication.

6.1 Matrix-Vector Product

y = Ax where A ∈ ℝᵐˣⁿ, x ∈ ℝⁿ → y ∈ ℝᵐ
• Result: m-dimensional vector. Each output element yᵢ = row i of A dotted with x.
yᵢ = Σⱼ Aᵢⱼ xⱼ (dot product of row i with x)
• Deep learning meaning: y = Wx + b — the LINEAR LAYER. W ∈ ℝⁿˣᵈ transforms d-dim
input x to n-dim output y. This is called an affine transformation.
✎ Example: W = [[2,1],[0,3]], x = [4,2]ᵀ → y = [2×4+1×2, 0×4+3×2] = [10, 6]ᵀ. Two neurons, 2 inputs
each.

6.2 Matrix-Matrix Product

C = AB where A ∈ ℝᵐˣⁿ, B ∈ ℝⁿˣᵖ → C ∈ ℝᵐˣᵖ
• CRITICAL RULE: Inner dimensions must match. (m×n) × (n×p) = (m×p). The n's must be
equal.

Page 5 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

Cᵢⱼ = Σₖ Aᵢₖ Bₖⱼ (dot product of row i of A with column j of B)

• Requires n multiplications and n-1 additions per output element. Total: m×p×n operations.
• NOT commutative: AB ≠ BA in general.
• Associative: (AB)C = A(BC). Distributive: A(B+C) = AB + AC.
✎ Example: A (2×3) × B (3×2) = C (2×2). Valid. | A (2×3) × B (2×3) — INVALID (3≠2). Error!

6.3 Matrix Multiplication as Batched Neural Network Forward Pass

When processing N examples simultaneously (mini-batch), the linear layer becomes:
Y = XW + b where X ∈ ℝᴺˣᵈ (batch), W ∈ ℝᵈˣⁿ (weights), b ∈ ℝⁿ → Y ∈
ℝᴺˣⁿ
• X: Each ROW is one training example (N examples × d features).
• W: Columns are weight vectors for each output neuron.
• b: Added to every row (broadcasting).
• Y: Each ROW is the pre-activation output for one example.
• This single matrix multiplication simultaneously processes the entire batch — GPU
parallelism exploited.
✎ Example: Batch of 64 images, 784 features → 512 hidden neurons: X(64×784) × W(784×512) +
b(512) = Y(64×512). One op.

6.4 Element-wise (Hadamard) Product

C = A ⊙ B where Cᵢⱼ = Aᵢⱼ × Bᵢⱼ (same shape required)
• NOT standard matrix multiplication. Both matrices must have identical shape.
• Used in: LSTM gates (forget gate ⊙ cell state), attention masking, dropout masks.
✎ Example: A = [[1,2],[3,4]], B = [[5,6],[7,8]] → A ⊙B = [[5,12],[21,32]]

💡 EXAM TIP: Matrix multiplication is GUARANTEED exam content. Remember:

(m×n)×(n×p)=(m×p) — inner dims must match. y=Wx+b is a linear layer. NOT commutative.
Show element formula Cᵢⱼ = ΣAᵢₖBₖⱼ.

7. Identity and Inverse Matrices

7.1 Identity Matrix
📌 Identity Matrix (I): A square matrix with 1s on the main diagonal and 0s everywhere else.
Denoted Iₙ for n×n. Acts like the number 1 in multiplication.
I₃ = [[1,0,0],[0,1,0],[0,0,1]]
AI = IA = A for any matrix A (identity element of matrix
multiplication)
Ix = x for any vector x (multiplication by I returns x unchanged)
• Why important: Defines the concept of inverse (A A⁻¹ = I). Used in theoretical derivations.
• In deep learning: Residual connections y = F(x) + x add the identity to the transformation.
Identity mapping is the 'skip connection'.
✎ Example: In ResNet: output = F(x, {Wᵢ}) + x. The x term is an identity shortcut — gradients flow
directly through it, solving vanishing gradient.

Page 6 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

7.2 Matrix Inverse

⁻ Inverse Matrix (A⁻¹): For square matrix A, its inverse A⁻¹ satisfies: A A⁻¹ = A⁻¹ A = I. Only
square matrices MAY have an inverse.
AA⁻¹ = A⁻¹A = I
• A matrix is invertible (non-singular) if and only if its determinant ≠ 0.
• Solution to Ax = b: Multiply both sides by A⁻¹ → x = A⁻¹b. Solves linear systems.
• NOT all matrices are invertible: If A is singular (det = 0), A⁻¹ does not exist.
Properties of Inverse:
• (A⁻¹)⁻¹ = A
• (AB)⁻¹ = B⁻¹A⁻¹ (reverse order!)
• (Aᵀ)⁻¹ = (A⁻¹)ᵀ
When A⁻¹ Does NOT Exist (Singular Matrix):
• Rows/columns are linearly dependent.
• System Ax = b has infinitely many solutions or no solution.
• In practice: Computing A⁻¹ is numerically unstable and expensive (O(n³)). Use LU
decomposition or iterative solvers instead.
✎ Example: A = [[2,1],[1,1]] → det(A) = 2×1-1×1 = 1 ≠ 0 → invertible. A ⁻¹ = [[1,-1],[-1,2]]. Check:
AA⁻¹ = I ✓

⁻EXAM TIP: Identity and Inverse: AI = IA = A (identity property), AA⁻¹ = I (inverse definition), x
= A⁻¹b solves Ax=b, inverse exists iff det ≠ 0. ResNet identity connection uses Ix = x concept.

8. Linear Dependence and Span

8.1 Linear Combination
📌 Linear Combination: A new vector formed by multiplying each vector in a set by a scalar and
summing the results.
v = α₁v₁ + α₂v₂ + ... + αₙvₙ where αᵢ are scalar coefficients

✎ Example: v = 2[1,0] + 3[0,1] = [2,3]. The vector [2,3] is a linear combination of the standard basis
vectors.

8.2 Span
📌 Span: The set of ALL possible linear combinations of a set of vectors. span({v₁,...,vₙ}) is the
entire 'space' reachable by combining these vectors.
• If span({v₁,...,vₙ}) = ℝⁿ, the vectors span the entire n-dimensional space.
• Significance: The column span of A determines which vectors b make Ax = b solvable.
✎ Example: span({[1,0],[0,1]}) = all of ℝ² (can reach any 2D point). span({[1,0],[2,0]}) = only the x-
axis (both vectors point in same direction).

8.3 Linear Independence

📌 Linear Independence: A set of vectors {v₁,...,vₙ} is linearly independent if NO vector in the set
can be expressed as a linear combination of the others. Equivalently: α₁v₁+...+α ₙv ₙ=0 implies all
αᵢ=0.
• Linearly DEPENDENT: At least one vector is redundant — can be written as combination of
others.

Page 7 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

• Linearly INDEPENDENT: No redundancy — each vector adds a new 'direction'.

✎ Example: v₁=[1,0,0], v₂=[0,1,0], v₃=[0,0,1] — linearly independent (standard basis). | v₁=[1,2],
v₂=[2,4] — dependent (v₂=2v₁).
Why Important in Deep Learning:
• Matrix rank: Number of linearly independent rows/columns. Rank(A) = dimensionality of the
column space.
• Full rank matrix: All rows and columns are independent. Invertible. No redundant
information.
• Feature redundancy: If two input features are linearly dependent (perfectly correlated), one
is redundant — doesn't add information. PCA removes such dependencies.
• Neural network width: If neurons become linearly dependent (same outputs), the network
loses capacity.

8.4 Rank of a Matrix

📌 Rank: The number of linearly independent rows (= number of linearly independent columns) of
a matrix. rank(A) ≤ min(m, n) for an m×n matrix.
• Full rank (square): rank = n → matrix is invertible.
• Rank-deficient: rank < min(m,n) → system Ax=b may have no solution or infinite solutions.
• In DL: Low-rank matrices appear in weight compression (LoRA — Low-Rank Adaptation of
Large Language Models — decomposes weight updates into low-rank matrices, reducing
parameters).
💡 EXAM TIP: Linear dependence/span: Define span, linear independence (formal definition:
α₁v₁+...=0 implies all αᵢ=0), rank. Give examples of dependent vs independent vectors.
Connect to PCA and feature redundancy.

9. Norms
📌 Norm: A function that maps a vector to a non-negative scalar — measuring the 'size' or 'length'
of a vector. Denoted ||x||. Must satisfy: non-negativity, positive definiteness, triangle inequality,
homogeneity.
Norms are used throughout deep learning: as regularisation penalties, as distance measures, in
gradient clipping, and in loss functions.

9.1 Lp Norm (General Definition)

||x||ₚ = (Σᵢ |xᵢ|ᵖ)^(1/p) for p ≥ 1

9.2 L1 Norm (Manhattan Norm)

||x||₁ = Σᵢ |xᵢ| = |x₁| + |x₂| + ... + |xₙ|
• Sum of absolute values of all components.
• Grows linearly — not disproportionately penalised for large components.
• Deep learning use: L1 regularisation (Lasso). Adds λ||w||₁ to the loss. Encourages
SPARSE weights (many exactly zero) — automatic feature selection.
• Robust to outliers (outliers contribute linearly, not quadratically).
✎ Example: x = [3, -4, 0, 2] → ||x||₁ = |3| + |-4| + |0| + |2| = 3+4+0+2 = 9

Page 8 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

9.3 L2 Norm (Euclidean Norm)

||x||₂ = √(Σᵢ xᵢ²) = √(x₁² + x₂² + ... + xₙ²)
• The standard geometric length (Euclidean distance from origin).
• Squared L2 norm: ||x||₂² = xᵀx = Σᵢ xᵢ². Often preferred computationally (avoids square root,
simpler gradient).
• Deep learning use: L2 regularisation (Ridge / Weight Decay). Adds λ||w||₂² to loss.
Penalises large weights — shrinks them toward zero but rarely to exactly zero. Most
common regulariser.
• Gradient of ||x||₂² = 2x (simple, efficient). Gradient of ||x||₂ = x/||x||₂.
✎ Example: x = [3, -4] → ||x||₂ = √(9+16) = √25 = 5 (Pythagorean theorem! 3-4-5 right triangle)

9.4 L∞ Norm (Max Norm / Chebyshev Norm)

||x||∞ = max(|x₁|, |x₂|, ..., |xₙ|)
• The maximum absolute value of any component.
• Use: Gradient clipping by max norm — clip so largest gradient component ≤ threshold.
Prevents exploding gradients in RNNs.
✎ Example: x = [3, -7, 2, 5] → ||x||∞ = max(3,7,2,5) = 7

9.5 Frobenius Norm (Matrix Norm)

||A||_F = √(ΣᵢΣⱼ Aᵢⱼ²) (square root of sum of squares of all elements)
• Extension of L2 norm to matrices.
• Used in: Matrix regularisation, nuclear norm approximations, weight initialisation analysis.
✎ Example: A = [[1,2],[3,4]] → ||A||_F = √(1+4+9+16) = √30 ≈ 5.48

9.6 Norms in Deep Learning Summary

Norm Formula Deep Learning Use

L1 ||x||₁ Σ|xᵢ| Lasso regularisation —
sparse weights, feature
selection
L2 ||x||₂ √(Σxᵢ²) Ridge/weight decay —
smooth weights, most
common
L2² ||x||₂² Σxᵢ² Computationally preferred (no
sqrt), backprop friendly
L∞ ||x||∞ max|xᵢ| Gradient clipping — prevent
exploding gradients
Frobenius ||A||_F √(ΣΣAᵢⱼ²) Matrix regularisation, weight
analysis

💡 EXAM TIP: Norms EXAM: Know L1/L2/L∞ formulas and their deep learning uses.
L1→sparsity (feature selection), L2→weight decay (most common), L∞→gradient clipping.
Calculate all three for x=[3,-4,0,2].

Page 9 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

10. Special Matrices and Vectors

10.1 Diagonal Matrix
📌 Diagonal Matrix: A square matrix where all off-diagonal entries are zero. Aᵢ ⱼ = 0 for all i≠j.
Denoted diag(v) where v is the vector of diagonal values.
diag(v) = [[v₁,0,...,0],[0,v₂,...,0],...,[0,0,...,vₙ]]
• Matrix-vector product is efficient: diag(v)x = v ⊙ x (element-wise multiply). Only n
multiplications vs n² for full matrix.
• Inverse: diag(v)⁻¹ = diag(1/v₁, 1/v₂,..., 1/vₙ) — just invert each diagonal element.
• Deep learning use: Batch normalisation scale parameters (γ), covariance matrices of
independent features.
✎ Example: diag([2,3,4]) × [1,1,1]ᵀ = [2,3,4]ᵀ. Much faster than full matrix multiply.

10.2 Symmetric Matrix

📌 Symmetric Matrix: A square matrix equal to its own transpose: A = Aᵀ. Element-wise: Aᵢⱼ = A ⱼᵢ.
• All eigenvalues are real (not complex). Eigenvectors are orthogonal.
• Deep learning use: Covariance matrices Σ = (1/N)XᵀX are always symmetric positive semi-
definite. Gram matrices in style transfer (neural style). Hessian matrix of loss (second-order
optimization).
✎ Example: [[4,2,1],[2,5,3],[1,3,6]] is symmetric (Aᵢⱼ = Aⱼᵢ for all i,j).

10.3 Orthogonal Matrix

📌 Orthogonal Matrix: A square matrix whose rows are mutually orthonormal AND columns are
mutually orthonormal. Q is orthogonal iff QᵀQ = QQᵀ = I, i.e., Q ⁻¹ = Qᵀ.
• Rows and columns form an orthonormal basis.
• Key property: ||Qx||₂ = ||x||₂ — orthogonal transformations PRESERVE vector lengths
(isometry). No stretching, only rotation/reflection.
• Deep learning use: Weight initialisation (orthogonal initialisation). QR decomposition for
stable Gram-Schmidt. Unitary matrices in complex number NNs.
• Computational advantage: Q⁻¹ = Qᵀ — inverting is just transposing! O(n²) vs O(n³) for
general inverse.
✎ Example: Rotation matrix R = [[cos θ, -sin θ],[sin θ, cos θ]] is orthogonal: RᵀR = I. Rotates vectors
without changing length.

10.4 Positive Definite and Positive Semi-Definite Matrices

📌 Positive Definite (PD): Matrix A where xᵀAx > 0 for all non-zero x. All eigenvalues are strictly
positive.
📌 Positive Semi-Definite (PSD): Matrix A where xᵀAx ≥ 0 for all x. All eigenvalues are non-
negative.
• Covariance matrices are always PSD. If features are not redundant, they are PD.
• Loss Hessian being PD at a critical point confirms it is a local MINIMUM (not saddle or
maximum).
• Gram matrices K = XᵀX are always PSD (kernel matrices in SVMs, attention scores).

10.5 Unit Vector

📌 Unit Vector: A vector with L2 norm equal to 1: ||x||₂ = 1. Also called a normalised vector.
x̂ = x / ||x||₂ (normalise x to unit length)
• Normalisation: Divide any non-zero vector by its L2 norm to get a unit vector.

Page 10 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

• Deep learning use: Layer normalisation divides activations by their norm. L2 normalisation
of embeddings before cosine similarity. Normalised weight vectors in cosine classifiers.
✎ Example: x = [3,-4]ᵀ → ||x||₂=5 → x̂ = [3/5,-4/5]ᵀ = [0.6,-0.8]ᵀ. Check: ||x̂ ||₂ = √(0.36+0.64) = 1 ✓

10.6 Orthogonal Vectors (Orthonormal Basis)

📌 Orthogonal Vectors: Two vectors x and y are orthogonal if xᵀy = 0 (dot product = zero, 90°
angle).
📌 Orthonormal: Vectors that are both mutually orthogonal AND all unit vectors: xᵢᵀx ⱼ = δᵢ ⱼ
(Kronecker delta).
• Standard basis {e₁,e₂,...,eₙ} is orthonormal. e₁=[1,0,...,0], e₂=[0,1,...,0], etc.
• Transformer multi-head attention: Heads may learn orthogonal attention patterns — each
head captures different relationship types.
💡 EXAM TIP: Special matrices: Know diag (efficient multiplication), symmetric (AᵀA form,
eigenvalues real), orthogonal (Q⁻¹=Qᵀ, preserves length), PD/PSD (Hessian, covariance). Unit
vector normalisation formula.

11. Eigendecomposition
Eigendecomposition is one of the most powerful tools in linear algebra and deep learning — it
reveals the fundamental structure of a matrix by decomposing it into its 'natural' transformation
components.

11.1 Eigenvalues and Eigenvectors

📌 Eigenvector: A non-zero vector v such that multiplying by matrix A only scales v (does not
rotate it): Av = λv.
📌 Eigenvalue: The scalar λ corresponding to eigenvector v. It is the scaling factor: λ > 1
stretches, 0 < λ < 1 shrinks, λ < 0 flips direction, λ = 0 collapses to zero.
Av = λv (eigenvalue equation)
Interpretation: Eigenvectors are the 'axes of the matrix's action'. When A transforms space, most
vectors change direction — but eigenvectors only change length (by factor λ).
• A matrix has up to n eigenvalues (with possible repeats) for n×n matrix.
• Eigenvectors are not unique (any scalar multiple is also an eigenvector).
• Convention: Use unit eigenvectors (||v||₂ = 1).

11.2 Finding Eigenvalues

Rearranging Av = λv: Av - λv = 0 → (A - λI)v = 0. For non-trivial solution: det(A - λI) = 0.
Characteristic equation: det(A - λI) = 0
• Solve the characteristic polynomial for λ. For n×n matrix: degree-n polynomial → n roots.
✎ Example: A = [[3,1],[0,2]]. det([[3-λ,1],[0,2-λ]]) = (3-λ)(2-λ) - 0 = 0 → λ₁=3, λ₂=2. Eigenvalues: 3
and 2.

11.3 Finding Eigenvectors

For each eigenvalue λᵢ, solve (A - λᵢI)v = 0 for v:
✎ Example: λ₁=3: (A-3I)v=0 → [[0,1],[0,-1]]v=0 → v₁=[1,0]ᵀ. λ₂=2: (A-2I)v=0 → [[1,1],[0,0]]v=0 →
v₂=[1,-1]ᵀ/√2.

Page 11 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

11.4 Eigendecomposition (Spectral Decomposition)

📌 Eigendecomposition: Factoring a matrix into eigenvalues and eigenvectors: A = VΛV⁻¹.
A = VΛV⁻¹
• V: Matrix whose columns are eigenvectors of A (V = [v₁|v₂|...|vₙ]).
• Λ (Lambda): Diagonal matrix of eigenvalues: Λ = diag(λ₁, λ₂,..., λₙ).
• V⁻¹: Inverse of eigenvector matrix.
• For symmetric matrices: A = QΛQᵀ (Q is orthogonal since eigenvectors are orthonormal).
• Condition: Eigendecomposition exists only for diagonalisable matrices. All n×n matrices
with n distinct eigenvalues are diagonalisable.

11.5 Applications of Eigendecomposition in Deep Learning

1. Principal Component Analysis (PCA):
• Compute covariance matrix C = (1/N)XᵀX.
• Eigendecompose C = QΛQᵀ.
• Eigenvectors (columns of Q) = principal components (directions of maximum variance).
• Eigenvalues (diagonal of Λ) = variance explained by each component.
• Project data onto top-k eigenvectors to reduce dimension while preserving maximum
information.
• Use in DL: Preprocessing high-dimensional data, visualising embeddings (PCA then t-
SNE).
2. Understanding Gradient Descent Convergence:
• The Hessian H (second-order derivative of loss) has eigenvalues determining convergence
rate.
• Largest eigenvalue λ_max determines maximum stable learning rate: α < 2/λ_max.
• Ratio λ_max/λ_min = condition number — high condition number makes gradient descent
slow (ill-conditioned problem).
3. Spectral Graph Theory:
• Graph Convolutional Networks (GCNs) use eigendecomposition of the graph Laplacian L.
• Spectral convolutions: Filter signals in the eigenvalue (frequency) domain.
4. Matrix Powers:
Aᵏ = VΛᵏV⁻¹ where Λᵏ = diag(λ₁ᵏ,...,λₙᵏ)
• Computing Aᵏ directly requires k matrix multiplications. Via eigendecomposition: just raise
diagonal elements to kth power — O(n) vs O(n³k).
• Stability: If all |λᵢ| < 1, matrix power → 0 (contraction). If any |λᵢ| > 1, power grows
(expansion). Critical for RNN stability analysis.
5. Singular Value Decomposition (SVD) — Generalisation:
📌 SVD: Any matrix A (m×n, not necessarily square) can be decomposed: A = UΣVᵀ where U ∈
ℝᵐˣᵐ, Σ ∈ ℝᵐˣⁿ (diagonal, non-negative), V ∈ ℝⁿˣⁿ are all orthogonal/unitary.
• Singular values (diagonal of Σ) = square roots of eigenvalues of AᵀA.
• Deep learning use: Weight matrix analysis, low-rank approximation, pseudoinverse (for
overdetermined/underdetermined systems), LoRA (Low-Rank Adaptation).
• SVD of weight matrices reveals the 'effective rank' of a neural network layer.
💡 EXAM TIP: Eigendecomposition EXAM: Define eigenvalue/eigenvector (Av=λv),
characteristic equation (det(A-λI)=0), decomposition A=VΛV ⁻¹. Applications: PCA (covariance
eigendecomp), Hessian condition number (convergence), RNN stability (|λ|<1 vs >1). Know
SVD = generalised eigendecomposition.

Page 12 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

Page 13 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

SECTION B — NUMERICAL COMPUTATION

12. Overflow and Underflow

Computers represent real numbers using finite-precision floating point (IEEE 754 standard). This
creates fundamental limitations — not all real numbers can be exactly represented. Two critical
failure modes are underflow and overflow.

12.1 Floating Point Representation

IEEE 754 Float32 (standard in DL):
• 1 sign bit + 8 exponent bits + 23 mantissa bits = 32 bits total.
• Range: approximately ±3.4 × 10³⁸ (overflow limit) to ±1.2 × 10⁻³⁸ (underflow limit).
• Precision: ~7 decimal significant digits. Numbers closer than 10⁻⁷ look the same.
• Float64 (double precision): 64 bits. Range: ±1.8×10³⁰⁸. Precision: ~15 digits. Slower on
GPUs.
• Float16 (half precision): 16 bits. Range: ±65,504. Precision: ~3 digits. Very fast on modern
GPUs (Tensor Cores). Risk of overflow!

12.2 Underflow
📌 Underflow: When a computation produces a number so close to zero that it rounds to 0.0 in
floating point, losing all precision. Typically when |x| < ~10⁻³⁸ for Float32.
• Problem: Dividing by an underflowed zero produces NaN (Not a Number) or Inf —
catastrophic for training.
• Common scenario in DL: Computing softmax probabilities. If logits are large negatives,
exp(logit) → 0 (underflow), then log(0) = -∞ (NaN in loss).
Softmax Underflow Example:
softmax(x)ᵢ = exp(xᵢ) / Σⱼ exp(xⱼ)
• If x = [-1000, -1001, -1002]: exp(-1000) = 0.0 in Float32 → denominator = 0 → NaN!
Numerically Stable Softmax (solution):
softmax(x)ᵢ = exp(xᵢ - max(x)) / Σⱼ exp(xⱼ - max(x))
• Subtract the maximum logit from all logits before exponentiating. Max term becomes
exp(0)=1. All others ≤ 1. No overflow or underflow.
• Mathematical equivalence: The max subtraction cancels in numerator and denominator.
✎ Example: x=[-1000,-1001,-1002]. max=−1000. Shifted: [0,-1,−2]. exp: [1, 0.368, 0.135].
sum=1.503. Softmax: [0.665, 0.245, 0.090]. Safe!

12.3 Overflow
📌 Overflow: When a computation produces a number larger than the maximum representable
value, resulting in Inf (infinity). Typically when |x| > ~3.4 × 10³⁸ for Float32.
• Common scenario: exp(x) overflows for x > ~88.7 in Float32. Very common with large logits
in softmax.
• Once a value becomes Inf: Inf × 0 = NaN, Inf - Inf = NaN. The NaN 'infects' all subsequent
operations — training crashes.
• Gradient explosion in deep networks: Gradients multiply across many layers. If weights > 1,
gradients grow exponentially → overflow.
Solutions to Overflow:

Page 14 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

• Numerical stabilisation: Use numerically stable implementations (stable softmax above).

• Gradient clipping: Clip gradients to a max norm before updating weights. Prevents gradient
explosion.
• Mixed precision training: Use Float32 for critical operations, Float16 for speed. 'Loss
scaling': multiply loss by large constant to keep Float16 gradients from underflowing.
• LogSumExp trick: Compute log(Σexp(xᵢ)) stably as max(x) + log(Σexp(xᵢ - max(x))).
log(Σⱼ exp(xⱼ)) = max(x) + log(Σⱼ exp(xⱼ - max(x))) [numerically
stable]

✎ Example: Log-sum-exp for x=[1000,1001,1002]: max=1002. log(exp(-2)+exp(-1)+exp(0))+1002 =

log(1.503)+1002 ≈ 1002.41. Safe!

12.4 Log-Space Computation

• Work in log-space for products of small probabilities: log P(a,b,c) = log P(a) + log P(b|a) +
log P(c|a,b).
• Instead of multiplying many small probabilities (→ underflow), add their logs.
• Cross-entropy loss: -log P(y|x) naturally works in log-space. log(softmax) computed stably
with log-sum-exp.
• PyTorch: [Link].log_softmax + NLLLoss = stable. [Link] does
this internally.
💡 EXAM TIP: Overflow/underflow EXAM: Define both, give the softmax example (exp(-
1000)=0→NaN), show numerically stable softmax formula (subtract max), and solutions:
gradient clipping, mixed precision, log-space computation.

13. Poor Conditioning

Conditioning describes how sensitive the output of a function is to small changes in input. Poorly
conditioned problems amplify small numerical errors into large output errors — making
computations unreliable.

13.1 Condition Number

📌 Condition Number: For a matrix A, the condition number κ(A) = ||A|| · ||A ⁻¹|| = λ_max/λ_min
(for symmetric PD matrices). Measures how much A amplifies errors.
κ(A) = λ_max / λ_min (ratio of largest to smallest eigenvalue)
• κ ≈ 1: Well-conditioned. Small input errors → small output errors. Reliable computation.
• κ >> 1 (e.g., 10⁶): Poorly conditioned (ill-conditioned). Small input errors → large output
errors. Unreliable.
• κ = ∞: Singular matrix. A⁻¹ doesn't exist.

13.2 Why Poor Conditioning Matters in Deep Learning

Gradient Descent Convergence:
• The loss surface curvature along different directions is given by Hessian eigenvalues.
• High condition number of Hessian = very elongated loss surface 'valley'.
• Gradient descent zig-zags along the narrow valley. Extremely slow convergence.
• Learning rate must be set for the sharpest direction (λ_max) → tiny steps in flat directions.
• Fix: Adaptive optimisers (Adam, RMSprop) normalise gradients by direction-specific
curvature → effectively reduces condition number.
Weight Initialisation:

Page 15 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

• If weight matrices are poorly conditioned at init, signals and gradients amplify or vanish
layer by layer.
• Well-conditioned init (Xavier/Glorot, He initialisation) ensures eigenvalues are near 1 at init.
• Batch normalisation: Normalises activations — reduces the effective condition number of
each layer.
Linear System Solving:
• Solving Ax = b for poorly conditioned A: rounding errors in A or b get amplified by κ(A).
• If κ(A) = 10⁶ and Float32 has 7 significant digits: solution accurate to only 7-6 = 1 digit!

13.3 Catastrophic Cancellation

📌 Catastrophic Cancellation: Loss of significant digits when subtracting two nearly equal
floating-point numbers.
✎ Example: a = 1.23456789, b = 1.23456788 (differ by 10⁻⁸). Float32 represents both as same
value → a-b = 0.0 (complete loss of information). True result: 10⁻⁸.
• Common in: Variance computation (Σx²/N - (Σx/N)²), softmax (without stabilisation),
computing small differences.
• Fix: Numerically stable algorithms that avoid cancellation (Welford's online algorithm for
variance, stable softmax).
💡 EXAM TIP: Poor conditioning: κ = λ_max/λ_min. High κ → slow gradient descent zig-zag.
Solution: Adam (adaptive LR per parameter), BatchNorm (reduces effective condition), Xavier
init (κ≈1 at start). Define catastrophic cancellation with example.

14. Gradient-Based Optimization

Optimization is the core of machine learning training. We want to find parameter values θ that
minimise a loss function L(θ). Gradient-based methods use the gradient (direction of steepest
increase) to guide the search.

14.1 The Optimization Problem

θ* = argmin_θ L(θ) (find θ that minimises loss L)
• L(θ): Scalar loss function measuring model error on training data.
• θ: All model parameters (weights + biases) — could be millions.
• θ*: Optimal parameters (global or local minimum).
• In practice: L is a highly non-convex function of θ for deep networks. Global optimum
unreachable — find 'good enough' local minimum or saddle point.

14.2 Gradient and Directional Derivative

📌 Gradient: Vector of all partial derivatives of a scalar function f w.r.t. all inputs: ∇f = [∂f/∂θ₁,
∂f/∂θ₂, ..., ∂f/∂θₙ]ᵀ. Points in direction of steepest INCREASE.
∇_θ L = [∂L/∂θ₁, ∂L/∂θ₂, ..., ∂L/∂θₙ]ᵀ
• The gradient is a vector with the same dimension as θ.
• Magnitude ||∇L|| indicates how steep the surface is.
• Direction: The gradient points toward the direction of maximum increase. Negative gradient
→ maximum decrease.
📌 Directional Derivative: Rate of change of f in direction unit vector u: D_u f = uᵀ ∇f = || ∇f||
cos(θ). Maximised when u = ∇f/||∇f||.

Page 16 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

14.3 Gradient Descent

📌 Gradient Descent: Iterative optimization algorithm: Start at random θ, repeatedly move in the
direction of the negative gradient by a small step size (learning rate α).
θ ← θ - α ∇_θ L(θ) (gradient descent update rule)
• α (learning rate): Controls step size. Too large → overshoot minimum, diverge. Too small
→ slow convergence.
• One step: Evaluate loss and gradient at current θ → move to new θ → repeat until
convergence.
• Convergence criteria: ||∇L||₂ < ε, or loss change < ε, or maximum iterations reached.

14.4 Variants of Gradient Descent

Batch Gradient Descent (BGD):
L = (1/N) Σᵢ₌₁ᴺ l(f(xᵢ;θ), yᵢ) [average over ALL N examples]
• Computes exact gradient over entire dataset before updating. Very slow for large N.
• Stable convergence. Guaranteed to descend. Memory intensive — all data must fit in RAM.
Stochastic Gradient Descent (SGD):
L ≈ l(f(xᵢ;θ), yᵢ) [one random example at a time]
• Updates after each single example. N updates per epoch vs 1 for BGD.
• Noisy gradient estimate — high variance. May help escape local minima.
• Very fast per step. Cannot exploit GPU parallelism well (batch size=1).
Mini-Batch Gradient Descent (standard in DL):
L ≈ (1/B) Σᵢ₌₁ᴮ l(f(xᵢ;θ), yᵢ) [B examples, typically B=32,64,128,256]
• Gold standard in deep learning. Balances: noise reduction (vs SGD) and speed (vs BGD).
• Exploits GPU parallelism: process B examples simultaneously.
• Typical batch sizes: 32 (small models), 256 (standard), 2048+ (large language models).

14.5 Advanced Optimisers

Momentum:
v ← βv - α∇L (accumulate velocity)
θ ← θ + v
• Accumulates a velocity vector v in directions of persistent gradient. β = 0.9 typical.
• Speeds through flat regions. Dampens oscillations in narrow valleys.
RMSprop:
v ← ρv + (1-ρ)(∇L)² (exponential moving average of squared gradients)
θ ← θ - (α/√v + ε) ∇L
• Adapts learning rate per parameter. Large gradient history → small effective LR. Small
gradient history → large effective LR.
• Good for non-stationary objectives (RNNs, RL).
Adam (Adaptive Moment Estimation — most widely used):
m ← β₁m + (1-β₁)∇L [1st moment — gradient mean]
v ← β₂v + (1-β₂)(∇L)² [2nd moment — gradient variance]
m̂ = m/(1-β₁ᵗ), v̂ = v/(1-β₂ᵗ) [bias correction]
θ ← θ - (α/√v̂ + ε) × m̂
• Combines momentum (m) and RMSprop (v). Default: β₁=0.9, β₂=0.999, ε=10⁻⁸, α=10 ⁻³.

Page 17 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

• Works well with sparse gradients. Adaptive LR per parameter. De facto standard for most
DL tasks.
• AdamW: Adam + weight decay correctly applied to parameters only (not to
bias/normalisation). Better generalisation.

14.6 Loss Surface Geometry

Global Minimum:
• Point where L achieves its lowest value over all θ. ∇L=0, all eigenvalues of Hessian > 0.
• In deep networks: Rarely found. May not even exist uniquely.
Local Minimum:
• Point lower than all nearby points but not globally lowest. ∇L=0, all Hessian eigenvalues >
0.
• In practice: Local minima of deep networks often have similar loss to global minimum — not
a problem!
Saddle Points:
• ∇L=0 but Hessian has both positive AND negative eigenvalues. A minimum in some
directions, maximum in others.
• Much more common than local minima in high-dimensional spaces. Gradient descent can
get stuck.
• Fix: Momentum and noise (SGD) help escape saddle points.
Plateaus:
• Regions where ||∇L|| ≈ 0 but not a minimum. Gradient descent becomes very slow.
• Fix: Adaptive learning rates (Adam) handle plateaus better than fixed-LR SGD.
Learning Rate Schedules:
• Cosine annealing: α follows cosine curve from α_max to α_min over training.
• Step decay: Multiply α by factor γ < 1 every k epochs.
• Warmup: Start with tiny α, increase to target α over first few thousand steps. Prevents early
instability. Standard in Transformer training.
• Cyclical LR: Oscillate between min and max. Can escape local minima by periodically
increasing LR.
∇EXAM TIP: Gradient descent is THE most important topic. Know: update rule θ←θ-α∇L, BGD
vs SGD vs mini-batch (trade-offs), Adam formula (m,v,bias correction), saddle points vs local
minima. Learning rate schedules: warmup, cosine, step decay.

15. Constraint Optimization

In unconstrained optimization, we minimise f(x) over all of ℝⁿ. In constrained optimization, we
restrict x to a feasible set defined by equality or inequality constraints.

15.1 Types of Constrained Problems

General Form:
min f(x) subject to: gᵢ(x) ≤ 0 (inequality), hⱼ(x) = 0 (equality)
• Equality constraints: hⱼ(x) = 0. Solution must lie exactly on the constraint surface.
• Inequality constraints: gᵢ(x) ≤ 0. Solution must lie in the feasible region.
• Feasible set: All x satisfying all constraints simultaneously.

Page 18 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

✎ Example: Minimise loss L(w) subject to ||w||₂² ≤ C (L2 constraint). Solution must lie inside an n-
sphere of radius √C.

15.2 Lagrangian and Lagrange Multipliers (Equality Constraints)

📌 Lagrangian: Function combining objective and equality constraints: L(x,λ) = f(x) + Σⱼ λⱼhⱼ(x). The
scalars λⱼ are Lagrange multipliers.
L(x,λ) = f(x) + Σⱼ λⱼhⱼ(x)
Necessary condition for optimum: ∇ₓL = 0 AND hⱼ(x) = 0 for all j. This converts a constrained
problem into an unconstrained system of equations.
• Geometric intuition: At the constrained optimum, ∇f must be parallel to ∇h (cannot reduce f
further without violating constraint). λ is the scaling factor.
✎ Example: Minimise f(x,y) = x²+y² subject to x+y=1. Lagrangian: L = x²+y² + λ(x+y-1). ∂L/∂x =
2x+λ=0, ∂L/∂y = 2y+λ=0, x+y=1 → x=y=0.5, λ=-1.

15.3 KKT Conditions (Inequality Constraints)

📌 KKT Conditions (Karush-Kuhn-Tucker): Necessary conditions for a local optimum of a
constrained problem with inequality constraints. Generalisation of Lagrange multipliers.
For min f(x) subject to gᵢ(x) ≤ 0, the Generalised Lagrangian is:
L(x,μ) = f(x) + Σᵢ μᵢgᵢ(x) where μᵢ ≥ 0
KKT Conditions (all must hold at optimum x*):
• Stationarity: ∇ₓL = ∇f(x*) + Σᵢ μᵢ∇gᵢ(x*) = 0
• Primal feasibility: gᵢ(x*) ≤ 0 for all i (must satisfy constraints).
• Dual feasibility: μᵢ ≥ 0 for all i (multipliers non-negative for inequality constraints).
• Complementary slackness: μᵢgᵢ(x*) = 0 for all i. Either constraint is active (gᵢ=0) or multiplier
is zero (μᵢ=0).

15.4 Projection-Based Methods

📌 Projected Gradient Descent: Gradient descent with a projection step: take a gradient step,
then project back into the feasible set.
x ← Proj_C(x - α∇f(x))
• Proj_C: Projection onto feasible set C. Finds the nearest point in C to the given point.
• L2 ball constraint: Project by normalising: if ||x||₂ > C, set x ← C × x/||x||₂.
• Box constraint [a,b]: Project by clipping: x ← clip(x, a, b).
✎ Example: Weight clipping in Wasserstein GAN: After each gradient step, clip all weights to [-c,c].
This enforces a Lipschitz constraint on the discriminator.

15.5 Constrained Optimization in Deep Learning

Regularisation as Constraint:
• L2 regularisation: Minimise L(θ) subject to ||θ||₂² ≤ C. Lagrangian: L + λ||θ||₂². The
constraint becomes the regularisation term.
• L1 regularisation: Minimise L(θ) subject to ||θ||₁ ≤ C. Lasso constraint.
• Via Lagrangian duality: Constrained problem ↔ penalised unconstrained problem. Trading
C for λ.
Max-Norm Constraint:
• Constrain weight vectors of each neuron to have ||w||₂ ≤ c. Applied after gradient update.
• More stable than L2 weight decay — doesn't shrink weights but caps their maximum size.
Entropy Maximisation (RL and Information Theory):

Page 19 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

• Maximise expected reward R(θ) subject to H(π) ≥ H_min (minimum entropy constraint on
policy π).
• Prevents policy from collapsing to deterministic. Used in SAC (Soft Actor-Critic) RL
algorithm.
Fairness Constraints (ML Ethics):
• Minimise prediction error subject to P(ŷ=1|group=A) = P(ŷ=1|group=B) (demographic
parity).
• Adds equality constraints ensuring model treats demographic groups equally.
Neural Architecture Search (NAS):
• Minimise validation loss subject to FLOPs ≤ budget, Parameters ≤ memory limit, Latency ≤
inference time.
• Multi-objective constrained optimization — find architectures on the Pareto front.
💡 EXAM TIP: Constrained optimization: Lagrangian formula L=f+Σλh, KKT 4 conditions
(stationarity, primal feasibility, dual feasibility, complementary slackness), projected gradient
descent. Connect to DL: L2 regularisation = L2 constraint, max-norm constraint, WGAN weight
clipping.

MODULE 1 — COMPLETE QUICK REVISION SUMMARY

LINEAR ALGEBRA — Key Formulas to Memorise:
• Scalar: single number. Vector: 1D array. Matrix: 2D array. Tensor: nD array.
• Dot product: x·y = Σxᵢyᵢ (scalar result). Matrix product: Cᵢⱼ = ΣAᵢₖBₖⱼ. Shapes:
(m×n)×(n×p)=(m×p).
• Identity: AI = IA = A. Inverse: AA⁻¹ = I → x = A ⁻¹b solves Ax=b.
• Linear independence: α₁v₁+...+αₙvₙ=0 implies all αᵢ=0. Rank = # independent rows/cols.
• L1: Σ|xᵢ| L2: √Σxᵢ² L∞: max|xᵢ| Frobenius: √ΣΣAᵢⱼ²
• Diagonal: diag(v)x = v⊙x. Symmetric: A=Aᵀ. Orthogonal: QᵀQ=I → Q ⁻¹=Qᵀ, preserves
length.
• Eigenvalue equation: Av=λv. Find λ: det(A-λI)=0. Decomposition: A=VΛV⁻¹.
NUMERICAL COMPUTATION — Key Concepts:
• Underflow: number → 0 (too small). Overflow: number → Inf (too large). Both cause NaN.
• Stable softmax: subtract max(x) before exp. LogSumExp trick for log-space.
• Condition number κ = λ_max/λ_min. High κ → slow GD convergence. Adam fixes this
adaptively.
• Gradient descent: θ←θ-α∇L. BGD (all data), SGD (1 sample), mini-batch (B samples).
• Adam: m=β₁m+(1-β₁)∇L, v=β₂v+(1-β₂)(∇L)², θ←θ-(α/√v̂ )m̂ . β₁=0.9, β₂=0.999.
• Loss surface: global min (rare), local min (common, usually fine), saddle points (∇L=0,
mixed Hessian eigenvalues), plateaus.
• Lagrangian: L=f+Σλh (equality). KKT: +μᵢgᵢ, μᵢ≥0, μᵢgᵢ=0 (inequality).
• L2 regularisation = L2 constraint via Lagrangian duality. WGAN = projected GD (weight
clipping).

Topic Likely Exam Question Key Answer Points

Scalars/Vectors/ 5 marks — Explain with DL Definitions + notation + 2 DL
Matrices/Tensors examples examples each + tensor ranks
table

Page 20 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 1: Applied Math & ML Basics + Numerical Computation

Topic Likely Exam Question Key Answer Points

Matrix Multiplication 5 marks — Explain matrix Formula Cᵢⱼ=ΣAᵢₖBₖⱼ, shape
product with example rule, forward pass Y=XW+b
Norms 5 marks — Define L1, L2, L∞ Formulas + L1→sparsity,
with uses L2→weight decay,
L∞→gradient clipping
Eigendecomposition 5-10 marks — Explain Av=λv, det(A-λI)=0, A=VΛV⁻¹,
eigenvalues/vectors and uses PCA, Hessian condition number
Overflow/Underflow 5 marks — Explain with Definitions, softmax NaN
softmax example example, stable softmax
formula, log-space
Condition Number 5 marks — Poor conditioning in κ=λ_max/λ_min, zig-zag in
DL narrow valleys, Adam solution,
catastrophic cancellation
Gradient Descent 10 marks — Gradient descent Update rule, BGD/SGD/mini-
variants + Adam batch trade-offs, Adam formula,
loss surface geometry
Constrained Optimization 5 marks — Lagrangian + KKT Lagrangian formula, 4 KKT
conditions conditions, DL applications (L2
reg, WGAN)

— END OF DEEP LEARNING MODULE 1 NOTES —

15 Lectures · 15 Topics · Section A: 10 (Linear Algebra) + Section B: 5 (Numerical Computation)
Good luck with your Deep Learning examination! 🎓

Page 21 | SVKM's UPG College | Deep Learning Notes 2024-25