Deep Learning

Comprehensive Study Notes

Week 1 – Week 3 | Lectures & Supporting Materials

Week 1 – Introduction to AI, Machine Learning & Deep Learning

1.1 Key Definitions

• Artificial Intelligence (AI): The concept of creating smart intelligent machines that can imitate
human behaviour.
• Machine Learning (ML): A subset of AI that provides systems the ability to automatically learn
and improve from experience without being explicitly programmed.
• Deep Learning (DL): A subset of ML that uses vast volumes of data and complex algorithms
(deep neural networks with more than 2 layers) to train a model automatically.

Hierarchy: AI ⊃ Machine Learning ⊃ Deep Learning

1.2 Machine Learning vs Deep Learning

Feature Machine Learning Deep Learning

Data requirement Can train on smaller datasets Requires large amounts of data
Human intervention More human intervention to Learns on its own from
correct and learn environment and past mistakes
Training speed Shorter training time, lower Longer training time, higher
accuracy accuracy
Correlations Makes simple, linear Makes non-linear, complex
correlations correlations
Hardware Can train on a CPU Needs a specialised GPU

1.3 Types of Machine Learning

Supervised Learning
Uses labelled data to train models. The output is already known; the model maps inputs to their
respective outputs.
 • Example: Image classifier trained to identify cats.
• Algorithms: Linear Regression, Logistic Regression, SVM, KNN, Decision Tree, Random Forest,
Naive Bayes
Unsupervised Learning
Uses unlabelled data. The model discovers patterns and features by itself without a predefined target
variable.
• Example: Grouping vehicles into buses and trucks without labels.
• Algorithms: K-Means Clustering, Hierarchical Clustering, Principal Component Analysis (PCA)
Reinforcement Learning
Trains a machine to take suitable actions and maximise rewards. Uses an agent and environment;
there is no predefined target variable.
• Example: An agent learning to identify shapes through feedback.
• Algorithms: Q-Learning, Monte Carlo

1.4 Types of Deep Learning

• Supervised Deep Learning: ANN, CNN, RNN
• Unsupervised Deep Learning: Self-Organising Map, Boltzmann Machine, AutoEncoder

1.5 Deep Learning Applications

• Healthcare / Biology: Cancer detection, drug discovery
• Consumer Web & Mobile: Image classification, speech recognition
• Media & Entertainment: Video captioning, real-time translation
• Autonomous Cars: Lane tracking, vehicle detection
• Security & Public Safety: Face recognition, video surveillance

Week 1 – Artificial Neural Networks (ANN)

2.1 Biological vs Artificial Neuron

Inspired by the human brain (drawn by Santiago Ramon y Cajal). A biological neuron has: Dendrites
(receive signals), Soma/Cell body (processes signals), Axon (transmits output) and Synapse (junction
to next neuron). The artificial neuron mirrors this structure.

2.2 Perceptron (Single Neuron) Architecture

Each perceptron performs two steps:
1. Step 1 – Weighted Sum: Z = b + Σ(Wi × Xi)
2. Step 2 – Activation: o = g(Z)
Where: X = input vector, W = weight matrix, b = bias (constant), g = activation function.

2.3 Network Architecture (Layers)

• Input Layer: Receives and preprocesses data in various formats specified by the programmer.
• Hidden Layer(s): Situated between input and output. Performs all the math to uncover hidden
patterns.
• Output Layer: Transforms hidden-layer output into the final result.
 Shallow Network = 1 hidden layer. Deep Network = 2+ hidden layers.

2.4 Three Core Topics in ANNs

3. Forward Propagation
4. Backward Propagation
5. Activation Functions

2.5 Forward Propagation

Data moves left-to-right through the network. At each neuron:
• Compute weighted sum: Z = w₁·x₁ + w₂·x₂ + … + b
• Apply activation function: a = f(Z)
The output of one layer becomes the input for the next.
Sigmoid Activation Function
a = f(z) = 1 / (1 + e^-z)
Sigmoid squashes any value into the range (0, 1) — ideal for binary classification outputs.

Worked Example (Forward Pass)

Given: x₁ = 0.1, w₁ = 0.15, b₁ = 0.40
z₁ = 0.15 × 0.1 + 0.40 = 0.415
a₁ = 1 / (1 + e^-0.415) = 0.6023
Next neuron: a₁ = 0.6023, w₂ = 0.45, b₂ = 0.65
z₂ = 0.45 × 0.6023 + 0.65 = 0.9210
a₂ = 1 / (1 + e^-0.921) = 0.7153

Week 3 – Linear Regression

3.1 Definition
Linear regression predicts the relationship between two variables by assuming a linear connection
between the independent (X) and dependent (Y) variables. It finds the optimal line that minimises the
sum of squared differences between predicted and actual values.

3.2 Types of Linear Regression

• Simple Linear Regression: One independent variable, one dependent variable.
• Multiple Linear Regression: Two or more independent variables.
• Logistic Regression: Used for classification (outputs a probability).

3.3 Simple Linear Regression

The model estimates the slope and intercept of the best-fit line:
Yᵢ = β₀ + β₁ Xᵢ
• β₀ (Intercept): Predicted value of Y when X = 0.
• β₁ (Slope): Change in Y for each unit increase in X.
 • Xᵢ: Independent variable (predictor).
• Yᵢ: Dependent variable (output).

3.4 Residuals (Random Error)

The residual is the difference between the observed and predicted value:
εᵢ = y_predicted − yᵢ
Goal: minimise residuals — the best-fit line has the least total error.

3.5 Best-Fit Line & Cost Function

Mathematically, the best-fit line minimises the Residual Sum of Squares (RSS). The standard cost
function used is Mean Squared Error (MSE):
MSE = (1/N) Σ (yᵢ − (B1·xᵢ + B0))²
The cost function guides the algorithm to find optimal values of B0 and B1.

3.6 Formulas to Calculate Slope & Intercept

b₀ = [ (Σy)(Σx²) − (Σx)(Σxy) ] / [ n(Σx²) − (Σx)² ]
b₁ = [ n(Σxy) − (Σx)(Σy) ] / [ n(Σx²) − (Σx)² ]

3.7 Worked Example: Predicting Glucose Level from Age

Dataset: 6 subjects with known ages (X) and glucose levels (Y).

Subject Age (X) Glucose (Y) XY X²

1 43 99 4257 1849
2 21 65 1365 441
3 25 79 1975 625
4 42 75 3150 1764
5 57 87 4959 3249
6 59 81 … …

Using Σy=486, Σx=247, Σx²=11409, Σxy=20485, n=6:

b₀ = (486×11409 − 247×20485) / (6×11409 − 247²) = 4848979 / 7445 ≈
65.14
b₁ = (6×20485 − 247×486) / (6×11409 − 247²) = 2868 / 7445 ≈ 0.3853
Regression equation:
y' = 65.14 + 0.3853 x
Prediction for age 55:
y' = 65.14 + (0.3853 × 55) = 86.33
The estimated glucose level for a person aged 55 is 86.33 units.
CNN – Convolutional Neural Networks

CNNs are composed of convolutional layers and pooling layers specifically designed to automatically
learn hierarchical features from visual data (images, video).
• Convolutional layer: Applies filters to extract feature maps from the input.
• Pooling layer: Reduces spatial dimensions (downsampling) of the feature maps.
• Flatten layer: Converts 2D feature maps into a 1D vector.
• Fully Connected layer: Performs classification using the extracted features.
CNNs are particularly powerful for image classification, object detection, and video analysis tasks.

RNN – Recurrent Neural Networks

RNNs have connections that loop back on themselves, allowing them to maintain hidden states and
capture temporal dependencies. They process sequences one element at a time while maintaining
memory of past elements.
• Hidden state: Carries information from previous time steps.
• Applications: Natural language processing, speech recognition, time-series forecasting.
Unlike ANNs/CNNs, RNNs share parameters across time steps, making them suited for sequential data.

Quick Reference – Key Formulas

Concept Formula / Description

Simple Linear Regression Yᵢ = β₀ + β₁ Xᵢ
Mean Squared Error (MSE) MSE = (1/N) Σ (yᵢ − (B1·xᵢ + B0))²
Residual εᵢ = y_predicted − yᵢ
Weighted Sum (Perceptron) Z = b + Σ (Wᵢ × Xᵢ)
Sigmoid Activation a = 1 / (1 + e^⁻ᵣ)
Intercept (b₀) [ (Σy)(Σx²) − (Σx)(Σxy) ] / [ n(Σx²) − (Σx)² ]
Slope (b₁) [ n(Σxy) − (Σx)(Σy) ] / [ n(Σx²) − (Σx)² ]

End of Study Notes