DEEP LEARNING
Comprehensive Course Notes
For First-Year Data Science Students
Prepared by: Data Science Department
Academic Year: 2025 – 2026
Module 1: Foundations & Prerequisites
1.1 What is Artificial Intelligence, Machine Learning, and Deep
Learning?
Before diving into deep learning, it is important to understand where it fits in the larger
technology landscape. Three terms are often used together — Artificial Intelligence (AI),
Machine Learning (ML), and Deep Learning (DL) — but they are not the same thing.
Term Definition
Artificial Intelligence (AI) The broadest field. AI refers to any technique that enables
machines to mimic human intelligence — including rule-based
systems, expert systems, and learning-based approaches.
Machine Learning (ML) A subset of AI. Instead of being explicitly programmed with
rules, ML systems learn patterns from data and improve over
time.
Deep Learning (DL) A subset of ML. DL uses multi-layered neural networks
inspired by the human brain to automatically learn complex
representations from raw data.
Key Insight
Think of it as nested circles: AI is the outer circle, ML is inside AI, and Deep
Learning is inside ML. Deep Learning is a powerful tool within the broader AI
ecosystem.
1.2 The AI/ML/DL Hierarchy — Real-World Examples
Artificial Intelligence: A chess program that follows coded rules to play chess.
Machine Learning: A spam filter that learns which emails are spam by studying
thousands of examples.
Deep Learning: A system that recognises faces in photos, understands spoken
language, or generates realistic images — all from raw pixels or audio.
1.3 Why Deep Learning? — Key Breakthroughs
Deep learning has become dominant because of three converging forces:
• Big Data — the internet has produced vast amounts of labelled data (images,
text, video).
• Compute Power — Graphics Processing Units (GPUs) allow parallel computation
at scale.
• Algorithmic Advances — improved techniques like ReLU, Dropout, and the Adam
optimiser dramatically improved training.
Notable breakthroughs: ImageNet (2012), AlphaGo (2016), GPT-3 (2020), Stable
Diffusion (2022), GPT-4 (2023).
1.4 Essential Mathematics Refresher
Linear Algebra
• Scalar — a single number, e.g., 5.
• Vector — an ordered list of numbers, e.g., [1, 2, 3]. Represents a point or
direction in space.
• Matrix — a 2D grid of numbers. Used to represent datasets and transformations.
• Dot product — multiplying two vectors element-wise and summing the results.
Central to neural network computations.
Probability & Statistics
• Probability — measures the likelihood of an event (0 to 1).
• Mean & Variance — describe the centre and spread of data.
• Probability distributions — e.g., Normal (Gaussian), Bernoulli, Softmax output.
Calculus Intuition
• Derivative — measures how much a function changes when its input changes.
Used to find minima of the loss function.
• Gradient — the multi-dimensional equivalent of a derivative; a vector of partial
derivatives.
• Chain rule — allows us to compute derivatives of composed functions (the
backbone of backpropagation).
Module 2: The Building Block — The Neuron
2.1 Biological vs. Artificial Neuron
The artificial neuron is loosely inspired by neurons in the human brain. A biological
neuron receives signals through dendrites, processes them in the cell body, and sends
an output signal through the axon. An artificial neuron performs an analogous
mathematical operation.
Biology Artificial Neuron
Dendrites Input values (x₁, x₂, ..., xₙ)
Synapse strength Weights (w₁, w₂, ..., wₙ)
Cell body Weighted sum + bias: z = Σ(wᵢxᵢ) + b
Axon / firing Activation function: output = f(z)
2.2 The Perceptron
The perceptron (Rosenblatt, 1958) is the simplest form of artificial neuron. It takes
binary inputs, computes a weighted sum, and outputs either 0 or 1 based on a
threshold.
Formula: output = 1 if (w·x + b > 0) else 0
Limitation
A single perceptron can only learn linearly separable problems. It cannot solve XOR
— this motivated the development of multi-layer networks.
2.3 Inputs, Weights, and Bias
• Inputs (x) — the data fed into the neuron (e.g., pixel values, temperature
readings).
• Weights (w) — determine the importance of each input. Learned during training.
• Bias (b) — a learnable constant that shifts the activation function, giving the
model more flexibility.
2.4 Activation Functions
Activation functions introduce non-linearity, allowing neural networks to learn complex
patterns. Without them, stacking layers would still only produce a linear model.
Function Description & Use Case
Sigmoid Maps output to (0, 1). Used in output layer for binary classification.
Suffers from vanishing gradients.
Tanh Maps output to (−1, 1). Zero-centred, slightly better than Sigmoid
for hidden layers.
ReLU Rectified Linear Unit. f(x) = max(0, x). Default for hidden layers.
Fast and avoids vanishing gradients.
Leaky ReLU Variant of ReLU that allows small negative values to prevent 'dying
neurons'.
Softmax Converts a vector of numbers into a probability distribution. Used in
the output layer for multi-class classification.
Module 3: Neural Networks
3.1 From One Neuron to a Network
A neural network is simply a collection of neurons organised into layers. Each neuron in
one layer is connected to neurons in the next layer, forming a dense web of
connections. As data flows through the network, it is progressively transformed into
more abstract representations.
3.2 Network Layers
Layer Type Role
Input Layer Receives the raw input data. The number of neurons equals the
number of features in the dataset.
Hidden Layer(s) Intermediate layers that learn abstract representations. 'Deep'
networks have many hidden layers.
Output Layer Produces the final prediction. Number of neurons depends on the
task (1 for regression, N for N-class classification).
3.3 Forward Propagation
Forward propagation is the process of passing data from the input layer through the
hidden layers to the output layer. At each neuron, the weighted sum is calculated and
passed through an activation function.
Step-by-step: Input → Weighted sum (z = Wx + b) → Activation f(z) → Output to next
layer → Final prediction
3.4 Loss Functions
The loss function (also called cost function) measures how far the network's predictions
are from the true values. The goal of training is to minimise this loss.
• Mean Squared Error (MSE) — used for regression tasks. Penalises large errors
heavily.
• Binary Cross-Entropy — used for binary classification (two classes).
• Categorical Cross-Entropy — used for multi-class classification.
3.5 Backpropagation & Gradient Descent
Backpropagation is the algorithm used to train neural networks. It works in two phases:
1. Forward pass — compute predictions and calculate the loss.
2. Backward pass — compute the gradient of the loss with respect to every weight
using the chain rule.
3. Update weights — adjust weights in the direction that reduces the loss: w = w − η
× ∂L/∂w
Gradient Descent Intuition
Imagine you are blindfolded on a hilly landscape and want to reach the lowest point
(minimum loss). You feel the slope under your feet (gradient) and take a small step
downhill. Repeat until you stop descending — that is gradient descent.
3.6 Learning Rate
The learning rate (η) controls the size of the steps taken during gradient descent.
• Too large — overshoots the minimum; training becomes unstable.
• Too small — converges very slowly; may get stuck.
• Typical values — start with 0.001 or 0.01 and tune as needed.
Module 4: Training a Neural Network
4.1 The Training Loop
Training is the process of repeatedly presenting data to the network and adjusting its
weights to reduce the loss. Key terminology:
• Epoch — one complete pass through the entire training dataset.
• Batch — a subset of the training data used in one weight update. Mini-batch
gradient descent uses batches typically of 32–256 samples.
• Iteration — one weight update step (one batch processed).
4.2 Overfitting vs. Underfitting
Situation Description & Solution
Underfitting The model is too simple to capture the patterns in the data. High training
AND test error. Fix: use a deeper/wider network, train longer.
Good Fit The model learns the underlying patterns well. Low training error, low
test error. This is the goal!
Overfitting The model memorises the training data but fails on unseen data. Low
training error, HIGH test error. Fix: regularisation, more data, simpler
model.
4.3 Regularisation Techniques
• L1 Regularisation (Lasso) — adds the sum of absolute weight values to the loss.
Encourages sparse models.
• L2 Regularisation (Ridge) — adds the sum of squared weight values to the loss.
Prevents any weight from growing too large.
• Dropout — randomly sets a fraction of neurons to zero during training, forcing the
network to learn redundant representations and preventing co-adaptation.
• Early Stopping — monitor validation loss during training; stop when it starts
increasing.
4.4 Batch Normalisation
Batch Normalisation (BatchNorm) normalises the output of each layer so that it has a
mean of 0 and a standard deviation of 1. Benefits include faster training, reduced
sensitivity to weight initialisation, and slight regularisation effect.
4.5 Optimisers
Optimiser Description
SGD Stochastic Gradient Descent. Simple but requires careful learning rate
tuning. Can use momentum to accelerate.
Adam Adaptive Moment Estimation. Combines momentum and adaptive
learning rates. Default choice for most tasks.
RMSprop Adapts learning rate per parameter. Works well for recurrent networks.
AdaGrad Adapts learning rate based on historical gradients. Good for sparse data.
4.6 Model Evaluation Metrics
Metric Description
Accuracy Proportion of correct predictions. Simple but misleading on
imbalanced datasets.
Precision Of all predicted positives, how many were actually positive?
Precision = TP / (TP + FP)
Recall Of all actual positives, how many did we correctly predict? Recall =
TP / (TP + FN)
F1-Score Harmonic mean of Precision and Recall. Balances both concerns.
Confusion Matrix A table showing TP, TN, FP, FN. Essential for understanding
classification errors.
Module 5: Convolutional Neural Networks (CNNs)
5.1 Why CNNs for Images?
A regular (fully-connected) neural network would treat each pixel in a 224×224 image as
a separate input — that is 50,176 inputs! This is computationally impractical and ignores
spatial relationships between neighbouring pixels. CNNs solve this by exploiting the
spatial structure of images.
Core Idea
Instead of connecting every neuron to every pixel, CNNs slide small filters (kernels)
over the image to detect local patterns such as edges, corners, and textures —
regardless of where they appear in the image.
5.2 The Convolution Operation
A filter (kernel) is a small matrix (e.g., 3×3 or 5×5) that slides across the input image. At
each position, the filter performs an element-wise multiplication with the overlapping
image region and sums the results, producing a single output value. Sliding this filter
across the entire image produces a feature map.
Example: A 3×3 edge-detection filter will produce a feature map highlighting where
edges exist in the image.
5.3 Key CNN Concepts
• Filters / Kernels — learnable parameter matrices that detect specific features.
• Feature Maps — the output of applying a filter to an input. Each filter produces
one feature map.
• Stride — how many pixels the filter moves at each step. Larger stride = smaller
output.
• Padding — adding zeros around the border of the input to control the size of the
feature map. 'Same' padding keeps output the same size as input.
• Depth — the number of filters applied. More filters = more features detected.
5.4 Pooling Layers
Pooling reduces the spatial size of feature maps, decreasing the number of parameters
and computation, and providing a degree of translation invariance.
• Max Pooling — takes the maximum value in each pooling window. Most
common; preserves the strongest feature signal.
• Average Pooling — takes the mean of values in each window. Sometimes used
in later layers.
5.5 CNN Architecture Overview
A typical CNN architecture is:
Input Image → [Conv → ReLU → Pool] × N → Flatten → Fully Connected Layers →
Output (Softmax)
Architecture Significance
LeNet-5 (1998) Pioneer CNN for handwritten digit recognition. Small by
modern standards.
AlexNet (2012) Sparked the deep learning revolution. Won ImageNet by a
large margin.
VGGNet (2014) Very deep (16–19 layers) using only 3×3 convolutions. Simple
and effective.
ResNet (2015) Introduced skip connections allowing training of 100+ layer
networks.
5.6 Mini-Project: Image Classification
Task: Build a CNN using Keras to classify images from the CIFAR-10 dataset (10
classes: aeroplane, car, bird, cat, etc.).
• Load and normalise the CIFAR-10 dataset.
• Build a CNN with at least 2 convolutional blocks.
• Train for 20 epochs and plot training vs. validation accuracy.
• Report the final test accuracy and confusion matrix.
Module 6: Recurrent Neural Networks (RNNs)
6.1 Why RNNs for Sequential Data?
Standard neural networks assume that inputs are independent of each other. But in
many real-world problems, order matters:
• Language — the next word depends on the previous words.
• Time series — tomorrow's stock price depends on recent prices.
• Speech — a sound depends on preceding sounds.
RNNs maintain a hidden state that acts as a memory, passed from one time step to the
next.
6.2 The Recurrence Relation
At each time step t, the RNN takes two inputs: the current input x ₜ and the previous
hidden state hₜ₋₁. It produces a new hidden state hₜ and an output yₜ.
hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁ + b)
yₜ = Wᵧhₜ + bᵧ
6.3 The Vanishing Gradient Problem
When training RNNs on long sequences using backpropagation through time (BPTT),
gradients are multiplied repeatedly as they propagate backwards. This causes:
• Vanishing gradients — gradients shrink to near zero; the network cannot learn
long-term dependencies.
• Exploding gradients — gradients grow uncontrollably; training becomes unstable.
Solution
Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) were
specifically designed to solve the vanishing gradient problem by introducing gating
mechanisms.
6.4 LSTM & GRU
LSTM (Long Short-Term Memory)
LSTMs add a cell state (long-term memory) alongside the hidden state. Three gates
control information flow:
• Forget gate — decides what to erase from cell state.
• Input gate — decides what new information to add to cell state.
• Output gate — decides what to output based on the cell state.
GRU (Gated Recurrent Unit)
A simpler variant of LSTM with only two gates (reset and update). Fewer parameters,
often similar performance, and faster to train.
6.5 Applications
• Natural Language Processing — text generation, machine translation, question
answering.
• Sentiment Analysis — classifying text as positive, negative, or neutral.
• Time-Series Forecasting — stock prices, weather, energy demand.
• Speech Recognition — converting spoken audio to text.
6.6 Mini-Project: Sentiment Analysis
Task: Build an LSTM model to classify movie reviews from the IMDB dataset as
positive or negative.
• Load the IMDB dataset (25,000 training, 25,000 test reviews).
• Preprocess: tokenise and pad sequences to equal length.
• Build an Embedding + LSTM model.
• Achieve and report test accuracy. Target: >85%.
Module 7: Modern Deep Learning Concepts
7.1 Transfer Learning
Transfer learning allows us to leverage a model pre-trained on a large dataset (e.g.,
ImageNet with 1.2 million images) and adapt it to our smaller, specific task. This
dramatically reduces training time and data requirements.
• Feature extraction — freeze all pre-trained layers, add a new output head, and
train only the new head.
• Fine-tuning — unfreeze the top layers of the pre-trained model and train them
with a very small learning rate alongside your new head.
Popular pre-trained models: VGG16, ResNet50, InceptionV3, MobileNet, EfficientNet.
7.2 Introduction to the Transformer Architecture
Transformers (Vaswani et al., 2017) revolutionised natural language processing — and
later computer vision. Instead of processing sequences step-by-step like RNNs,
Transformers process all tokens simultaneously using a mechanism called Self-
Attention.
• Self-Attention — each word can 'attend' to every other word in the sentence,
capturing long-range dependencies without the vanishing gradient problem.
• Multi-Head Attention — runs multiple attention mechanisms in parallel, capturing
different types of relationships.
• Positional Encoding — adds information about word order since the architecture
is order-agnostic.
Why It Matters
Transformers are the foundation of modern AI systems including BERT, GPT-4, and
Claude. Understanding attention is key to understanding contemporary AI.
7.3 Autoencoders
An autoencoder is a neural network trained to compress data into a low-dimensional
representation (encoding) and then reconstruct the original input from that compressed
form (decoding).
• Encoder — maps input to a compressed latent representation.
• Latent Space — the compressed representation. Forces the network to learn
only the most essential features.
• Decoder — reconstructs the original input from the latent representation.
Applications: dimensionality reduction, anomaly detection, image denoising, data
compression.
7.4 Generative Adversarial Networks (GANs) — Conceptual Overview
GANs (Goodfellow et al., 2014) consist of two competing networks trained
simultaneously:
• Generator — tries to create fake data (images, audio) that looks real.
• Discriminator — tries to distinguish between real and generated (fake) data.
Through this adversarial competition, the generator improves until the discriminator can
no longer tell real from fake. GANs can generate photorealistic images, deepfakes, and
synthetic training data.
7.5 Ethics in Deep Learning
As practitioners, we have a responsibility to understand the societal impact of the
models we build.
Issue Description
Bias & Fairness Models trained on biased data will reproduce and amplify those
biases. Facial recognition systems have shown higher error rates
for darker skin tones.
Explainability Deep learning models are often 'black boxes'. Techniques like
SHAP and LIME help explain model decisions.
Data Privacy Training on personal data raises privacy concerns. Techniques like
federated learning and differential privacy address this.
Environmental Impact Large language models consume significant computing resources
and energy. Responsible AI includes considering environmental
cost.
Module 8: Tools & Practical Deep Learning
8.1 Python Ecosystem
Library Role
NumPy Numerical computing library. Provides N-dimensional arrays and
mathematical functions.
Pandas Data manipulation and analysis. Reading datasets, cleaning data,
exploratory analysis.
Matplotlib / Seaborn Data visualisation. Plotting training curves, distributions, and
evaluation metrics.
Scikit-learn Classical ML tools, preprocessing utilities, and model evaluation
helpers.
8.2 TensorFlow & Keras
TensorFlow (developed by Google) is one of the most widely used deep learning
frameworks. Keras is a high-level API built on TensorFlow that makes building and
training models straightforward.
Core Keras workflow:
4. Define the model architecture using [Link]() or the Functional API.
5. Compile: specify optimizer, loss function, and metrics.
6. Train: call [Link]() with training data.
7. Evaluate: call [Link]() on test data.
8. Predict: call [Link]() on new data.
8.3 PyTorch
PyTorch (developed by Meta/Facebook) is the preferred framework in research. It uses
dynamic computation graphs, making debugging more intuitive. It is also widely used in
production.
Core PyTorch workflow:
9. Define model as a class inheriting from [Link].
10. Define the forward() method.
11. Set up a DataLoader, loss function (criterion), and optimiser.
12. Write the training loop manually (forward, loss, backward, step).
8.4 Google Colab
Google Colab provides free GPU access via a Jupyter-notebook interface in the
browser. It is the recommended environment for this course.
• Access at: [Link]
• Enable GPU: Runtime → Change runtime type → GPU.
• Install packages with: !pip install package_name
• Mount Google Drive to save model weights and data.
8.5 Reading Model Performance
• Training loss decreasing + Validation loss decreasing → Good. Continue training.
• Training loss decreasing + Validation loss increasing → Overfitting. Stop and
regularise.
• Both losses high and stagnant → Underfitting. Increase model capacity or train
longer.
• Loss not decreasing at all → Check learning rate, data preprocessing, and model
architecture.
8.6 End-to-End Project Walkthrough
Task: Classify handwritten digits using the MNIST dataset.
13. Load Data — import MNIST from [Link]; normalise pixel values to [0,1].
14. Explore Data — print shapes, visualise sample images.
15. Build Model — Dense layers with ReLU activations; Softmax output for 10
classes.
16. Compile — optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'].
17. Train — [Link](X_train, y_train, epochs=10, validation_split=0.1).
18. Evaluate — report test accuracy (~98% achievable on MNIST).
19. Visualise — plot training curves; show sample predictions with true labels.
GROUP ASSIGNMENT
Deep Learning Algorithm Presentation
Detail Information
Course Deep Learning for Beginners — Year 1 Data Science
Assignment Type Group Presentation
Total Marks 100 marks
Presentation Duration 20 minutes + 10 minutes Q&A
Submission Deadline Week 12 (exact date to be confirmed by lecturer)
Group Size 3 – 4 students per group
Overview
Each group will choose one deep learning algorithm from the list below and prepare a
20-minute group presentation. The presentation must demonstrate a thorough
understanding of the algorithm, its mathematical foundations, practical implementation,
and real-world relevance. Every group member is expected to present a portion of the
work.
Algorithm Selection Table
Each group must choose a unique algorithm. Selections are on a first-come, first-served
basis — notify your lecturer once your group has decided.
Group Algorithm Applications Suggested Demo
Group 1 Convolutional Neural Network Image classification, object Build a CNN image
(CNN) detection, medical imaging classifier
Group 2 Long Short-Term Memory Text generation, sentiment Sentiment analysis
(LSTM) analysis, time-series on IMDB
forecasting
Group 3 Transformer / Self-Attention Language translation, Text classification
chatbots, question answering with BERT
Group 4 Generative Adversarial Image synthesis, data Generate
Network (GAN) augmentation, art generation handwritten digits
Group 5 Autoencoder Anomaly detection, image Denoise noisy
denoising, dimensionality MNIST images
reduction
Group 6 Recurrent Neural Network Language modelling, speech Character-level text
(RNN) processing, sequence generator
prediction
Group 7 Transfer Learning Medical image classification, Fine-tune ResNet
(ResNet/VGG) domain adaptation on custom dataset
Group 8 Graph Neural Network (GNN) Social network analysis, Node classification
molecular property prediction on Cora dataset
Group 9 Variational Autoencoder (VAE) Image generation, latent Generate new digit
space interpolation images
Group 10 Attention Mechanism (without Image captioning, alignment Attention-based
Transformer) in seq-to-seq models image captioning
Presentation Requirements
Your presentation must cover all of the following sections, allocated within your 20-
minute slot:
Section Content
1. Introduction (2 min) Briefly introduce the algorithm. What problem does it solve?
What is its history?
2. Architecture & Theory (5 Explain how the algorithm works. Use diagrams. Describe the
min) key components and mathematical operations (at an intuitive
level).
3. Training Process (3 min) How is the model trained? What is the loss function? What
optimiser is typically used?
4. Code Implementation (5 Live demo or walk-through of a Python implementation using
min) TensorFlow/Keras or PyTorch. Show the model building,
training, and results.
5. Real-World Applications Give 3 real-world examples of the algorithm being used in
(2 min) industry or research.
6. Advantages & Limitations What does the algorithm do well? Where does it struggle?
(2 min) What are its computational requirements?
7. Q&A (10 min) Be prepared to answer questions from the lecturer and peers
about your algorithm and implementation.
Marking Rubric
Criterion Description Marks Score
Conceptual How well does the group explain the 30
Understanding algorithm? Are explanations clear and
accurate?
Technical Implementation Quality and correctness of the code demo. 25
Does it run? Are results presented?
Presentation Quality Clear slides, good structure, time 20
management, confident delivery.
Real-World Relevance Depth and quality of application examples 10
selected.
Q&A Performance Ability to answer questions accurately and 10
confidently.
Group Collaboration Evidence that all members contributed 5
meaningfully.
TOTAL 100
Submission Deliverables
Each group must submit the following on the due date:
20. Presentation slides (PDF format) — uploaded to the course portal.
21. Jupyter Notebook (.ipynb) — containing all code used in the demo, with outputs
and comments.
22. One-page group reflection — each member writes 1 paragraph describing their
individual contribution.
23. Group member list — names and student IDs of all members.
General Rules & Academic Integrity
• All work must be the group's own. Direct copying from online sources without
attribution is plagiarism.
• You may reference tutorials and documentation but must understand and explain
every line of code you present.
• Groups that exceed 25 minutes will be stopped.
• Attendance on presentation day is compulsory for all group members. Absent
members receive zero for the Q&A section.
• Presentation order will be assigned randomly one week before the presentation
day.
Tip for Success
Start early! Spend the first week understanding the theory, the second week on
implementation, and the third week polishing your slides and rehearsing. The groups
that do best are those that truly understand their algorithm — not just copy code.
Good luck to all groups! Present with confidence, curiosity, and clarity.