DEEP LEARNING
Complete Semester Study Guide
M.E. – CS – R2023 – CBCS • P23CSP13 • Category: PEC • L:3 T:0 P:0 C:3
KPR Institute of Engineering and Technology, Coimbatore – 641 497
Unit Title Periods
I Basics of Neural Networks 9
II Fundamentals of Deep Networks 9
III Major Architectures of Deep Networks 9
IV Tuning Specific Deep Network Architectures 9
V Applications of Deep Learning 9
Total 45
COURSE OBJECTIVES
• To explain the basic concepts of neural networks and deep networks.
• To discuss the major architectures of deep networks.
• To demonstrate the applications of deep learning.
UNIT 1 • 9 PERIODS
P23CSP13
■ BASICS OF NEURAL NETWORKS Deep Learning
1 Neural Network Basics & Binary Classification
σ y = σ(W·x + b) | L layers | Backpropagation
σ
σ
σ
σ σ
σ σ
Layer Layer 1 Layer 2 Layer
A Neural Network
Input is a computational model inspired by biological
Hidden Hidden neurons. It consists
Outputof layers of interconnected
nodes (neurons) that transform input data to produce an output through learned weights and biases.
Binary Classification maps input features to one of two classes (y ∈ {0, 1}). The output neuron produces a
probability using the sigmoid activation function, and a threshold (typically 0.5) decides the class.
• Input Layer: receives raw features (pixels, sensor values, text vectors)
• Hidden Layers: learn increasingly abstract representations
• Output Layer: produces final prediction or probability
• Weights (W) & Biases (b): learnable parameters updated during training via backpropagation
2 Logistic Regression & Gradient Descent
← Minimum ∇L = gradient of loss
α = learning rate
w = w - α·∇L
Start
Gradient Descent — Minimizing Loss Step by Step
Logistic Regression applies the sigmoid function to produce a probability: ■ = σ(wTx + b) where σ(z) = 1/(1+e−z).
Gradient Descent is the core optimization algorithm. It minimizes the loss by repeatedly computing the gradient
and updating parameters in the direction that reduces the loss.
• Update rule: w ← w − α · (∂L/∂w) and b ← b − α · (∂L/∂b)
• Learning Rate (α): controls step size — too large diverges, too small converges slowly
• Variants: Batch GD (all data), Mini-batch GD (subset), Stochastic GD (one sample)
• Advanced optimizers: Momentum, RMSProp, Adam (adaptive learning rates)
3 Activation Functions
Sigmoid σ(x)=1/(1+e■■)
ReLU max(0,x)
Tanh
Activation Functions — Introduce Non-linearity
Activation functions introduce non-linearity, allowing networks to learn complex patterns beyond simple linear
transformations.
• Sigmoid σ(x): Output [0,1] — used in binary classification output; suffers from vanishing gradient
• Tanh(x): Output [−1,1] — zero-centered, better than sigmoid for hidden layers
• ReLU: f(x) = max(0,x) — most popular; computationally fast; avoids vanishing gradient
• Leaky ReLU: f(x) = max(0.01x, x) — fixes the 'dying ReLU' problem
• Softmax: converts raw scores to probabilities summing to 1 — used in multi-class output layers
4 Loss Functions & Hyperparameters
Loss Function Formula Use Case
Binary Cross-Entropy −[y·log(■) + (1−y)·log(1−■)] Binary Classification
Categorical Cross-Entropy −Σ y■·log(■■) Multi-class Classification
Mean Squared Error (1/n)Σ(y − ■)² Regression
Hinge Loss max(0, 1 − y·f(x)) SVM / Margin-based
Hyperparameters are settings configured BEFORE training (not learned from data):
• Learning rate (α): step size for gradient descent (typical: 0.001 – 0.1)
• Number of layers & neurons: define model capacity and expressiveness
• Batch size: number of samples per gradient update (32, 64, 128, 256)
• Epochs: full passes over the training dataset
• Regularization: L1 (sparse), L2 (weight decay), Dropout — prevent overfitting
UNIT 2 • 9 PERIODS
P23CSP13
■■ FUNDAMENTALS OF DEEP NETWORKS Deep Learning
1 Defining Deep Learning
Deep Learning is a subset of Machine Learning using neural networks with many hidden layers to automatically
learn hierarchical feature representations from raw data.
• Traditional ML requires hand-crafted features; Deep Learning learns features automatically
• Each layer learns increasingly abstract representations: edges → shapes → objects → concepts
• 'Deep' refers to the number of hidden layers — more depth = more expressive power
• Enabled by: large datasets, GPU computing, improved architectures (ReLU, BatchNorm, Residuals)
Layer What it Learns Example (Images)
Layer 1 Low-level features Edges, gradients, corners
Layer 2 Mid-level patterns Curves, textures, shapes
Layer 3 High-level concepts Eyes, wheels, noses
Layer N Abstract semantics Faces, cars, animals
2 Architectural Principles & Building Blocks
Common Architectural Principles:
• Depth: multiple stacked layers enable hierarchical learning
• Shared Weights: same parameters applied across input positions (CNNs) — reduces parameters drastically
• Local Connectivity: neurons connect only to local regions, not entire input
• Skip Connections: shortcuts between non-adjacent layers (ResNet) — enable training of 100+ layer networks
• Normalization: Batch Norm / Layer Norm stabilizes and accelerates training
Key Building Blocks:
• Dense (FC) Layer: every neuron connects to every neuron in next layer — general-purpose
• Convolutional Layer: applies learned filters to detect local spatial patterns
• Pooling Layer: reduces spatial dimensions while preserving important features
• Dropout Layer: randomly deactivates neurons during training — powerful regularizer
• Embedding Layer: maps discrete tokens to dense continuous vectors (NLP)
UNIT 3 • 9 PERIODS
P23CSP13
■■ MAJOR ARCHITECTURES OF DEEP NETWORKSDeep Learning
1 Convolutional Neural Networks (CNN)
10
7×7 Output
14×14 Pool FC
28×28 +ReLU +ReLU Dense
Input Conv Pool
Conv
CNN Architecture: Input → [Conv→ReLU→Pool]■ → Flatten → FC → Softmax
CNNs are specialized architectures for processing grid-like structured data (images, audio spectrograms). They
exploit spatial locality and parameter sharing.
• Convolution: a small filter slides across the input, computing element-wise products — detects local patterns
• Feature Map: output of applying a filter; each filter detects a specific pattern (edge, curve, color)
• Stride: how many pixels the filter moves each step (stride=2 halves spatial dimensions)
• Padding: adding zeros around input to control output size
• Max Pooling: takes the maximum value in each region — translational invariance
• Output size formula: (N − F + 2P)/S + 1 where N=input size, F=filter, P=padding, S=stride
• Famous CNNs: LeNet → AlexNet → VGGNet → GoogLeNet → ResNet → EfficientNet → ViT
2 Recurrent Neural Networks (RNN)
x1 x2 x3 x4
RNN RNN RNN RNN
→ → →
cell cell cell cell
h1 h2 h3 h4
h_t = tanh(W_h·h_{t-1} + W_x·x_t + b)
RNNs are designed for sequential and temporal data. They maintain a hidden state that acts as memory, allowing
information to persist across timesteps.
• Hidden state update: ht = tanh(Whht−1 + Wxxt + b)
• Vanishing Gradient Problem: gradients shrink exponentially over long sequences — limits memory
• LSTM (Long Short-Term Memory): adds gates (Forget, Input, Output) to selectively retain information
• GRU (Gated Recurrent Unit): simplified LSTM with fewer parameters — often equally effective
• Applications: language modeling, machine translation, sentiment analysis, time series prediction
• Bidirectional RNN: processes sequences in both directions for richer context
3 Recursive Neural Networks & Unsupervised Pre-trained Networks
Input Output
Enc 1 Dec 2
Enc 2 ↑ Bottleneck ↑ Dec 1
Latent z Latent z
Autoencoder: Encoder compresses → Bottleneck → Decoder reconstructs
Recursive Neural Networks apply the same weights to tree-structured hierarchical inputs (e.g., sentence parse
trees in NLP). Different from RNNs — they operate on recursive rather than sequential structure.
Autoencoders learn compressed latent representations in an unsupervised manner:
• Encoder: compresses input x into a compact latent code z
• Decoder: reconstructs x■ from z — trained to minimize ||x − x■||²
• Variational Autoencoder (VAE): learns a probabilistic latent space — enables generation
Tuning Deep Networks:
• Weight Initialization: Xavier (tanh), He (ReLU) — prevent vanishing/exploding at start
• Batch Normalization: normalizes layer inputs — dramatically accelerates training
• Learning Rate Scheduling: decay LR over training (step decay, cosine annealing)
• Early Stopping: monitor validation loss and stop when it stops improving
UNIT 4 • 9 PERIODS
P23CSP13
■■ TUNING SPECIFIC DEEP NETWORK ARCHITECTURES
Deep Learning
1 CNN Architecture Deep Dive
Convolution Operation in Detail:
• A filter (kernel) of size F×F slides over the input with a defined stride
• Each position: element-wise multiply patch with filter → sum all values → one output number
• Multiple filters detect different patterns — stacking filters forms a feature map tensor
• Output spatial size = (N − F + 2P) / S + 1
Types of Convolutions:
• Standard Conv: basic sliding window — computationally expensive for large channels
• Depthwise Separable Conv: splits channel and spatial filtering — used in MobileNet (10× fewer params)
• Dilated/Atrous Conv: expanded receptive field without increasing parameters
• Transposed Conv: learned upsampling — used in segmentation and generative networks
Landmark CNN Architectures:
Architecture Year Key Innovation Depth
LeNet-5 1998 First CNN for digit recognition 7 layers
AlexNet 2012 ReLU, Dropout, GPU training 8 layers
VGGNet 2014 3×3 filters consistently stacked 16–19 L
GoogLeNet 2014 Inception modules, 1×1 conv bottleneck 22 layers
ResNet 2015 Skip connections — 152 layers possible 50–152 L
EfficientNet 2019 Compound scaling of depth/width/resolution B0–B7
2 Restricted Boltzmann Machines (RBM) & Deep Belief Networks (DBN)
Restricted Boltzmann Machine (RBM): An undirected probabilistic graphical model with two layers — Visible (v)
and Hidden (h). 'Restricted' means no intra-layer connections.
• Energy function: E(v,h) = −aTv − bTh − vTWh
• Learning: Contrastive Divergence (CD) algorithm — approximate maximum likelihood
• Training: positive phase (clamp data, compute hidden probs) → negative phase (reconstruct)
Deep Belief Network (DBN): A stack of RBMs trained greedily, one layer at a time:
• Step 1: Train bottom RBM on raw data
• Step 2: Use hidden activations as visible layer for next RBM
• Step 3: Repeat upward; fine-tune full network with backpropagation
• Key insight: DBNs solved the vanishing gradient problem BEFORE ReLU and BatchNorm became standard
Recurrent Neural Networks — Advanced Tuning:
• Gradient clipping: cap gradient norm to prevent exploding gradients
• Teacher forcing: use true labels during LSTM training for stability
• Sequence padding and masking for variable-length inputs
UNIT 5 • 9 PERIODS
P23CSP13
■ APPLICATIONS OF DEEP LEARNING Deep Learning
1 Computer Vision & Large-Scale Deep Learning
Task Model Description
Image Classification ResNet, EfficientNet, ViT Assigns one label to entire image
Object Detection YOLO, Faster R-CNN, SSD Locates and classifies multiple objects
Semantic Segmentation U-Net, DeepLab, SegFormer Labels every pixel in the image
Face Recognition FaceNet, ArcFace Identifies or verifies individuals
Image Generation GANs, Diffusion Models Creates new realistic images
Large-Scale Deep Learning Techniques:
• Data Parallelism: split mini-batches across multiple GPUs — each GPU holds a full model copy
• Model Parallelism: split the model itself across devices — for models too large to fit in one GPU
• Mixed Precision Training: use FP16 computations — 2× speed, half the memory
• Model Compression: Pruning (remove weak weights), Quantization (INT8), Knowledge Distillation
2 NLP, Speech Recognition & Recommender Systems
Q Query K Key V Value
Scaled Dot-Product Attention
Attention(Q, K, V)
softmax(QK^T / sqrt(d_k)) V
Multi-Head Attention
Transformer Self-Attention — Foundation of BERT, GPT, T5, Whisper
Natural Language Processing (NLP):
• Text Classification & Sentiment Analysis: LSTM, BERT fine-tuning
• Machine Translation: Seq2Seq + Attention, Transformer (encoder-decoder)
• Transformer Self-Attention: Attention(Q,K,V) = softmax(QKT/√dk)V — captures global context
• Modern LLMs (GPT, BERT, T5, LLaMA) are stacked Transformer blocks with billions of parameters
Speech Recognition Pipeline:
• Acoustic Model (CNN+RNN) → Language Model → CTC Decoding → Text output
• End-to-end models: Listen-Attend-Spell, Whisper (OpenAI) — trained on 680K hours of audio
Recommender Systems:
• Collaborative Filtering with neural embeddings — learns user and item representations
• Wide & Deep (Google): memorization (wide) + generalization (deep) combined
Healthcare Deep Learning:
• Chest X-ray diagnosis, retinal scan analysis, cancer pathology (CNN-based)
• Drug discovery: molecular property prediction, protein folding (AlphaFold)
3 Deep Learning Tools: TensorFlow, Keras & MatConvNet
TF K PT MC
TensorFlow Keras PyTorch MatConvNet
Production + Mobile High-level API Research + Dynamic MATLAB CNN
✓ Build ✓ Build ✓ Build ✓ Build
✓ Train ✓ Train ✓ Train ✓ Train
✓ Deploy ✓ DeployWorkflow: Define → Compile✓→Deploy
Common Fit → Evaluate → Deploy ✓ Deploy
TensorFlow (Google, 2015): Production-scale open-source ML framework. Supports eager execution (TF 2.x),
TensorBoard for visualization, TFServing for deployment, and TFLite for mobile/edge.
Keras: High-level API integrated into TensorFlow as [Link]. Extremely developer-friendly with Sequential and
Functional APIs. Key methods: [Link]() → [Link]() → [Link]().
MatConvNet: MATLAB toolbox for CNNs. Used in academic research, supports pre-trained models for image
classification and detection tasks.
PyTorch (Meta, 2016): Dynamic computation graphs, Pythonic API, dominant in research. Ecosystem: torchvision,
torchaudio, torchtext. Most SOTA papers use PyTorch.
• Typical Workflow: Load data → Define model → Compile (loss + optimizer) → Fit (train) → Evaluate →
Save/Deploy
REFERENCES
1. Adam Gibson, Josh Patterson — Deep Learning: A Practitioner's Approach, O'Reilly Media, 2017.
2. Ian Goodfellow, Yoshua Bengio and Aaron Courville — Deep Learning, MIT Press, 2016.
3. Yuxi (Hayden) Liu — Python Machine Learning by Example, First Edition, 2017.
4. Daniel Graupe — Deep Learning Neural Networks: Design and Case Studies, World Scientific Publishing, 2016.
5. Yu and Li Deng — Deep Learning: Methods and Applications, Now Publishers Inc., 2014.