Deep Learning

Complete BTech Exam Notes

Unit 4 & Unit 5

Topic Unit

Recurrent Neural Networks (RNNs) Unit 4

Backpropagation Through Time (BPTT) Unit 4

Long Short-Term Memory (LSTM) Unit 4

Bidirectional LSTMs & RNNs Unit 4

Gated Recurrent Unit (GRU) Unit 4

TensorFlow Unit 5

Keras Unit 5

PyTorch Unit 5

Image Processing & Segmentation Unit 5

Object Detection Unit 5

Speech Recognition Unit 5

Image Recognition Unit 5

Recommendation Systems Unit 5

UNIT 4: Recurrent Neural Networks (RNNs)

1. Sequence Modelling using RNNs

Sequence modelling deals with data where order matters — like sentences, time series, audio, or video. Unlike
feedforward networks that treat each input independently, RNNs maintain a memory of past inputs.

Core Equations:
h(t) = tanh(W_h · h(t-1) + W_x · x(t) + b)
y(t) = W_y · h(t) + b_y
Where: x(t) = input at time t, h(t) = hidden state (memory), y(t) = output at time t

Types of RNN Architectures:

Type Description Example Use

One-to-One Single input → Single output Image classification

One-to-Many Single input → Sequence output Image captioning

Many-to-One Sequence input → Single output Sentiment analysis

Many-to-Many (same) Sequence → Sequence (same length) Video frame labelling

Many-to-Many (Enc-Dec) Sequence → Sequence (diff length) Machine translation

Problems with Vanilla RNNs:

• Vanishing Gradient: Gradients shrink exponentially → network forgets long-term dependencies
• Exploding Gradient: Gradients grow exponentially → unstable training (fixed using gradient clipping)

2. Backpropagation Through Time (BPTT)

BPTT is the algorithm used to train RNNs. Since RNNs are unrolled through time, backpropagation is applied
across all time steps.

Steps of BPTT:
• Forward Pass: Compute outputs for all time steps t = 1 to T
• Compute Loss: Total loss L = Σ L(t) for t = 1 to T
• Backward Pass: Propagate gradients backward through all unrolled time steps
• Update Weights: Use gradients to update W_h, W_x, W_y

Key Formula:
∂L/∂W = Σ (∂L(t)/∂W) for each time step t

Truncated BPTT:
To avoid vanishing/exploding gradients, we only backpropagate through k time steps instead of all T steps.
This is a practical approximation used in real training.

3. Long Short-Term Memory (LSTM)

LSTMs were introduced by Hochreiter & Schmidhuber (1997) to solve the vanishing gradient problem and
learn long-term dependencies.

An LSTM has two state vectors: h(t) — Hidden state (short-term memory) and C(t) — Cell state (long-term
memory, the 'conveyor belt').

The Three Gates + Cell Update:

Forget Gate — What to forget from cell state:
f(t) = σ(W_f · [h(t-1), x(t)] + b_f) Output: 0 = forget, 1 = remember

Input Gate — What new info to store:

i(t) = σ(W_i · [h(t-1), x(t)] + b_i) C■(t) = tanh(W_C · [h(t-1), x(t)] + b_C)

Cell State Update:

C(t) = f(t) * C(t-1) + i(t) * C■(t)

Output Gate — What to output:

o(t) = σ(W_o · [h(t-1), x(t)] + b_o) h(t) = o(t) * tanh(C(t))

Why LSTMs Work:

• The cell state flows through with only linear interactions (multiplicative gates)
• Gradients flow without vanishing over long sequences
• The network learns what to remember and forget

LSTM Applications:
Machine translation, Speech recognition, Text generation, Stock price prediction

4. Bidirectional LSTMs (BiLSTMs)

A standard LSTM processes sequences left to right only. But in many tasks (e.g., NLP), future context is also
important.

Example: 'He said, Teddy bears are on sale' vs 'He said, Teddy Roosevelt was great' — the word 'Teddy' needs
future context to be understood.

Architecture:
Forward LSTM: x(1) → x(2) → ... → x(T) (left to right)
Backward LSTM: x(T) → x(T-1) → ... → x(1) (right to left)
Output at t: ■(t) = [h_forward(t) ; h_backward(t)] (concatenated)

Applications:
Named Entity Recognition (NER), POS tagging, Question Answering, Sentiment Analysis

Limitation:
Cannot be used for real-time / online predictions — needs full sequence upfront.

5. Bidirectional RNNs (BiRNNs)

Same as BiLSTM but uses vanilla RNN cells instead of LSTM cells.
 Feature BiRNN BiLSTM

Memory Short-term only Long + short term

Gradient issues Vanishing gradients Handles well

Complexity Lower Higher

Performance Moderate Better for long sequences

6. Gated Recurrent Unit (GRU) Architecture

GRU (Cho et al., 2014) is a simpler alternative to LSTM with fewer parameters but comparable performance.

Feature LSTM GRU

Gates 3 (forget, input, output) 2 (reset, update)

States 2 (h, C) 1 (h)

Parameters More Fewer

Training Speed Slower Faster

Performance Slightly better (long seq) Comparable

GRU Equations:
Reset Gate: r(t) = σ(W_r · [h(t-1), x(t)])
Update Gate: z(t) = σ(W_z · [h(t-1), x(t)])
Candidate: h■(t) = tanh(W · [r(t) * h(t-1), x(t)])
Final State: h(t) = (1 - z(t)) * h(t-1) + z(t) * h■(t)

• When z(t) ≈ 1 → carry forward past state (remember)

• When z(t) ≈ 0 → use new candidate (update)
UNIT 5: Deep Learning Tools & Applications

1. TensorFlow
Developed by Google Brain (2015), open-source. Supports static computation graphs (TF 1.x) and eager
execution (TF 2.x).

Key Features:
• [Link]: Core data structure (n-dimensional arrays)
• [Link]: Trainable parameters
• [Link]: Automatic differentiation
• TensorBoard: Visualization tool for loss, accuracy, graphs

Basic Workflow (Python):

model = [Link]([
[Link](128, activation='relu'),
[Link](10, activation='softmax')
])
[Link](optimizer='adam', loss='sparse_categorical_crossentropy')
[Link](X_train, y_train, epochs=10)

Ecosystem:
TensorFlow Lite (mobile/edge), [Link] (browser), TF Extended/TFX (production pipelines)

2. Keras
High-level API for building neural networks. Now fully integrated into TensorFlow as [Link]. Originally a
standalone library by François Chollet.

API Type Description

Sequential API Linear stack of layers — simplest approach

Functional API Complex multi-input/output models

Subclassing API Custom model classes with full flexibility

Key Layers:
Dense (fully connected), Conv2D (convolutional), LSTM/GRU (recurrent), Dropout (regularization),
BatchNormalization (normalize activations), Flatten/Reshape (shape manipulation)

Callbacks:
• ModelCheckpoint: Save best model during training
• EarlyStopping: Stop when validation loss plateaus
• ReduceLROnPlateau: Reduce learning rate dynamically

3. PyTorch
Developed by Facebook AI Research / FAIR (2016). Uses dynamic computation graphs (define-by-run).
Preferred in research settings due to flexibility.
 Feature PyTorch TensorFlow

Graph type Dynamic Static (TF1) / Dynamic (TF2)

Debugging Easy (pythonic) Harder

Deployment TorchServe TF Serving, TFLite

Research use Very popular Also popular

Industry use Growing fast Dominant

Key Components:
[Link] (core array), autograd (auto-differentiation), DataLoader (batched data), [Link]
(optimizers: SGD, Adam), torchvision (image datasets and transforms)

4. Deep Learning Applications in Image Processing

CNN Architecture: Input Image → Conv → ReLU → Pool → Conv → ReLU → Pool → FC → Output

Model Year Key Innovation

LeNet 1998 First CNN architecture

AlexNet 2012 Deep CNN, ReLU, Dropout

VGGNet 2014 Small 3×3 filters, very deep

ResNet 2015 Skip/residual connections

Inception 2014 Multi-scale filters in parallel

EfficientNet 2019 Compound scaling strategy

5. Image Segmentation
Assigns a class label to every pixel in an image (pixel-level classification).

Types:
Type Description Example

Semantic Every pixel labelled by class; no instance distinction All cars = "car"

Instance Distinguishes individual objects of same class Car1, Car2 separately

Panoptic Combines semantic + instance segmentation Full scene understanding

Key Architectures:
• FCN: Replaces FC layers with conv layers, outputs spatial heatmaps
• U-Net: Encoder-Decoder with skip connections; used in medical imaging
• DeepLab: Uses atrous (dilated) convolutions for multi-scale context
• Mask R-CNN: Extends Faster R-CNN with mask prediction branch

6. Object Detection
Locates and classifies multiple objects in an image with bounding boxes. Output: [class_label, x, y, width,
height, confidence]
Two-Stage Detectors (Accurate, Slower):
• R-CNN (2013): Region proposals → CNN features → SVM classifier
• Fast R-CNN (2015): Entire image through CNN once → RoI pooling
• Faster R-CNN (2016): Introduces Region Proposal Network (RPN) — end-to-end trainable

One-Stage Detectors (Fast, Real-time):

• YOLO: Divides image into S×S grid; each cell predicts bounding boxes + class probabilities
• SSD: Predicts at multiple scales using feature pyramid

Key Metrics:
IoU (Intersection over Union): overlap between predicted and ground truth box | mAP: mean Average Precision
| NMS: Non-Maximum Suppression to remove duplicates

7. Speech Recognition (ASR)

Converts audio waveform → text transcript. Pipeline: Audio Signal → Feature Extraction → Acoustic Model →
Language Model → Text

Feature Extraction:
• MFCC (Mel Frequency Cepstral Coefficients): Most common audio features
• Spectrogram: Time-frequency representation of audio

Deep Learning Models:

Model Description

RNN/LSTM + CTC CTC handles alignment between audio and text; no pre-segmented data needed

Attention Encoder-Decoder Encoder processes audio; Decoder generates text with attention

Whisper (OpenAI) Transformer-based; robust ASR across multiple languages

wav2vec 2.0 (Meta) Self-supervised pre-training on raw audio

Challenges:
Accents and dialects, background noise, homophones ('there' vs 'their'), real-time processing requirements

8. Image Recognition
Classifies an entire image into one (or multiple) category labels. Pipeline: Image → CNN Feature Extractor →
Global Average Pooling → FC → Softmax → Class

Transfer Learning:
Use pre-trained model (on ImageNet) as feature extractor, then fine-tune on your specific dataset.
• Freeze early layers of pre-trained model (e.g. ResNet)
• Replace final FC layer with task-specific layer
• Train only the new layers (or fine-tune all layers slowly)

Data Augmentation:
Random crop, flip, rotation, colour jitter, Mixup, CutOut, CutMix — prevents overfitting on small datasets.
Modern Approaches:
• Vision Transformers (ViT): Apply transformer architecture to image patches
• EfficientNet: Best accuracy/efficiency trade-off
• CLIP: Joint image-text training for zero-shot recognition

9. Recommendation Systems
Predicts what items (movies, products, songs) a user would prefer.

Types:
Type Basis Approach

Collaborative Filtering User-item interaction history Matrix Factorization, NCF

Content-Based Filtering Item features/attributes Feature similarity matching

Hybrid Systems Both approaches combined Used by Netflix, Spotify

Deep Learning Models:

• NCF (Neural Collaborative Filtering): Replaces dot product with MLP for complex interactions
• Wide & Deep (Google): Wide = memorization; Deep = generalization; combined for app recommendations
• Autoencoders: Encode user preferences into latent space, reconstruct predicted ratings
• BERT4Rec: BERT-style transformer model for sequential recommendation

Key Challenges:
Cold Start (new users/items), Scalability (millions of users), Sparsity (few ratings per user), Filter Bubble
(only recommending similar items)
Quick Revision Summary

Topic Key Points

RNN Sequence model with hidden state h(t) = tanh(W_h·h(t-1) + W_x·x(t))

BPTT Backprop unrolled through time steps; Truncated BPTT for efficiency

LSTM Forget / Input / Output gates + Cell state C(t); solves vanishing gradient

BiLSTM Forward + Backward LSTM concatenated; captures past & future context

GRU Reset + Update gates, single state — simpler & faster than LSTM

TensorFlow Google, eager execution (TF2), production-ready, TensorBoard

Keras High-level API: Sequential / Functional / Subclass APIs

PyTorch Facebook, dynamic graph, research-friendly, autograd

Image Segmentation Pixel-level classification; U-Net, DeepLab, Mask R-CNN

Object Detection Bounding boxes; YOLO (fast), SSD, Faster R-CNN (accurate)

Speech Recognition Audio → Text; MFCC features, CTC, Transformers (Whisper)

Image Recognition Whole image classification; CNNs, Transfer Learning, ViT

Recommendation User-item matching; Collaborative Filtering, NCF, Wide & Deep

Good Luck with Your BTech Exam! ■