% Machine Learning Study Guide - Complete Unit-Wise Coverage

% With Q&A; Format and Exam-Friendly Answers

% November 2025
---
# Machine Learning Study Guide
## Complete Unit-Wise Coverage with Q&A; and Exam Answers
---
\newpage
## UNIT I: MACHINE LEARNING BASICS
### SHORT NOTES
**1. Learning Algorithms**
- Procedures that automatically improve through experience
- Learn patterns from training data
- Examples: Linear regression, decision trees, neural networks, SVM
**2. Capacity, Overfitting & Underfitting**
- **Capacity**: Model's ability to fit diverse functions
- **Overfitting**: Model fits training data too well but generalizes poorly
- **Underfitting**: Model is too simple to capture underlying patterns
- Solution: Use validation set and regularization
**3. Hyperparameters & Validation Sets**
- **Hyperparameters**: Parameters set before training (learning rate, batch size, regularization
strength)
- **Validation Set**: Separate dataset used to tune hyperparameters without contaminating test set
- Prevents overfitting to test data
**4. Estimators, Bias & Variance**
- **Estimator**: Function that estimates model parameters from data
- **Bias**: Error from overly simplistic model assumptions
- **Variance**: Sensitivity of predictions to changes in training data
- **Trade-off**: High bias = underfitting; high variance = overfitting
**5. Maximum Likelihood Estimation (MLE)**
- Finds parameters that maximize probability of observing the data
- Formula: $\theta^* = \arg\max P(\text{data}|\theta)$
- Widely used in statistical machine learning
**6. Bayesian Statistics**
- Uses prior beliefs $P(\theta)$ and updates with data likelihood $P(\text{data}|\theta)$
- Posterior: $P(\theta|\text{data}) \propto P(\text{data}|\theta) \times P(\theta)$
- Provides probabilistic framework for learning
**7. Supervised Learning Algorithms**
- Learn from labeled data (input-output pairs)
- Types: Regression (continuous output), Classification (discrete output)
- Examples: Linear regression, logistic regression, decision trees, SVM, neural networks
**8. Unsupervised Learning Algorithms**
- Learn patterns from unlabeled data
- Types: Clustering (k-means, hierarchical), dimensionality reduction (PCA)
- Finds hidden structure without guidance
**9. Stochastic Gradient Descent (SGD)**
- Optimization technique for training models
- Updates parameters using small random batches instead of full dataset
- Advantages: Memory efficient, faster convergence, online learning capability
**10. Building ML Algorithms**
- Steps: Data collection $\to$ Preprocessing $\to$ Feature engineering $\to$ Model selection $\to$
Training $\to$ Evaluation $\to$ Hyperparameter tuning
**11. Deep Learning Motivation**
- Challenges in shallow models: Manual feature engineering, poor scaling to high-dimensional data
- Deep learning learns features automatically through multiple layers
**12. Deep Feedforward Networks**
- Neural networks with input $\to$ hidden layers $\to$ output (one-directional flow)
- Each layer transforms input via weight matrices and activation functions
- Suitable for regression and classification
**13. Learning XOR Problem**
- Simple problem that reveals limitations of linear models (single perceptron)
- Requires hidden layer with non-linearity to solve
- Demonstrates power of deep networks
**14. Gradient-Based Learning**
- Minimizes loss function $L$ by computing $\nabla L$ with respect to parameters
- Updates: $\theta \gets \theta - \alpha\nabla L$ (where $\alpha$ = learning rate)
- Foundation of backpropagation
**15. Hidden Units & Architecture**
- **Hidden units**: Neurons in intermediate layers that learn internal representations
- **Architecture**: Network structure (depth, width, activation functions)
- Design choices significantly impact model capacity and performance
**16. Backpropagation & Differentiation**
- Efficient algorithm for computing gradients via chain rule
- Computes $\frac{\partial L}{\partial w}$ for all parameters by moving backward through network
- Enables training of deep networks in reasonable time
---
### QUESTION & ANSWER FORMAT
**Q1: What is the difference between overfitting and underfitting?**
A:
- **Overfitting**: Model learns training data too well, including noise. High training accuracy but poor
test performance. Caused by excessive model complexity.
- **Underfitting**: Model is too simple to capture data patterns. Poor performance on both training
and test sets. Caused by insufficient model complexity or inadequate training.
**Q2: Explain the bias-variance tradeoff.**
A:
- **Bias**: Error from model assumptions. High bias $\to$ underfitting
- **Variance**: Sensitivity to training data changes. High variance $\to$ overfitting
- Total error = Bias$^2$ + Variance + Irreducible Error
- Must balance: Reduce both but one typically increases when decreasing the other
**Q3: What is the purpose of a validation set?**
A:
- Separate dataset independent of training and test sets
- Used to tune hyperparameters without contaminating test data
- Provides unbiased performance estimates during training
- Prevents information leakage to final evaluation
**Q4: How does Stochastic Gradient Descent differ from Batch Gradient Descent?**
A:
- **SGD**: Updates parameters using one or small batch of samples per iteration
- **BGD**: Updates using entire training dataset
- SGD advantages: Faster convergence, memory efficient, handles large datasets, better
generalization
- SGD disadvantage: Noisier updates, may not converge smoothly
**Q5: What problem does the XOR example highlight?**
A:
- XOR problem is not linearly separable
- Single perceptron (linear model) cannot solve it
- Requires non-linear decision boundary
- Demonstrates need for hidden layers in neural networks
**Q6: Define Maximum Likelihood Estimation.**
A:
- Statistical method to find parameters that maximize probability of observed data
- Finds: $\theta^* = \arg\max P(\text{data}|\theta)$
- Intuition: Choose parameters most likely to produce the observed data
- Widely used in probabilistic models
**Q7: What is backpropagation and why is it important?**
A:
- Algorithm that computes gradients efficiently using chain rule
- Moves backward through network layers computing $\frac{\partial L}{\partial w}$
- Enables training of deep networks by solving vanishing gradient problem partially
- Critical for modern deep learning
---
### EXAM-STYLE ANSWERS (5-10 MARKS)
**Question: Explain the machine learning pipeline with focus on overfitting and underfitting
problems. (10 marks)**
**Answer:**
The machine learning pipeline consists of systematic steps to build predictive models:
1. **Data Collection & Preprocessing**: Gather raw data and handle missing values, normalization
2. **Feature Engineering**: Extract relevant features for model learning
3. **Model Selection**: Choose appropriate algorithm (linear models, trees, neural networks)
4. **Training**: Fit model to training data using optimization
5. **Evaluation & Tuning**: Test on validation/test sets, adjust hyperparameters
**Overfitting & Underfitting Problems:**
**Overfitting** occurs when model learns training data patterns including noise:
- Causes: Excessive model complexity, too many parameters
- Symptoms: High training accuracy, low test accuracy
- Solutions: Regularization (L1/L2), early stopping, dropout, more training data
**Underfitting** occurs when model is too simple:
- Causes: Insufficient model complexity, inadequate training
- Symptoms: Poor performance on both training and test sets
- Solutions: Increase model complexity, add more features, train longer
**Preventing Both**: Use validation set to monitor performance and select hyperparameters that
balance bias-variance tradeoff, ensuring good generalization.
---
**Question: What is Stochastic Gradient Descent? Discuss its advantages over Batch Gradient
Descent. (10 marks)**
**Answer:**
**Stochastic Gradient Descent (SGD)** is an optimization algorithm that updates model parameters
using gradients computed on small random subsets (batches) of training data, rather than the entire
dataset.
**Update Rule**: $\theta \gets \theta - \alpha\nabla L(\theta; x_i, y_i)$
where $\alpha$ is learning rate, and $(x_i, y_i)$ is a single or small batch sample.
**Advantages of SGD over Batch Gradient Descent (BGD):**
1. **Memory Efficiency**: Processes one/few samples at a time; doesn't require entire dataset in
memory
2. **Computational Speed**: Each iteration faster; converges in fewer iterations for large datasets
3. **Escape Local Minima**: Noise in updates helps escape shallow local minima
4. **Online Learning**: Can continuously learn from streaming data without retraining
5. **Better Generalization**: Noise acts as regularization, reducing overfitting
6. **Scalability**: Handles massive datasets BGD cannot process
**Disadvantages**: Noisy updates lead to unstable convergence; requires careful learning rate
scheduling.
**Variants**: Mini-batch SGD balances BGD and SGD by using small batches (32-256 samples).
---
\newpage
**Question: Explain deep feedforward networks and the XOR problem. Why is XOR important in
deep learning? (10 marks)**
**Answer:**
**Deep Feedforward Networks (DFNs)**
DFNs are multilayer neural networks with unidirectional data flow:
- Input Layer $\to$ Hidden Layers (1 or more) $\to$ Output Layer
- Each neuron computes: $h = \sigma(Wx + b)$ where $\sigma$ is activation function
- Learn complex non-linear functions through composition of simpler functions
**Architecture**:
- Multiple hidden layers increase model capacity
- Activation functions (ReLU, sigmoid, tanh) introduce non-linearity
- Output layer configured for task (regression: linear, classification: softmax)
**The XOR Problem**
XOR (exclusive OR) is a simple Boolean function:
- Input: Two binary values (0,0), (0,1), (1,0), (1,1)
- Output: 1 if inputs differ, 0 if same
**Why XOR is Important:**
1. **Not Linearly Separable**: Cannot draw single straight line separating classes (output=1 vs
output=0)
2. **Exposes Single Perceptron Limitation**: A linear model cannot solve it
3. **Motivates Hidden Layers**: Requires at least one hidden layer with non-linear activation
4. **Demonstrates Deep Learning Necessity**: Shows why neural networks need depth for
non-trivial problems
**Solution**:
- Hidden layer with 2+ neurons learns representations separating classes
- Output layer combines hidden representations to solve XOR
- Proves: depth + non-linearity = increased model expressiveness
This historically motivated shift from shallow to deep learning models.
---
**Question: Explain bias and variance tradeoff with an example. (5 marks)**
**Answer:**
**Bias-Variance Tradeoff**
Total prediction error decomposes as:
$$\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}$$
**Bias**: Error from oversimplified model assumptions
- High bias $\to$ underfitting $\to$ poor training performance
- Example: Fitting linear model to curved data
**Variance**: Sensitivity of predictions to training data fluctuations
- High variance $\to$ overfitting $\to$ high test error despite good training accuracy
- Example: High-degree polynomial overfits training points
**Tradeoff**: Improving one typically worsens the other
- Simple models: Low variance, high bias
- Complex models: Low bias, high variance
- Optimal model: Balanced bias and variance for minimum test error
**Example**: Predicting house prices
- **Linear model** (high bias, low variance): Simple but misses patterns
- **High-degree polynomial** (low bias, high variance): Fits training data perfectly but fails on new
data
- **Optimal model**: Moderate complexity capturing main patterns without overfitting
---
**Question: What is backpropagation? How does it work? (5 marks)**
**Answer:**
**Backpropagation** is an efficient algorithm for computing gradients in neural networks using the
chain rule, enabling training of deep networks.
**How It Works**:
1. **Forward Pass**: Compute predictions and loss $L$ through all layers
2. **Backward Pass**: Compute gradients by moving backward:
- $\frac{\partial L}{\partial w_{\text{output}}} = \frac{\partial L}{\partial \hat{y}} \times \frac{\partial
\hat{y}}{\partial w_{\text{output}}}$
- $\frac{\partial L}{\partial w_{\text{hidden}}} = \frac{\partial L}{\partial \hat{y}} \times \frac{\partial
\hat{y}}{\partial h} \times \frac{\partial h}{\partial w_{\text{hidden}}}$
- Continue recursively through all layers
3. **Parameter Update**: $\theta \gets \theta - \alpha\nabla L$ (where $\alpha$ = learning rate)
**Why Important**:
- Computes gradients in $O(n)$ operations instead of $O(n^2)$
- Enables training of deep networks (20+ layers)
- Foundation of modern deep learning
---
\newpage
## UNIT III: CONVOLUTIONAL NEURAL NETWORKS (CNNs)
### SHORT NOTES
**1. Convolution Operation**
- Core operation that applies filters to extract local features
- Filter (kernel) slides across input, computing dot product at each position
- Output: Feature map showing filter activation strength
**2. Motivation for CNNs**
- Exploit spatial structure in images: objects have local parts
- Parameter sharing: same filter detects features everywhere
- Translation invariance: same features recognized in different locations
- Much fewer parameters than fully connected networks
**3. Pooling**
- Reduces spatial dimensions of feature maps
- Max pooling: Takes maximum value in window
- Average pooling: Computes average in window
- Benefits: Reduces computation, adds robustness, provides translation invariance
**4. Convolution & Pooling as Strong Priors**
- Encode assumptions about data structure
- Convolution: Assumes local connectivity and weight sharing
- Pooling: Assumes local region aggregation is sufficient
- These assumptions reduce effective model capacity, aiding generalization
**5. Variants of Convolution**
- **1×1 Convolution**: Changes channel dimensions without spatial operations
- **Dilated Convolution**: Increases receptive field without more parameters
- **Transposed Convolution**: Upsamples feature maps (for segmentation, generation)
- **Depthwise Separable**: Separates spatial and channel operations for efficiency
**6. Structured Outputs**
- CNNs produce various output types: Classification (class probabilities), Detection (bounding
boxes), Segmentation (pixel-wise labels)
- Different output layers configured for tasks
**7. Data Types CNNs Handle**
- Images (2D convolutions)
- Video (3D convolutions)
- Time series (1D convolutions)
- Graphs (graph convolutions)
**8. Efficient Convolution Algorithms**
- **FFT-based Convolution**: Uses Fast Fourier Transform for large filters
- **Winograd Convolution**: Reduces arithmetic operations
- **Grouped Convolution**: Processes groups of channels separately
- **Implementation tricks**: GPU acceleration, memory optimization
**9. Random or Unsupervised Features**
- When labeled data is limited, use pre-trained models (transfer learning)
- Random filters sometimes work surprisingly well as initialization
- Unsupervised learning (autoencoders) learns useful representations
---
### QUESTION & ANSWER FORMAT
**Q1: What is the convolution operation and how does it differ from fully connected layers?**
A:
Convolution applies a small filter across input, computing dot products locally. Key differences from
fully connected:
- **Sparse connectivity**: Each output connects to small input window (not all inputs)
- **Weight sharing**: Same filter coefficients used everywhere
- **Equivariance to translation**: Recognizes features in different positions
- Result: Far fewer parameters, better spatial reasoning, translation invariance
**Q2: Why are pooling layers important in CNNs?**
A:
- Reduces spatial dimensions, cutting computation by 75% (2×2 pooling)
- Provides translation invariance: slight spatial shifts don't change output
- Helps prevent overfitting by discarding fine positional details
- Enables hierarchical feature learning: combine local features into higher-level concepts
- Common: Max pooling (robust) or average pooling (smooth)
**Q3: What are variants of convolution and when is each used?**
A:
- **1×1 Convolution**: Reduce/increase channels, combine feature maps efficiently
- **Dilated Convolution**: Increase receptive field exponentially without more parameters; used in
semantic segmentation
- **Transposed Convolution**: Upsample feature maps; used in image generation and
segmentation
- **Depthwise Separable**: Efficient convolution factoring spatial and channel dimensions; used in
mobile networks
**Q4: Explain the concept of "convolution and pooling as infinitely strong prior".**
A:
These operations encode strong assumptions about data:
- **Prior assumption**: Important information is local; global connections less important
- **Consequence**: Dramatically reduces model flexibility and parameters
- **Benefit**: Acts as regularization, improving generalization to unseen data
- **Trade-off**: If task requires global reasoning, this prior can limit performance
- **Result**: Models learn more efficiently with less data, achieving better generalization
---
### EXAM-STYLE ANSWERS
**Question: Explain convolution operation in CNNs. How does it differ from fully connected layers?
What are its advantages? (10 marks)**
**Answer:**
**Convolution Operation**
Convolution is the fundamental operation in CNNs. A filter (kernel) of size $F \times F$ slides
across input image, computing element-wise products and summing results at each position:
$$\text{Output}[i,j] = \sum_a \sum_b \text{Filter}[a,b] \times \text{Input}[i+a, j+b]$$
The filter learns to detect features like edges, textures, shapes.
**Advantages of Convolution**
1. **Parameter Efficiency**: Weight sharing reduces parameters dramatically (100× fewer for
images)
2. **Translation Invariance**: Same feature detected regardless of position
3. **Local Connectivity**: Exploits spatial locality; objects composed of local parts
4. **Hierarchical Feature Learning**: Early layers learn simple features (edges), deeper layers learn
complex patterns (faces)
5. **Computational Efficiency**: Parallelizable, fast on GPUs
6. **Better Generalization**: Fewer parameters reduce overfitting risk
**Why CNNs Work for Vision**: Images have inherent spatial structure; convolution operations
respect and exploit this structure.
---
\newpage
**Question: What is pooling? Explain max pooling and average pooling with their advantages and
disadvantages. (10 marks)**
**Answer:**
**Pooling Operation**
Pooling reduces spatial dimensions by aggregating information from regions. Applies
non-overlapping window (typically 2×2) across feature map.
**Max Pooling**: Takes maximum value in each window
- Formula: $\text{Output} = \max(\text{Input window})$
- Advantages: Robust to small translations, captures strongest features, efficient
- Disadvantages: Loses spatial precision, discards weak signals
**Average Pooling**: Computes average value in each window
- Formula: $\text{Output} = \text{mean}(\text{Input window})$
- Advantages: Considers all values, smooth output
- Disadvantages: Blurs features, less selective than max pooling
**Advantages of Pooling**
1. **Dimensionality Reduction**: 2×2 pooling reduces spatial size 4×, computations 16×
2. **Translation Invariance**: Features recognized despite small spatial shifts
3. **Overfitting Prevention**: Reduces parameters and model complexity
4. **Computational Efficiency**: Speeds up training and inference
5. **Receptive Field Growth**: Enables large receptive fields without many layers
6. **Hierarchical Abstraction**: Combines local features into higher-level representations
**Disadvantages**
- Information loss: Discards fine positional details
- May harm tasks requiring precise localization (segmentation)
---
**Question: Explain the role of convolution and pooling as a strong prior in neural networks. (5
marks)**
**Answer:**
**Strong Prior Concept**
A prior is an assumption built into the model structure about how data is organized. Convolution and
pooling encode strong assumptions:
**Assumptions Made**:
1. **Locality**: Important information is local; distant pixels rarely interact
2. **Translation Invariance**: Features important regardless of position
3. **Hierarchical Composition**: Complex patterns built from simpler local patterns
**Benefits of This Prior**:
- Dramatically reduces model complexity (from billions to millions of parameters)
- Acts as regularization: constrains hypothesis space to reasonable possibilities
- Improves generalization: learns better with less data
- Faster learning: model searches reduced hypothesis space
**Trade-offs**:
- If task violates assumptions (requires global reasoning), prior limits performance
- Not ideal for tasks like global image understanding without context
- But excellent for vision tasks where assumptions mostly hold
**Result**: CNNs achieve exceptional performance on image tasks with far fewer parameters than
fully connected networks, demonstrating that good priors enable efficient learning.
---
\newpage
## UNIT IV: RECURRENT AND RECURSIVE NEURAL NETWORKS
### SHORT NOTES
**1. Recurrent Neural Networks (RNNs)**
- Process sequential data by maintaining hidden state $h_t$
- Update rule: $h_t = \sigma(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$
- Output: $y_t = W_{hy} h_t + b_y$
- Can represent arbitrary sequences given sufficient hidden units
**2. Bidirectional RNNs**
- Process sequence both forward and backward
- Captures both past (forward) and future (backward) context
- Final representation: concatenate forward and backward hidden states
- Useful for tasks like named entity recognition, pos tagging
**3. Encoder-Decoder & Seq2Seq**
- Encoder: RNN reads entire input sequence, produces context vector
- Decoder: RNN generates output sequence conditioned on context
- Applications: Machine translation, summarization, chatbots
- Handles variable-length input-output pairs
**4. Challenge of Long-Term Dependencies**
- Problem: Gradients vanish/explode during backprop through time
- RNNs cannot learn correlations across long time gaps
- Error signal from distant timesteps becomes negligible
- Limits ability to capture long-range dependencies
**5. LSTM & Gated RNNs**
- LSTM: Long Short-Term Memory with forget gate, input gate, output gate
- Solves vanishing gradient problem through constant error carousel
- Forget gate: controls what to forget from cell state
- Input gate: controls what new information to add
- Output gate: controls what to expose
- GRU: Simplified LSTM with reset and update gates
**6. Optimization for Long-Term Dependencies**
- Gradient clipping: Clip gradients to prevent explosion
- Careful initialization: Orthogonal or identity matrix initialization
- Regularization: Weight noise, dropout
- Architecture: LSTMs/GRUs, residual connections
- Training tricks: Layer normalization, careful learning rate scheduling
---
### EXAM-STYLE ANSWERS
**Question: Explain RNNs and discuss the vanishing gradient problem. How do LSTMs address
this problem? (10 marks)**
**Answer:**
**Recurrent Neural Networks (RNNs)**
RNNs process sequential data by maintaining hidden state that summarizes past information:
$$h_t = \sigma(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$$
$$y_t = W_{hy} h_t + b_y$$
Where $W_{hh}$ weights information from previous timestep. This recurrent connection allows
RNNs to represent sequences of any length, making them ideal for time series, language, speech.
**Vanishing Gradient Problem**
During backpropagation through time (BPTT), gradients are multiplied at each timestep:
$$\frac{\partial L}{\partial h_0} = \frac{\partial L}{\partial h_T} \times \frac{\partial h_T}{\partial
h_{T-1}} \times \cdots \times \frac{\partial h_1}{\partial h_0}$$
Each Jacobian $\frac{\partial h_t}{\partial h_{t-1}}$ has eigenvalues typically $< 1$. Multiplying 100+
such terms causes gradients to vanish ($\to 0$).
Consequence: Model cannot learn dependencies across many timesteps; cannot capture
long-range patterns.
**LSTM Solution**
LSTMs introduce cell state $c_t$ with three gating mechanisms:
1. **Forget Gate**: $f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$ decides fraction of previous cell
state to retain
2. **Input Gate**: $i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$ decides fraction of new information to
add
3. **Cell State Update**: $c_t = f_t \odot c_{t-1} + i_t \odot \tanh(W_c \cdot [h_{t-1}, x_t] + b_c)$
4. **Output Gate**: $o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$ decides fraction of cell state to
expose
**Key Insight**: Cell state gradient $\frac{\partial c_T}{\partial c_0} = \prod f_t$, where $f_t$ are
forget gate outputs (typically 0.5-0.7, not $< 0.1$). Gradients don't vanish; LSTMs learn to control
what information flows through cell state.
Result: Can learn dependencies across 100+ timesteps effectively.
---
\newpage
**Question: Explain encoder-decoder architecture and seq2seq models. Provide an example. (10
marks)**
**Answer:**
**Encoder-Decoder Architecture**
Two-part neural network for sequence-to-sequence tasks:
**Encoder Phase**:
1. Takes input sequence $x = (x_1, x_2, \ldots, x_m)$
2. Processes through RNN/LSTM: $h_t = \text{RNN}_{\text{enc}}(x_t, h_{t-1})$
3. Produces context vector $c = h_m$ (final hidden state)
4. Context summarizes entire input sequence
**Decoder Phase**:
1. Takes context vector $c$ as initial hidden state
2. Generates output sequence $y = (y_1, y_2, \ldots, y_n)$ left-to-right
3. At each step: $h_t = \text{RNN}_{\text{dec}}(y_{t-1}, h_{t-1})$, $y_t = \text{softmax}(W \cdot h_t)$
4. Predicts next token based on previous tokens and context
**Training**: Given pairs $(x, y)$, maximize $P(y|x) = \prod P(y_t|y_1 \ldots y_{t-1}, c)$
**Key Advantage**: Handles variable-length input-output without alignment information.
**Example: Machine Translation**
Input (English): "Hello, how are you?"
$\to$ [Hello, comma, how, are, you, question-mark]
Encoder: LSTM reads sequence, produces context vector of dimension 256
Output (French): Decoder generates [Bonjour, comment, allez-vous, question-mark]
**Applications**:
- Machine translation (most common)
- Image captioning: CNN encodes image $\to$ RNN decoder generates caption
- Abstractive summarization: Encode document $\to$ Decode summary
- Conversational AI: Encode user query $\to$ Decode response
**Modern Enhancement**: Attention mechanism allows decoder to focus on relevant input parts
rather than using fixed context vector.
---
**Question: What is LSTM? How does it address the limitations of standard RNNs? (5 marks)**
**Answer:**
**LSTM (Long Short-Term Memory)**
LSTM is an RNN variant with special gating mechanisms solving vanishing gradient problem:
**Key Components**:
1. **Forget Gate**: Controls what to forget from previous cell state
2. **Input Gate**: Controls what new information to add
3. **Cell State**: Flows with approximately constant gradient
4. **Output Gate**: Controls hidden state exposure
**Mechanism**: Instead of simple recurrence, LSTM uses multiplicative gating:
- $c_t = f_t \odot c_{t-1} + i_t \odot \text{new\_info}$
- $h_t = o_t \odot \tanh(c_t)$
**Advantages Over Standard RNNs**:
1. **Solves Vanishing Gradients**: Cell state gradient $\approx$ constant (≈ 0.5-0.7 per step), not
exponentially small
2. **Long-Range Dependencies**: Can learn across 100+ timesteps
3. **Selective Memory**: Gates allow selective information flow
4. **Flexibility**: Works for diverse sequence tasks
**Result**: LSTMs became standard for sequential tasks, enabling practical deep learning on
sequences.
---
\newpage
## UNIT V: PRACTICAL METHODOLOGY
### SHORT NOTES
**1. Performance Metrics**
- **Accuracy**: $\frac{\text{TP} + \text{TN}}{\text{Total}}$; works for balanced datasets
- **Precision**: $\frac{\text{TP}}{\text{TP} + \text{FP}}$; important when false positives costly
- **Recall**: $\frac{\text{TP}}{\text{TP} + \text{FN}}$; important when false negatives costly
- **F1-Score**: Harmonic mean of precision-recall; single metric combining both
- **ROC-AUC**: Plots true positive rate vs false positive rate; threshold-independent
- **Confusion Matrix**: Shows TP, TN, FP, FN
**2. Default Baseline Models**
- Simple models to establish performance floor
- Ensure complex model improvements are meaningful
- Examples: Random classifier, majority class prediction, simple linear model
- Baseline performance indicates problem difficulty
**3. Determining Whether to Gather More Data**
- Plot learning curves: training and validation error vs. data size
- If curves diverge (underfitting): gather more data helps
- If curves converge (overfitting): more data helps, but diminishing returns
- If validation error plateaus: data volume sufficient, focus on model/features
**4. Selecting Hyperparameters**
- **Grid Search**: Try all combinations of hyperparameter values
- **Random Search**: Random sampling of hyperparameter space (faster than grid)
- **Bayesian Optimization**: Model hyperparameter-performance relationship, suggest promising
values
- **Cross-validation**: Use k-fold validation to estimate hyperparameter performance
**5. Debugging Strategies**
- Verify data pipeline: Check data loading, preprocessing, normalization
- Overfit on small batch: Model should perfectly memorize small data
- Monitor loss curves: Training loss should decrease, validate that optimization works
- Gradient analysis: Check gradients aren't vanishing/exploding
- Progressive unfreezing: Gradually increase model complexity
- Comparative analysis: Compare variants systematically
**6. Multi-Digit Number Recognition Example**
- Task: Recognize multiple digit sequence from image (like SVHN dataset)
- Approach: CNN for feature extraction + RNN for sequence prediction
- Architecture: CNN extracts spatial features; LSTM predicts digit sequence
- Challenges: Variable digit count, overlapping digits, image quality variations
- Solutions: Data augmentation, ensemble methods, explicit position classifiers
---
### EXAM-STYLE ANSWERS
**Question: Explain performance metrics for classification. Which metrics are appropriate for
imbalanced datasets? (10 marks)**
**Answer:**
**Classification Performance Metrics**
Metrics evaluate how well model predicts correct classes. Based on confusion matrix:
- **TP (True Positive)**: Correctly predicted positive
- **TN (True Negative)**: Correctly predicted negative
- **FP (False Positive)**: Incorrectly predicted positive
- **FN (False Negative)**: Incorrectly predicted negative
**Common Metrics**:
1. **Accuracy = $\frac{\text{TP} + \text{TN}}{\text{Total}}$**
- Overall correctness
- Misleading on imbalanced data
2. **Precision = $\frac{\text{TP}}{\text{TP} + \text{FP}}$**
- Of predicted positives, how many correct?
- Important when false positives costly
3. **Recall = $\frac{\text{TP}}{\text{TP} + \text{FN}}$**
- Of actual positives, how many found?
- Important when false negatives costly
4. **F1-Score = $\frac{2 \times (\text{Precision} \times \text{Recall})}{\text{Precision} +
\text{Recall}}$**
- Harmonic mean; balances precision-recall
5. **ROC-AUC**: Plots TPR vs FPR at various thresholds
- Threshold-independent
- AUC $\in [0,1]$; 0.5 = random, 1.0 = perfect
**For Imbalanced Datasets** (e.g., 95% negative, 5% positive):
Accuracy misleads: 95% accuracy possible by predicting all negative.
**Appropriate Metrics**:
- **Precision, Recall, F1**: Account for class imbalance
- **ROC-AUC**: Not affected by class distribution
**Conclusion**: Choose metrics aligned with problem costs. For imbalanced data, avoid accuracy;
use Precision, Recall, F1-Score, or ROC-AUC.
---
\newpage
**Question: Explain learning curves and how to determine if gathering more data helps. (10
marks)**
**Answer:**
**Learning Curves**
Graph showing training and validation error vs. training set size, revealing whether model suffers
from bias (underfitting) or variance (overfitting) problems.
**Curve Interpretation**
**Case 1: High Bias (Underfitting)**
- Both curves high and close together
- Training error $\approx$ validation error $\approx$ high value
- Curves flat regardless of data size
- **Solution**: Increase model complexity, train longer, better features
**Case 2: High Variance (Overfitting)**
- Training error low, validation error high
- Large gap between curves
- Validation curve still decreasing with more data
- **Solution**: Gather more data, add regularization, simpler model
**Case 3: Optimal**
- Curves converge at reasonable error level
- Small gap between training and validation
- Curves plateau early
- **Solution**: Focus on other improvements (better features, hyperparameters)
**Practical Value**: Learning curves prevent wasted effort on wrong improvements.
---
**Question: Describe hyperparameter tuning methods and practical guidelines. (10 marks)**
**Answer:**
**Hyperparameters to Tune**
1. **Learning Rate**: Controls step size; typical range: 0.001 to 0.1
2. **Batch Size**: Typical: 32, 64, 128, 256
3. **Number of Epochs**: Typical: 10-100; use early stopping
4. **Network Architecture**: Depth and width choices
5. **Regularization**: L1/L2, dropout strength
6. **Activation Functions**: ReLU, tanh, sigmoid
**Tuning Methods**
**1. Grid Search**
- Exhaustively try all combinations
- Advantages: Comprehensive, finds global optimum in search space
- Disadvantages: Exponentially expensive; impractical for many hyperparameters
- Use: Few hyperparameters, narrow ranges
**2. Random Search**
- Sample random combinations from hyperparameter space
- Advantage: Faster exploration
- Disadvantage: May require more trials
- Use: Many hyperparameters or large ranges
**3. Bayesian Optimization**
- Model hyperparameter-performance relationship as Gaussian process
- Balances exploration and exploitation
- Advantage: Sample-efficient
- Use: Expensive to evaluate; many hyperparameters
**4. Cross-Validation**
- k-fold cross-validation prevents overfitting to single validation split
- Provides robust performance estimates
**Practical Guidelines**
1. Start simple with default values
2. Coarse-to-fine: broad search first, then refine
3. Plot validation curves for each hyperparameter
4. Use random search or Bayesian optimization for many hyperparameters
5. Monitor training loss curves
6. Ensemble multiple models with different hyperparameters
---
\newpage
**Question: Propose a solution for multi-digit number recognition from images. (10 marks)**
**Answer:**
**Multi-Digit Number Recognition Task**
Objective: Read variable-length sequences of digits from images (license plates, house numbers)
**Proposed Architecture**
Two-stage approach combining CNNs and RNNs:
**Stage 1: Feature Extraction (CNN)**
- Input: Image (64×64×3 RGB)
- Architecture:
- Conv(32 filters) $\to$ ReLU $\to$ MaxPool
- Conv(64 filters) $\to$ ReLU $\to$ MaxPool
- Conv(128 filters) $\to$ ReLU $\to$ MaxPool
- Flatten $\to$ FC(1024)
- Output: Feature vector (1024-dim)
**Stage 2: Sequence Prediction (RNN)**
- Input: Feature vector from CNN
- Architecture:
- LSTM(128 units, bidirectional)
- Output: Dense(10) for 10 digit classes
- Output: Sequence of digits with confidences
**Challenges & Solutions**
| Challenge | Solution |
|-----------|----------|
| Variable digit count | RNN learns to predict STOP token |
| Overlapping/touching digits | Larger CNN receptive field |
| Image quality variations | Data augmentation |
| Small dataset | Transfer learning from ImageNet |
| Computational cost | MobileNet backbone |
| Imbalanced digits | Class weighting, over-sampling |
**Training Procedure**
1. Data preparation and normalization
2. Data augmentation: rotation, brightness, contrast, translation
3. Loss: Categorical cross-entropy for each position
4. Optimizer: Adam, learning rate 0.001, batch size 32
5. Early stopping on validation accuracy
**Evaluation Metrics**
- Sequence accuracy: All digits correct
- Per-digit accuracy: Individual position accuracy
- Character error rate: Fraction of incorrect digits
**Advanced Improvements**
- Attention mechanism for decoder
- Ensemble multiple models
- Test-time augmentation
---
## SUMMARY TABLE: KEY CONCEPTS
| Unit | Key Concepts | Applications |
|------|--------------|--------------|
| I | SGD, bias-variance, backprop | All supervised learning |
| III | Convolution, pooling, efficiency | Image classification, detection, segmentation |
| IV | RNN, LSTM, encoder-decoder | Translation, time series, speech |
| V | Metrics, hyperparameter tuning | Model evaluation, optimization |
---
## EXAM TIPS
1. **Memory jogs**: Use diagrams (backprop chain, CNN layers, LSTM gates, learning curves)
2. **Practical examples**: Reference real applications
3. **Math notation**: Write equations clearly with variable definitions
4. **Comparisons**: Contrast approaches explicitly
5. **Trade-offs**: Discuss costs and benefits
6. **Depth**: 5-mark answers: Main idea + 2-3 points; 10-mark: Comprehensive with examples
---
\newpage
Good luck with your exams!