Deep Learning Exam Study Guide
1. Introduction to Deep Learning
**Definition:** Subset of machine learning based on artificial neural networks that learn
representations from data automatically.
**Key Points:**
- Learns hierarchical representations.
- Involves layers of neurons (input, hidden, output).
**Example:** Image classification with digits (MNIST dataset).
**Diagram:**
Input -> Hidden Layer 1 -> Hidden Layer 2 -> Output
**Applications:**
- Computer Vision (Face Recognition)
- NLP (Translation)
- Speech Recognition
2. Bayesian Learning
**Definition:** Approach where predictions are made using probabilities and updated using
Bayes' Theorem.
**Bayes' Theorem:**
P(H|D) = [P(D|H) * P(H)] / P(D)
**Example:** Spam email classification updating as new messages come in.
**Usage:**
- Uncertainty Estimation
- Active Learning in ML
**Numerical Example:**
P(Spam|Word) = (P(Word|Spam) * P(Spam)) / P(Word)
3. Linear Classifiers and SVMs
**Linear Classifier:** Classifies data by learning a linear decision boundary.
**SVM (Support Vector Machine):** Maximizes margin between classes.
**Diagram:**
| o |
| | <--- Hyperplane (Margin)
| x |
**Example:** Binary classification of emails using a hyperplane.
4. Optimization Techniques
**Key Methods:**
- Gradient Descent
- Stochastic Gradient Descent
- Mini-Batch Gradient Descent
**Formula:**
w = w - alpha * grad(loss)
**Numerical Example:**
Loss = (y - y')^2
Gradient = 2*(y - y') * x
5. Neural Networks and Backpropagation
**Neural Networks:** Composed of layers with weights and activation functions.
**Backpropagation:** Updates weights using gradients computed via chain rule.
**Example:** XOR problem solved using 2-layer neural net.
**Diagram:**
Input -> [Weights] -> Hidden Layer -> Output
6. Unsupervised Learning and Autoencoders
**Unsupervised Learning:** Model learns from unlabeled data.
**Autoencoders:** Learn to encode input into compressed representation and decode it
back.
**Use Case:** Noise removal from images.
**Structure:**
Encoder -> Bottleneck -> Decoder
7. Convolutional Neural Networks (CNNs)
**Definition:** Designed for processing grid-like data (e.g., images).
**Layers:**
- Convolution
- Activation (ReLU)
- Pooling
- Fully Connected
**Diagram:**
Image -> Conv -> Pool -> FC -> Output
**Applications:** Image Classification, Object Detection
8. Advanced Optimization Methods
- **Momentum:** Adds velocity term to accelerate descent.
- **RMSProp:** Adjusts learning rate per parameter.
- **Adam:** Combines Momentum + RMSProp.
**Numerical Example:**
Adam with beta1 = 0.9, beta2 = 0.999 for weight updates.
9. Regularization and Normalization
**Regularization:** Prevents overfitting.
- **Dropout:** Randomly drops neurons.
- **L2:** Penalizes large weights.
**Normalization:**
- **Batch Norm:** Normalizes input of each layer.
**Use Case:** Faster convergence, stable training.
10. Advanced Deep Architectures
- **ResNet:** Uses skip connections to avoid vanishing gradient.
- **Inception:** Combines multiple filter sizes in parallel.
**Diagram:**
Input -> [1x1, 3x3, 5x5] Convs -> Concatenate -> Output
11. Computer Vision Applications
**Denoising:** Autoencoders, CNNs
**Segmentation:** U-Net, Mask-RCNN
**Object Detection:** YOLO, SSD, RCNN
**Example:** Detect faces in real-time using YOLOv5.
12. RNNs, LSTM, GRU
**RNN:** Suitable for sequence data but suffers vanishing gradient.
**LSTM:** Uses cell state, forget gate to retain memory.
**GRU:** Simplified version of LSTM.
**Example:** Predict next word in sentence using LSTM.
**Diagram:**
Input -> RNN/LSTM/GRU -> Output
Internal Tips
- Understand equations behind optimizers.
- Practice drawing block diagrams.
- Solve small numerical updates (gradient descent, Adam).
- Use mnemonics for remembering steps.
Flash Cards
**Q:** Role of Batch Norm?
**A:** Stabilizes training, accelerates convergence.
**Q:** Difference between GRU and LSTM?
**A:** GRU has fewer gates, simpler, faster.
**Q:** Dropout in testing phase?
**A:** Not used; full network used with scaled weights.
NPTEL Notes Links
[NPTEL Deep Learning - IIT
Ropar]([Link]
[NPTEL Deep Learning - IIT
KGP]([Link]