Page |1
1. Foundations (VERY IMPORTANT – Frequently Reused)
1.1 Perceptron
• Linear classifier
• Output:
y=sign(wTx+b)y = sign(w^Tx + b)y=sign(wTx+b)
Key Points:
• Cannot solve non-linear problems (XOR)
• Leads to multi-layer networks
1.2 Feedforward Neural Network (FNN)
Structure:
• Input → Hidden → Output
• Fully connected layers
Forward Pass:
al=σ(Wlal−1+bl)a^l = \sigma(W^l a^{l-1} + b^l)al=σ(Wlal−1+bl)
2.3 Backpropagation (HIGHLY IMPORTANT)
He explicitly derives this in class
Core Idea:
• Use chain rule
• Compute gradient layer by layer backward
Key Equations:
∂L∂Wl=δl(al−1)T\frac{\partial L}{\partial W^l} = \delta^l (a^{l-1})^T∂Wl∂L=δl(al−1)T
δl=(Wl+1)Tδl+1⋅σ′(zl)\delta^l = (W^{l+1})^T \delta^{l+1} \cdot \sigma'(z^l)δl=(Wl+1)Tδl+1⋅σ′(zl)
Exam Tip:
• Expect step-by-step derivation
• Understand:
o Why gradients flow backward
o Vanishing gradient issue
3. Training Concepts (Practical + Theory)
Dataset Handling
• Train vs Test (in-sample vs out-of-sample)
Overfitting
• Model performs well on training but poor on test
Techniques:
• Regularization
• More data
• Simpler model
4. Attention Mechanism (CORE TOPIC)
He builds intuition using sentence context
Idea:
• Not all words are equally important
• Model “focuses” on relevant words
Example:
Page |2
“Black cat sat…”
→ “black” influences “cat”
Attention Formula:
Attention(Q,K,V)=softmax(QKTdk)VAttention(Q,K,V) =
softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)VAttention(Q,K,V)=softmax(dkQKT)V
Components:
• Query (Q): current word
• Key (K): all words
• Value (V): representations
5. Self-Attention
Key Idea:
• Each word attends to all other words
Why important:
• Captures long-range dependencies
• Parallel computation (unlike RNN)
Masking (VERY IMPORTANT)
Strongly emphasized in class
Purpose:
• Prevent future word leakage
Example:
Predict next word → cannot see future words
6. Transformers
Architecture:
• Multi-head attention
• Feedforward layers
• Positional encoding
Multi-Head Attention
Idea:
• Multiple attention mechanisms in parallel
Output:
MultiHead=Concat(head1,...,headn)WOMultiHead = Concat(head_1, ...,
head_n)W^OMultiHead=Concat(head1,...,headn)WO
Benefit:
• Learns different relationships simultaneously
7. BERT vs GPT
BERT:
• Bidirectional
• Uses masking
• Task: understanding
GPT:
• Autoregressive
• Left → Right
• Task: generation
Page |3
8. Autoencoders
Structure:
• Encoder → Latent → Decoder
Objective:
min∣∣x−x^∣∣min ||x - \hat{x}||min∣∣x−x^∣∣
Variational Autoencoder (VAE)
Key Difference:
• Learns distribution, not fixed vector
Loss:
Loss=Reconstruction+KL divergenceLoss = Reconstruction + KL\
divergenceLoss=Reconstruction+KL divergence
9. RNN / LSTM (Sequential Models)
Introduced as sequence modeling tools
RNN:
• Maintains hidden state
• Problem: vanishing gradients
LSTM:
• Gates solve long-term dependency
10. Graph Neural Networks (GNN)
Mentioned as upcoming topic
Idea:
• Nodes + edges
• Message passing between nodes
11. Diffusion Models
Core Idea:
• Add noise → learn to remove noise
Used in:
• Image generation
12. Neural Network Compression
Motivation:
• Reduce model size
• Same performance
Techniques:
• Weight sharing
• Pruning
• Quantization
Attention
Definition:
A method that lets a model focus on the most important parts of the input instead of treating everything
equally.
Example:
In the sentence “The cat sat on the mat”, to understand “sat”, the model focuses more on “cat” than “mat”.
Page |4
Self-Attention
Definition:
A mechanism where each word in a sentence looks at all other words to understand its meaning.
Example:
In “The bank is near the river”, the word “bank” looks at “river” to understand it means riverbank, not a
financial bank.
Transformers
Definition:
A neural network architecture that uses self-attention to process all words in parallel instead of one by one.
Example:
Used in language translation where the entire sentence is processed at once instead of word-by-word.
Performers
Definition:
A faster version of transformers that approximates attention to reduce computation cost.
Example:
Used when handling very long text (like books) where normal transformers become too slow.
BERT
Definition:
A model that reads text in both directions (left and right) to understand context.
Example:
In “He went to the bank to deposit money”, BERT understands “bank” means a financial institution.
GPT
Definition:
A model that generates text by predicting the next word based on previous words.
Example:
Given “The sky is”, GPT can generate “blue”.
Autoencoders
Definition:
A model that learns to compress data and then reconstruct it back.
Example:
Compressing an image into a smaller representation and then recreating the same image.
Variational Autoencoders (VAE)
Definition:
An autoencoder that learns a distribution of data instead of fixed values, allowing generation of new data.
Example:
Generating new faces that look realistic but are not real people.
Deep Reinforcement Learning
Definition:
A method where an agent learns by trial and error using rewards and penalties.
Page |5
Example:
A game-playing AI learns to win by getting rewards for good moves and penalties for bad ones.
Graph Neural Networks (GNN)
Definition:
A model designed to work on graph data where items are connected.
Example:
Predicting friendships in a social network based on existing connections.
Diffusion Models
Definition:
Models that learn to generate data by gradually adding noise and then learning to remove it.
Example:
Creating realistic images by starting from random noise and refining it step by step.
Large Language Models (LLMs)
Definition:
Very large models trained on huge text data to understand and generate human-like language.
Example:
Chatbots that can answer questions, write essays, or generate code.
If you want, I can next convert this into:
• ultra-short revision notes (1–2 lines each)
• or expected exam questions based on these topics
1. What is Attention? Why is it needed?
Answer:
Attention allows a model to focus on important parts of input instead of treating all inputs equally. It helps
capture relevant context in sequences.
2. What is the difference between Attention and Self-Attention?
Answer:
Attention compares two different sequences (e.g., input and output).
Self-attention compares elements within the same sequence.
3. What are Query, Key, and Value in attention?
Answer:
Query: what we are looking for
Key: what we compare with
Value: actual information retrieved after matching
4. Why do we divide by √d in attention?
Answer:
To prevent very large values in dot product, which would make softmax unstable and gradients vanish.
Page |6
5. What is masking in transformers?
Answer:
Masking prevents the model from seeing future words during training, ensuring correct sequence prediction.
6. What is Multi-Head Attention?
Answer:
Multiple attention operations run in parallel to capture different types of relationships in data.
7. Why are transformers better than RNNs?
Answer:
Transformers process data in parallel and capture long-range dependencies better than RNNs.
8. What is positional encoding?
Answer:
A way to add order information to input since transformers do not process sequences sequentially.
9. What is BERT?
Answer:
A bidirectional transformer model that reads context from both left and right.
10. What is GPT?
Answer:
An autoregressive model that predicts the next word using previous words.
11. Difference between BERT and GPT?
Answer:
BERT: bidirectional, used for understanding
GPT: unidirectional, used for generation
12. What is an Autoencoder?
Answer:
A model that compresses input into a smaller representation and reconstructs it.
13. What is the bottleneck layer in autoencoder?
Answer:
The compressed representation that forces the model to learn important features.
14. What is a Variational Autoencoder (VAE)?
Answer:
An autoencoder that learns probability distributions and can generate new data.
15. What is KL divergence in VAE?
Answer:
A measure to ensure learned distribution is close to a normal distribution.
Page |7
16. What is Reinforcement Learning?
Answer:
Learning by interacting with an environment using rewards and penalties.
17. What is a policy in reinforcement learning?
Answer:
A strategy that tells the agent what action to take in a given state.
18. What is a Graph Neural Network (GNN)?
Answer:
A model that learns from graph-structured data using node connections.
19. What is a Diffusion Model?
Answer:
A model that generates data by learning to remove noise step-by-step.
20. What is Neural Network Compression? Why is it needed?
Answer:
Reducing model size while maintaining performance. Needed to save memory and computation.
1. Attention
Q: How does attention handle long sentences better?
Answer: It directly looks at all words and assigns importance, instead of remembering everything in one
vector.
Example: In a long sentence, it can still focus on the word “doctor” even if it appeared far earlier.
Q: What if all attention weights are equal?
Answer: Then the model treats all words equally and loses focus.
Example: Important word “not” in “not good” gets ignored → wrong meaning.
Q: Can attention work without softmax?
Answer: No, softmax converts scores into probabilities so they can be compared.
Example: Without softmax, weights may not sum to 1 → unclear importance.
2. Self-Attention
Q: Why is it computationally expensive?
Answer: Each word compares with every other word → O(n²).
Example: 100 words → 100 × 100 comparisons.
Q: Time complexity?
Answer: O(n²) with respect to sequence length.
Q: Can it capture order?
Answer: Not by itself, needs positional encoding.
Example: “dog bites man” vs “man bites dog” → same without position info.
3. Query, Key, Value
Q: Are Q, K, V learned?
Answer: Yes, they are learned through training.
Page |8
Q: Why three vectors?
Answer: To separate “search”, “matching”, and “information”.
Example: Query = question, Key = index, Value = answer.
Q: What if Q = K = V?
Answer: It still works but reduces flexibility.
Example: Less ability to learn different relationships.
4. Scaling factor (√d)
Q: What if we don’t scale?
Answer: Values become large → softmax becomes sharp.
Example: One word gets almost 100% attention.
Q: Effect on gradients?
Answer: Gradients become very small → slow learning.
Q: Always required?
Answer: Mostly yes in high dimensions.
5. Masking
Q: What if masking is not used?
Answer: Model sees future words → cheating during training.
Example: Predicting next word but already seeing it.
Q: Used in BERT or GPT?
Answer:
BERT → yes (masked words)
GPT → yes (future masking)
Q: Padding vs causal mask?
Answer:
Padding mask → ignores empty tokens
Causal mask → blocks future words
6. Multi-Head Attention
Q: Why not one large head?
Answer: Multiple heads learn different relationships.
Example: One head learns grammar, another learns meaning.
Q: How combined?
Answer: Concatenation + linear layer.
Q: Same learning in each head?
Answer: No, each learns different patterns.
7. Transformers vs RNN
Q: Drawback of transformers?
Answer: High memory and computation.
Q: When RNN useful?
Answer: Small data or low compute.
Example: Simple time-series prediction.
Q: Memory comparison?
Answer: Transformers use more memory than RNNs.
Page |9
8. Positional Encoding
Q: Why needed?
Answer: Transformers process all words together, no order awareness.
Q: If removed?
Answer: Sentence meaning breaks.
Example: “eat food” vs “food eat” look same.
Q: Learnable vs fixed?
Answer:
Fixed → predefined
Learnable → trained with data
9. BERT
Q: Why not good for generation?
Answer: It sees both sides, so not suited for predicting next word.
Q: Masked language modeling?
Answer: Some words are hidden and model predicts them.
Q: Input format?
Answer: Sentence with special tokens like [CLS], [SEP]
10. GPT
Q: Why unidirectional?
Answer: To predict next word step by step.
Q: Autoregressive?
Answer: Uses previous words to predict next.
Q: Can it see both sides?
Answer: No, only left context.
11. BERT vs GPT
Q: Better for QA?
Answer: BERT, because it understands full context.
Q: Faster inference?
Answer: BERT for classification, GPT for generation tasks.
Q: Can they be combined?
Answer: Yes, hybrid models exist.
12. Autoencoder
Q: If bottleneck large?
Answer: Model may just copy input (no learning).
Example: Like memorizing instead of understanding.
Q: Used for classification?
Answer: Yes, features can be used.
Q: Difference from PCA?
Answer: PCA is linear, autoencoder is non-linear.
13. Bottleneck Layer
P a g e | 10
Q: Why compression?
Answer: Forces model to learn important features.
Q: If no bottleneck?
Answer: Model just memorizes.
Q: Prevent overfitting?
Answer: Yes, by limiting capacity.
14. Variational Autoencoder
Q: Why randomness?
Answer: To generate new data.
Q: Reparameterization trick?
Answer: Converts random sampling into differentiable form.
Q: Difference from autoencoder?
Answer: AE → fixed output
VAE → probabilistic output
15. KL Divergence
Q: Why normal distribution?
Answer: Easy to sample and stable.
Q: If removed?
Answer: Model loses generative ability.
Q: Symmetric?
Answer: No, KL(P||Q) ≠ KL(Q||P)
16. Reinforcement Learning
Q: Difference from supervised learning?
Answer: No labeled data, learning from rewards.
Q: Delayed reward problem?
Answer: Reward comes later, hard to link action.
Q: Real-life example?
Answer: Learning to drive through practice.
17. Policy
Q: Deterministic vs stochastic?
Answer:
Deterministic → fixed action
Stochastic → probability-based action
Q: Optimal policy?
Answer: Best strategy maximizing reward.
Q: How learned?
Answer: Through trial and error.
18. Graph Neural Networks
Q: How do nodes communicate?
Answer: By passing messages to neighbors.
P a g e | 11
Q: Message passing?
Answer: Aggregating neighbor information.
Q: Without edges?
Answer: No communication → no learning.
19. Diffusion Models
Q: Why add noise?
Answer: To learn reverse process of denoising.
Q: Training?
Answer: Learn to remove noise step-by-step.
Q: Difference from GAN?
Answer: GAN uses generator vs discriminator
Diffusion uses noise removal
20. Neural Network Compression
Q: Pruning?
Answer: Remove unimportant weights.
Q: Quantization?
Answer: Reduce precision (e.g., 32-bit → 8-bit)
Q: Trade-off?
Answer: Smaller model vs slight accuracy loss