Deep Learning Notes — Part 2: Intermediate For Data Science Placements

DEEP LEARNING NOTES

Part 2 — Intermediate | For Data Science Interns & Placements | 8 Core Intermediate Topics

Part 1: Basic (Done) Part 2: Intermediate (NOW) Part 3: Advanced (Coming)

This document covers 8 intermediate Deep Learning topics — each with a definition, full explanation,
mathematical formula, real interview questions, common mistakes, use/do-not-use guidance, visual description,
Python code, quick summary, and algorithm comparison.

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

INTERMEDIATE
1 Batch Normalization

📖 DEFINITION

Batch Normalization (BatchNorm) is a technique that normalizes the output of each layer during
training so that the inputs to the next layer always have a consistent mean of 0 and standard
deviation of 1. This stabilizes and dramatically speeds up training.

💡 EXPLANATION

Imagine you are a teacher grading 30 students across wildly different subjects — some score 2/100, others
98/100. If you normalize all scores to the same scale first, comparing and improving becomes much easier.
BatchNorm does the same for neuron outputs.
Without BatchNorm, as data passes through many layers, the distribution of values can shift wildly — called
Internal Covariate Shift. This forces the network to constantly readjust, slowing learning.
BatchNorm fixes this by: (1) computing mean and variance of the current mini-batch, (2) normalizing the output,
(3) then applying learnable scale (gamma) and shift (beta) parameters so the network can undo the normalization
if needed.
Example: Training a deep ResNet without BatchNorm is like running a relay race where each runner uses a
completely different speed scale. With BatchNorm, every runner uses the same scale — the team runs smoothly
and fast.

📐 MATHEMATICAL FORMULA

Normalize:
x_hat = (x - mean_batch) / sqrt(variance_batch + epsilon)
Scale and Shift (learnable):
y = gamma * x_hat + beta
Where:
gamma = learned scale parameter
beta = learned shift parameter
epsilon = small constant (~1e-5) to avoid division by zero

🎤 REAL INTERVIEW QUESTIONS

Q1 What problem does Batch Normalization solve?

Q2 What is Internal Covariate Shift?

Q3 What are the learnable parameters in BatchNorm?

Q4 Should BatchNorm be applied before or after the activation function?

Q5 What happens to BatchNorm during inference/testing?

⚠️ COMMON MISTAKES TO AVOID

✗ Using BatchNorm with very small batch sizes (batch size < 8) — statistics become unreliable

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

✗ Forgetting that BatchNorm behaves differently during training vs inference (uses running mean/var at test time)
✗ Placing it wrong — generally use Conv → BatchNorm → ReLU order
✗ Using BatchNorm with RNNs — Layer Normalization works better for sequences

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Deep CNNs and feedforward networks ✗ Very small batch sizes — use Layer Norm
✓ When training is unstable or very slow instead
✓ When you want to use higher learning rates ✗ RNNs and sequence models — Layer Norm is
safely preferred
✗ When batch statistics are not representative of
the full data

VISUAL / DIAGRAM DESCRIPTION

Draw a pipeline: [Layer Output] → [Compute Mean & Variance of batch] → [Normalize to mean=0, var=1]
→ [Scale by gamma, Shift by beta] → [Next Layer Input]. Add a note: during training, use batch stats;
during inference, use running averages stored from training.

🐍 PYTHON CODE SNIPPET

import tensorflow as tf
from [Link] import layers

# Standard order: Conv → BatchNorm → Activation

model = [Link]([
layers.Conv2D(64, (3,3), padding='same', use_bias=False),
[Link](), # normalize outputs
[Link]('relu'),

layers.Conv2D(128, (3,3), padding='same', use_bias=False),

[Link](),
[Link]('relu'),

layers.GlobalAveragePooling2D(),
[Link](10, activation='softmax')
])

# In PyTorch
import [Link] as nn
layer = [Link](
nn.Conv2d(64, 128, 3, padding=1, bias=False),
nn.BatchNorm2d(128),
[Link]()
)

⚡ QUICK REVISION SUMMARY

→ BatchNorm normalizes layer outputs to mean=0, std=1 within a mini-batch

→ Fixes Internal Covariate Shift — makes deep networks train faster and more stably
→ Has learnable gamma (scale) and beta (shift) so the network can adapt
→ Behaves differently at train time (batch stats) vs test time (running stats)

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

→ Standard order: Conv → BatchNorm → ReLU

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm / Concept Key Difference

Batch Normalization Normalizes across the batch — best for CNNs with large batches

Layer Normalization Normalizes across features — best for RNNs and Transformers

Instance Normalization Normalizes each sample independently — used in style transfer

Group Normalization Normalizes within groups of channels — good for small batches

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

INTERMEDIATE
2 Recurrent Neural Networks (RNN)

📖 DEFINITION

An RNN is a neural network designed for sequential data — where the order of inputs matters.
Unlike regular networks, RNNs have a memory (hidden state) that carries information from
previous time steps to influence future predictions.

💡 EXPLANATION

Imagine reading a book. When you read the word "bank," you need context — are we talking about a river bank or
a financial bank? Previous words (context) determine the meaning. RNNs work exactly like this.
In a regular neural network, each input is processed independently. In an RNN, each time step produces a hidden
state that is passed to the next time step — creating a chain of memory.
The hidden state acts like a running summary: "here is everything I have seen so far." At each step, the network
reads the new input AND the previous hidden state together to produce the next output and hidden state.
Example: Autocomplete on your phone. When you type "I want to eat," the RNN reads each word sequentially,
building context, and predicts "pizza" based on the full sequence — not just the last word.

📐 MATHEMATICAL FORMULA

Hidden state update:

h_t = tanh(W_h * h_(t-1) + W_x * x_t + b)
Output:
y_t = W_y * h_t + b_y
Where:
h_t = current hidden state
h_(t-1) = previous hidden state (memory)
x_t = current input
W_h, W_x, W_y = weight matrices

🎤 REAL INTERVIEW QUESTIONS

Q1 What is the difference between RNN and a regular feedforward network?

Q2 What is the vanishing gradient problem in RNNs?

Q3 What is the hidden state in an RNN?

Q4 When would you use a bidirectional RNN?

Q5 Why did LSTMs replace vanilla RNNs?

⚠️ COMMON MISTAKES TO AVOID

✗ Using vanilla RNN for long sequences — it forgets early context (use LSTM or GRU instead)
✗ Not truncating backpropagation through time (BPTT) for very long sequences
✗ Forgetting to reset hidden states between independent sequences

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

✗ Not padding sequences to the same length in batches

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Time series prediction (stock prices, weather) ✗ Very long sequences — use LSTM/GRU or
✓ Text generation and language modeling Transformers
✓ Speech recognition ✗ Parallel processing requirements — RNNs are
✓ Any data where order and sequence matter sequential and slow
✗ Image data — use CNNs instead

VISUAL / DIAGRAM DESCRIPTION

Draw a chain of boxes labeled h0 → h1 → h2 → h3. Each box receives an input x_t from below and
passes its hidden state to the right. Above each box is the output y_t. An arrow loops back from each
hidden state to the next — this loop is the memory.

🐍 PYTHON CODE SNIPPET

import tensorflow as tf
from [Link] import layers

# Simple RNN for sequence classification

model = [Link]([
[Link](input_dim=10000, output_dim=64),
[Link](128, return_sequences=True), # returns all steps
[Link](64), # returns last step only
[Link](1, activation='sigmoid')
])

# In PyTorch
import [Link] as nn
rnn = [Link](input_size=64, hidden_size=128,
num_layers=2, batch_first=True)
output, hidden = rnn(x) # x shape: (batch, seq_len, features)

⚡ QUICK REVISION SUMMARY

→ RNNs process sequential data by maintaining a hidden state (memory)

→ The hidden state summarizes everything seen so far in the sequence
→ Vanilla RNNs suffer from vanishing gradients — cannot remember long-term context
→ LSTM and GRU are improved versions that solve the long-term memory problem
→ Used for text, speech, time series — any ordered sequential data

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm / Concept Key Difference

Feedforward Network No memory — each input processed independently

Vanilla RNN Has memory but forgets long sequences (vanishing gradient)

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

LSTM Solves vanishing gradient with gates — remembers long sequences

Transformer No recurrence — uses attention to handle sequences in parallel

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

INTERMEDIATE
3 LSTM & GRU (Long Short-Term Memory)

📖 DEFINITION

LSTM is an advanced RNN that uses special gates (Forget Gate, Input Gate, Output Gate) to
selectively remember or forget information over long sequences. GRU is a simplified version with
just two gates but similar performance.

💡 EXPLANATION

Think of your brain before an important exam. You selectively remember important concepts (KEEP), forget
irrelevant details (FORGET), and use the knowledge at the right moment (OUTPUT). LSTMs work exactly this
way.
Vanilla RNNs have one problem — they forget things from far back in the sequence (vanishing gradient). LSTMs
solve this with a Cell State — like a conveyor belt that carries information across many time steps with minimal
modification.
Forget Gate: Decides what to throw away from cell state ("forget this old info")
Input Gate: Decides what new information to store ("learn this new info")
Output Gate: Decides what to output at this time step ("use this relevant info")
GRU (Gated Recurrent Unit) simplifies this to just two gates (Reset and Update) — nearly same performance but
faster to train.

📐 MATHEMATICAL FORMULA

LSTM Gates:
f_t = sigmoid(W_f * [h_(t-1), x_t] + b_f) Forget gate
i_t = sigmoid(W_i * [h_(t-1), x_t] + b_i) Input gate
o_t = sigmoid(W_o * [h_(t-1), x_t] + b_o) Output gate
Cell State Update:
C_t = f_t * C_(t-1) + i_t * tanh(W_c * [h_(t-1), x_t] + b_c)
Hidden State:
h_t = o_t * tanh(C_t)

🎤 REAL INTERVIEW QUESTIONS

Q1 How does LSTM solve the vanishing gradient problem?

Q2 What are the three gates in an LSTM and what does each do?

Q3 What is the difference between the cell state and hidden state in LSTM?

Q4 When would you choose GRU over LSTM?

Q5 What is a bidirectional LSTM and when is it used?

⚠️ COMMON MISTAKES TO AVOID

✗ Using LSTM when Transformers would be better for very long sequences

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

✗ Not using return_sequences=True when stacking multiple LSTM layers

✗ Forgetting to reset cell states between independent batches
✗ Using too many LSTM layers without dropout — overfitting is common

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Long text sequences (sentiment analysis, ✗ Very long documents (1000+ tokens) — use
translation) Transformers/BERT instead
✓ Time series with long-term dependencies (ECG ✗ When training speed is critical — Transformers
signals, weather forecasting) are much faster with GPUs
✓ Speech recognition ✗ Simple short sequences — vanilla RNN or even
✓ Music generation and sequential creative tasks 1D CNN may suffice

VISUAL / DIAGRAM DESCRIPTION

Draw one LSTM cell. Show three sigmoid gates (Forget, Input, Output) as separate boxes. Show the Cell
State as a horizontal line running through the top (the conveyor belt). Show h_(t-1) and x_t as inputs on
the left. Show h_t as output on the right. Label each gate with its purpose.

🐍 PYTHON CODE SNIPPET

import tensorflow as tf
from [Link] import layers

# Stacked Bidirectional LSTM

model = [Link]([
[Link](10000, 128),

# Bidirectional reads sequence both forward and backward

[Link]([Link](128, return_sequences=True)),
[Link](0.3),

[Link]([Link](64)),
[Link](0.3),

[Link](64, activation='relu'),
[Link](1, activation='sigmoid') # binary sentiment
])

# GRU — simpler and often just as good

[Link](128, return_sequences=True)

⚡ QUICK REVISION SUMMARY

→ LSTMs fix vanishing gradients using three gates: Forget, Input, Output
→ The Cell State is the long-term memory (conveyor belt across time steps)
→ GRU is a simpler LSTM variant with two gates — faster but similar accuracy
→ Bidirectional LSTM reads sequence both forward and backward for richer context
→ LSTM was the go-to for NLP before Transformers took over in 2017

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm / Concept Key Difference

Vanilla RNN Simple but forgets long-range dependencies

LSTM 3 gates, cell state — best long-term memory, more parameters

GRU 2 gates, no cell state — faster, slightly less capacity than LSTM

Transformer No recurrence — attention-based, parallelizable, now state-of-art

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

INTERMEDIATE
4 Transfer Learning

📖 DEFINITION

Transfer Learning is the technique of taking a model pre-trained on a large dataset (like ImageNet
with 14 million images) and reusing its learned knowledge for a new, related task — saving
massive amounts of time and data.

💡 EXPLANATION

You already know how to drive a car. Learning to drive a truck is much easier because most skills transfer —
steering, braking, traffic rules. You only need to learn the new parts (bigger vehicle, different gear system).
Transfer Learning does the same with neural networks. A model trained on ImageNet has already learned to
detect edges, textures, shapes, and objects. When you apply it to a new task (say, detecting tumors in X-rays), it
already knows the basics — you just fine-tune it for the specific task.
Two common approaches:
(1) Feature Extraction: Freeze all pre-trained layers. Only train the new output head. Fast but less specialized.
(2) Fine-Tuning: Unfreeze some or all layers and train with a very small learning rate. Slower but more powerful.
Example: Google's MobileNet trained on ImageNet can detect your custom object (e.g., ripe vs. unripe mangoes)
with just 500 photos instead of millions.

📐 MATHEMATICAL FORMULA

Fine-tuning learning rate rule:

lr_new_layers = 1e-3 (train from scratch)
lr_frozen_layers = 0 (frozen, no update)
lr_fine_tuned = 1e-5 (very small, gentle updates)
Freeze ratio guideline:
Use bottom 70% of layers frozen
Fine-tune top 30% of layers
Always retrain the final classification head

🎤 REAL INTERVIEW QUESTIONS

Q1 What is Transfer Learning and why is it useful?

Q2 What is the difference between feature extraction and fine-tuning?

Q3 When should you fine-tune vs. use features only?

Q4 What pre-trained models do you know and when would you use each?

Q5 What is domain adaptation in Transfer Learning?

⚠️ COMMON MISTAKES TO AVOID

✗ Using too large a learning rate when fine-tuning — destroys pre-trained knowledge

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

✗ Not freezing any layers when data is very small — model will overfit
✗ Using an unrelated pre-trained model (e.g., NLP model for images)
✗ Forgetting to change the output layer to match your number of classes

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Small datasets (less than 10,000 samples) — ✗ When your data is completely unrelated to the
Transfer Learning is a must pre-training domain
✓ Medical imaging, satellite imagery, custom ✗ When you have millions of samples — training
object detection from scratch may be viable
✓ NLP tasks — use BERT, GPT pre-trained ✗ When the pre-trained model architecture does
models not fit your task structure
✓ When compute budget is limited

VISUAL / DIAGRAM DESCRIPTION

Draw two networks side by side. LEFT: Big pre-trained model (VGG/ResNet) trained on ImageNet —
many layers, fully trained. RIGHT: Your model — same base layers (FROZEN, shown in grey) + new
small head layers (TRAINABLE, shown in orange). Arrow pointing from left to right = transfer.

🐍 PYTHON CODE SNIPPET

import tensorflow as tf

# Load pre-trained MobileNetV2 (no top classifier)

base_model = [Link].MobileNetV2(
input_shape=(224, 224, 3),
include_top=False, # remove ImageNet classifier
weights='imagenet' # load pre-trained weights
)

# Phase 1: Feature Extraction (freeze base)

base_model.trainable = False

model = [Link]([
base_model,
[Link].GlobalAveragePooling2D(),
[Link](256, activation='relu'),
[Link](0.3),
[Link](5, activation='softmax') # your classes
])

# Phase 2: Fine-Tuning (unfreeze some layers)

base_model.trainable = True
for layer in base_model.layers[:-30]: # freeze all but last 30
[Link] = False

[Link](optimizer=[Link](1e-5),
loss='categorical_crossentropy', metrics=['accuracy'])

⚡ QUICK REVISION SUMMARY

→ Transfer Learning reuses a pre-trained model for a new related task

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

→ Saves massive training time and requires far less data

→ Feature extraction: freeze all layers, train only new head
→ Fine-tuning: unfreeze top layers and train with very small learning rate
→ Popular pre-trained models: VGG16, ResNet50, MobileNet (images), BERT, GPT (text)

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm / Concept Key Difference

Training from Scratch Needs millions of samples and weeks of GPU compute

Feature Extraction (Transfer) Fast, minimal data needed, less specialized

Fine-Tuning (Transfer) More accurate, needs slightly more data and time

Zero-Shot Learning No training at all — model generalizes to unseen classes

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

INTERMEDIATE
5 Data Augmentation

📖 DEFINITION

Data Augmentation is the technique of artificially creating new training examples from existing
ones by applying transformations like flipping, rotating, cropping, or adding noise — without
collecting new data.

💡 EXPLANATION

Suppose you want to teach a child to recognize a cat. You show them one photo. Now you flip it upside down,
change the brightness, zoom in — the child still recognizes it as a cat. You have created multiple learning
experiences from one photo.
Deep learning models need lots of data to generalize well. But collecting and labeling data is expensive. Data
Augmentation solves this by transforming existing images during training so the model sees many "versions" of
the same sample.
For images: flip, rotate, crop, zoom, brightness/contrast changes, adding Gaussian noise, cutout (randomly
removing patches).
For text: synonym replacement, back-translation, random insertion/deletion.
For time series: time warping, window slicing, adding noise.
Example: MedMNIST — a medical image dataset where augmentation is critical because getting 10,000 real X-
ray scans is impossible, but augmenting 1,000 into 50,000 is easy.

📐 MATHEMATICAL FORMULA

Augmentation probability:
Apply each transform with probability p (typically 0.5)
Mixup Augmentation:
x_mix = lambda * x1 + (1 - lambda) * x2
y_mix = lambda * y1 + (1 - lambda) * y2
lambda ~ Beta(alpha, alpha) (typically alpha = 0.2)
CutMix:
Replace rectangular region of image with patch from another image

🎤 REAL INTERVIEW QUESTIONS

Q1 Why is data augmentation important for deep learning?

Q2 What augmentations would you apply to medical images vs. natural images?

Q3 Does data augmentation help with overfitting? How?

Q4 What is MixUp and CutMix augmentation?

Q5 Should you apply augmentation to validation and test sets?

⚠️ COMMON MISTAKES TO AVOID

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

✗ Applying augmentation to validation and test sets — never do this!

✗ Using unrealistic augmentations (e.g., flipping medical scans upside down when orientation matters)
✗ Not adjusting bounding box coordinates after geometric augmentations in object detection
✗ Over-augmenting with too many transforms — can slow training and introduce noise

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Small image datasets (less than 10,000 images) ✗ Augmentation should NOT be applied to
✓ When model is overfitting on training data validation or test data
✓ Medical imaging, satellite data with limited ✗ Domain-inappropriate augmentations (horizontal
samples flip for text/documents)
✓ Almost always for image-based tasks ✗ When data is already very large —
augmentation helps less

VISUAL / DIAGRAM DESCRIPTION

Show one original cat image in the center. Surround it with 6 versions: flipped horizontally, rotated 30
degrees, zoomed in, brightness increased, random crop, and Gaussian noise added. All arrows point
outward from the original = one sample becomes many training examples.

🐍 PYTHON CODE SNIPPET

import tensorflow as tf
from [Link] import layers

# Option 1: Keras augmentation layers (applied during training only)

data_augmentation = [Link]([
[Link]('horizontal'),
[Link](0.15),
[Link](0.15),
[Link](0.15),
[Link](0.15),
])

# Option 2: Albumentations (more powerful, industry standard)

import albumentations as A
transform = [Link]([
[Link](p=0.5),
[Link](p=0.3),
[Link](shift_limit=0.1, scale_limit=0.1, rotate_limit=15, p=0.5),
[Link](p=0.2),
[Link](num_holes=8, max_h_size=16, p=0.3)
])

⚡ QUICK REVISION SUMMARY

→ Data Augmentation creates artificial training examples from existing data

→ Helps prevent overfitting by forcing the model to generalize
→ Common techniques: flip, rotate, crop, zoom, brightness, noise
→ Advanced techniques: MixUp, CutMix, AutoAugment
→ NEVER apply augmentation to validation or test sets

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm / Concept Key Difference

Basic Augmentation (flip, crop) Simple, fast, minimal risk of distortion

MixUp Blends two images and their labels — improves calibration

CutMix Pastes patches from one image into another — very effective

AutoAugment Uses reinforcement learning to find best augmentation policy

automatically

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

INTERMEDIATE
6 Hyperparameter Tuning

📖 DEFINITION

Hyperparameters are settings you choose BEFORE training — like learning rate, batch size,
number of layers, dropout rate. Hyperparameter Tuning is the process of finding the optimal
combination of these settings to maximize model performance.

💡 EXPLANATION

Think of baking a cake. The recipe's baking temperature, time, and ingredient ratios are hyperparameters. The
exact amounts of each ingredient the dough "learns" = model parameters. You tune the recipe settings
(hyperparameters) to get the perfect cake.
Model parameters (weights) are learned automatically during training. Hyperparameters must be set manually or
searched automatically.
Key hyperparameters to tune:
Learning rate — the most important hyperparameter. Too high = diverges. Too low = too slow.
Batch size — affects gradient stability and memory.
Number of layers/neurons — model capacity.
Dropout rate — regularization strength.
Optimizer type — Adam vs SGD vs RMSprop.
Search strategies:
Grid Search: Try every combination (exhaustive but slow).
Random Search: Try random combinations (faster, often better than grid).
Bayesian Optimization: Intelligent search based on past results (most efficient).

📐 MATHEMATICAL FORMULA

Learning Rate Schedule (Cosine Annealing):

lr(t) = lr_min + 0.5*(lr_max - lr_min)*(1 + cos(pi*t/T))
Learning Rate Range Test (LR Finder):
Increase lr exponentially each batch
Find point where loss starts decreasing steeply
Use lr just before the minimum loss point
Batch Size vs Learning Rate rule of thumb:
If batch size doubles → multiply learning rate by sqrt(2)

🎤 REAL INTERVIEW QUESTIONS

Q1 What is the difference between hyperparameters and model parameters?

Q2 How would you tune the learning rate for a new problem?

Q3 What is the difference between Grid Search and Random Search?

Q4 What is Bayesian Optimization in hyperparameter tuning?

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

Q5 What is a Learning Rate Scheduler and when would you use one?

⚠️ COMMON MISTAKES TO AVOID

✗ Tuning hyperparameters on the test set — causes data leakage

✗ Ignoring the learning rate — it is the most impactful hyperparameter
✗ Only using Grid Search — it is exponentially slow with many hyperparameters
✗ Not using a validation set for tuning — always tune on validation, evaluate on test

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Whenever you need to push model performance ✗ When you have very limited compute — use
beyond a baseline sensible defaults instead
✓ Competition settings where every 0.1% ✗ When data quality is the bottleneck — fix data
accuracy matters first, then tune
✓ Before deploying a production model ✗ When the model architecture itself is the
problem

VISUAL / DIAGRAM DESCRIPTION

Draw a 2D grid with Learning Rate on X-axis and Dropout on Y-axis. Grid Search = every cell tested
(shown as dots in every cell). Random Search = dots scattered randomly (covers more of the space
efficiently). Bayesian = dots concentrated near the best regions found so far.

🐍 PYTHON CODE SNIPPET

# Option 1: Keras Tuner (easy and Keras-native)

import keras_tuner as kt

def build_model(hp):
model = [Link]()
[Link]([Link](
units=[Link]('units', min_value=64, max_value=512, step=64),
activation='relu'
))
[Link]([Link](
[Link]('dropout', min_value=0.1, max_value=0.5, step=0.1)
))
[Link]([Link](10, activation='softmax'))
[Link](
optimizer=[Link](
[Link]('learning_rate', [1e-2, 1e-3, 1e-4])
),
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
return model

tuner = [Link](build_model, objective='val_accuracy', max_trials=20)

[Link](X_train, y_train, epochs=10, validation_data=(X_val, y_val))
best_model = tuner.get_best_models(1)[0]

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

⚡ QUICK REVISION SUMMARY

→ Hyperparameters are set before training; model parameters are learned during training
→ Learning rate is the most important hyperparameter to tune first
→ Grid Search = exhaustive but slow; Random Search = faster and surprisingly effective
→ Bayesian Optimization = smartest search, uses past results to guide next attempt
→ Always tune on validation set, NEVER on test set

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm / Concept Key Difference

Grid Search Tries all combinations — exhaustive, exponentially slow

Random Search Tries random combos — faster, covers space better

Bayesian Optimization Uses probabilistic model to find best params intelligently

Population-Based Training Evolves hyperparameters during training — used by DeepMind

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

INTERMEDIATE
7 Attention Mechanism

📖 DEFINITION

Attention is a technique that allows a neural network to focus on the most relevant parts of the
input when making a prediction — instead of treating all parts equally. It is the core idea behind
Transformers and modern NLP models like BERT and GPT.

💡 EXPLANATION

When you read the sentence "The trophy didn't fit in the suitcase because it was too big," what does "it" refer to?
Your brain focuses on (attends to) "trophy" because context makes it relevant. Attention mechanisms teach neural
networks to do the same.
In traditional RNNs, all past information is compressed into a single hidden state — a bottleneck. Attention solves
this by letting the model look at ALL previous positions and decide which ones matter most for the current
prediction.
How it works:
Query (Q): What am I looking for?
Key (K): What does each position offer?
Value (V): What information does each position contain?
Attention Score = how relevant each position is = softmax(Q * K^T / sqrt(d_k))
Final output = weighted sum of Values based on attention scores.
Example: In translation, when predicting the French word for "ate," the model attends strongly to "ate" in the
English source sentence, not "quickly" or "the."

📐 MATHEMATICAL FORMULA

Scaled Dot-Product Attention:

Attention(Q, K, V) = softmax(Q * K^T / sqrt(d_k)) * V
Where:
Q = Query matrix (what I am looking for)
K = Key matrix (what each position offers)
V = Value matrix (actual content at each position)
d_k = dimension of keys (scaling factor)
Multi-Head Attention:
MultiHead(Q,K,V) = Concat(head_1, ..., head_h) * W_O
head_i = Attention(Q*W_Qi, K*W_Ki, V*W_Vi)

🎤 REAL INTERVIEW QUESTIONS

Q1 What problem does the attention mechanism solve in seq2seq models?

Q2 What are Query, Key, and Value in attention?

Q3 What is the difference between self-attention and cross-attention?

Q4 Why do we scale by sqrt(d_k) in attention?

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

Q5 What is Multi-Head Attention and why is it better than single-head?

⚠️ COMMON MISTAKES TO AVOID

✗ Confusing self-attention (query from same sequence) with cross-attention (query from different sequence)
✗ Forgetting the scaling factor 1/sqrt(d_k) — causes vanishing gradients with large dot products
✗ Thinking attention is only used in NLP — it is used in vision (ViT), audio, and more
✗ Not using masking in decoder attention to prevent looking at future tokens

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Machine translation and text summarization ✗ Simple short sequence classification — LSTM
✓ Question answering systems may be simpler and sufficient
✓ Any task where long-range dependencies matter ✗ Very resource-constrained environments —
✓ Vision Transformers for image classification attention is O(n^2) in sequence length
✗ Real-time edge devices without GPU
acceleration

VISUAL / DIAGRAM DESCRIPTION

Draw a sentence: "The cat sat on the mat." Show attention scores as a heatmap grid (sentence x
sentence). When predicting the meaning of "sat," the model highlights "cat" (subject) and "mat" (location)
with darker colors — showing which words it attends to most.

🐍 PYTHON CODE SNIPPET

import tensorflow as tf

# Built-in Keras Multi-Head Attention

mha = [Link](
num_heads=8, # 8 parallel attention heads
key_dim=64 # dimension of each head
)

# Self-attention: query, key, value all from same sequence

output = mha(query=x, key=x, value=x)

# Cross-attention: query from decoder, key/value from encoder

output = mha(query=decoder_seq, key=encoder_seq, value=encoder_seq)

# Simple attention from scratch (educational)

import numpy as np
def attention(Q, K, V):
d_k = [Link][-1]
scores = [Link](Q, K.T) / [Link](d_k) # scale
weights = [Link](scores) / [Link]([Link](scores)) # softmax
return [Link](weights, V) # weighted sum of values

⚡ QUICK REVISION SUMMARY

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

→ Attention lets the model focus on the most relevant parts of the input
→ Uses three matrices: Query (what I want), Key (what I offer), Value (what I contain)
→ Attention score = softmax(Q * K^T / sqrt(d_k)) — tells how much to focus on each position
→ Multi-Head Attention runs multiple attention operations in parallel for richer representation
→ Self-attention is the core of Transformers, BERT, GPT — the backbone of modern NLP

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm / Concept Key Difference

RNN (no attention) Fixed-size hidden state bottleneck — forgets distant context

RNN + Attention Looks at all encoder hidden states — solved NMT before Transformers

Self-Attention (Transformer) Every position attends to every other — parallelizable and powerful

Multi-Head Attention Runs attention in parallel across different subspaces — richer features

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

INTERMEDIATE
8 Transformer Architecture (Basics)

📖 DEFINITION

The Transformer is a neural network architecture built entirely on Attention (no RNNs, no
convolutions). It processes the entire sequence in parallel and has become the foundation of all
modern AI models: BERT, GPT, T5, ChatGPT, and more.

💡 EXPLANATION

In 2017, the paper "Attention Is All You Need" introduced the Transformer. Before this, RNNs dominated NLP.
Transformers made everything faster and better.
Key idea: Instead of processing one token at a time (like RNN), process the ENTIRE sequence at once using self-
attention. This enables massive parallelization — train on thousands of GPUs simultaneously.
Encoder: Reads the full input and creates rich contextual representations. Each token "looks at" all other tokens
via self-attention.
Decoder: Generates the output one token at a time, attending to encoder output and its own previous outputs.
Positional Encoding: Since there is no recurrence, we must inject position information into the input embeddings.
Example: When you type a prompt into ChatGPT, your entire prompt is processed in parallel by the Transformer
encoder/decoder — not word by word like older systems. This is why it responds so fast.

📐 MATHEMATICAL FORMULA

Transformer block (one layer):

1. x = LayerNorm(x + MultiHeadAttention(x, x, x)) # self-attention
2. x = LayerNorm(x + FFN(x)) # feed-forward
Feed-Forward Network (FFN):
FFN(x) = max(0, x*W1 + b1)*W2 + b2
Positional Encoding:
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

🎤 REAL INTERVIEW QUESTIONS

Q1 What is the key innovation of the Transformer over RNNs?

Q2 What is positional encoding and why is it needed?

Q3 What is the difference between the Encoder and Decoder in a Transformer?

Q4 What is masked self-attention in the Transformer decoder?

Q5 How does BERT differ from GPT in terms of Transformer architecture?

⚠️ COMMON MISTAKES TO AVOID

✗ Thinking Transformers replaced all RNNs — RNNs still have edge cases where they are better

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

✗ Forgetting that Transformers have O(n^2) memory complexity with sequence length
✗ Confusing BERT (encoder-only) with GPT (decoder-only) with T5 (encoder-decoder)
✗ Not using positional encoding — the model has no way to know word order without it

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Natural Language Processing — translation, ✗ Very long sequences (100k+ tokens) without
summarization, QA, generation modifications like Longformer/BigBird
✓ Vision Transformers (ViT) for image ✗ On-device inference with tight memory
classification constraints without distillation
✓ Multimodal tasks combining text and images ✗ Simple classification tasks where a small CNN
(CLIP, DALL-E) or MLP suffices
✓ Any large-scale pre-training task

VISUAL / DIAGRAM DESCRIPTION

Draw the Transformer encoder stack: Input → Embedding + Positional Encoding → [Multi-Head Self-
Attention → Add & Norm → Feed Forward → Add & Norm] x N layers → Output Representation. Draw
decoder similarly but with an extra cross-attention layer attending to encoder output. Show that all
positions process in parallel (arrows go both ways).

🐍 PYTHON CODE SNIPPET

import tensorflow as tf
from [Link] import layers

# Transformer Encoder Block

class TransformerBlock([Link]):
def __init__(self, d_model, num_heads, dff, dropout=0.1):
super().__init__()
[Link] = [Link](num_heads=num_heads,
key_dim=d_model//num_heads)
[Link] = [Link]([
[Link](dff, activation='relu'),
[Link](d_model)
])
self.norm1 = [Link]()
self.norm2 = [Link]()
self.drop1 = [Link](dropout)
self.drop2 = [Link](dropout)

def call(self, x, training=False):

# Self-attention + residual
attn = [Link](x, x, x)
x = self.norm1(x + self.drop1(attn, training=training))
# Feed-forward + residual
ffn = [Link](x)
return self.norm2(x + self.drop2(ffn, training=training))

# Use pre-built BERT for NLP tasks

from transformers import TFBertModel
bert = TFBertModel.from_pretrained('bert-base-uncased')

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer
Deep Learning Notes — Part 2: Intermediate For Data Science Placements

⚡ QUICK REVISION SUMMARY

→ Transformer uses ONLY attention — no RNNs or convolutions

→ Processes the full sequence in parallel — much faster than RNNs
→ Positional Encoding injects word order information since there is no recurrence
→ Encoder-only (BERT) for understanding tasks; Decoder-only (GPT) for generation
→ Foundation of ALL modern AI: ChatGPT, BERT, T5, DALL-E, Stable Diffusion

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm / Concept Key Difference

RNN/LSTM Sequential processing, great memory, slow training

Transformer Encoder (BERT) Bidirectional, understands context both ways, great for classification/QA

Transformer Decoder (GPT) Unidirectional (left-to-right), great for text generation

Encoder-Decoder (T5, BART) Full Transformer, great for translation and summarization

Part 2: Intermediate Deep Learning | Topics: BatchNorm, RNN, LSTM, Transfer Learning, Augmentation, Hyperparameter Tuning, Attention, Transformer