Deep Learning Notes — Part 1: Basic For Data Science Placements

DEEP LEARNING NOTES

Part 1 — Basic | For Data Science Interns & Placements | Simple Enough for a 15-Year-Old
This document covers 8 fundamental Deep Learning topics — each with a definition, full explanation,
mathematical formula, real interview questions, common mistakes, use/don't-use guidance, visual description,
Python code, quick summary, and algorithm comparison.

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

FOUNDATIONAL
1 What is Deep Learning?

📖 DEFINITION

Deep Learning is a type of Machine Learning where computers learn from data using many layers
of calculations — just like how our brain uses layers of neurons to understand things.

💡 EXPLANATION

Imagine you want to teach a kid to recognize a dog. You don't give them rules like "4 legs + fur = dog." You just
show them 10,000 photos. Their brain figures out patterns on its own.
Deep Learning does the same thing. You feed it LOTS of data, and it builds its own internal understanding
through layers of math operations. Each layer learns something slightly more complex — first edges, then shapes,
then "dog face."
Example: Netflix's recommendation system uses deep learning. It doesn't just say "you watched action movies →
show action movies." It learns deep patterns: your viewing time, pause habits, what you rewatch — and predicts
what you'll love next.

📐 MATHEMATICAL FORMULA

Output = f( W·X + b )
→ W = Weights (what the model learns)
→ X = Input data
→ b = Bias (shifts the output)
→ f = Activation function (adds non-linearity)

🎤 REAL INTERVIEW QUESTIONS

Q1 What is the difference between Machine Learning and Deep Learning?

Q2 Why do we need multiple layers in a neural network?

Q3 Give a real-world example where deep learning outperforms traditional ML.

Q4 What kind of data is deep learning best suited for?

⚠️ COMMON MISTAKES TO AVOID

✗ Thinking deep learning is always better than ML (it's not — for small data, ML wins!)
✗ Forgetting that deep learning needs HUGE amounts of data
✗ Confusing 'deep' with 'accurate' — more layers does not always mean better

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Image recognition, NLP, Speech recognition ✗ Small datasets (less than 1000 rows)

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

✓ Large datasets (100k+ samples) ✗ When you need explainability and interpretability
✓ Unstructured data like images, audio, text ✗ Low compute budget situations

VISUAL / DIAGRAM DESCRIPTION

Picture a factory assembly line. Raw material (data) enters → Station 1 finds basic shapes → Station 2
identifies parts → Station 3 recognizes the full product. Each station equals one layer of the neural
network.

🐍 PYTHON CODE SNIPPET

# Hello Deep Learning with Keras!

import tensorflow as tf

model = [Link]([
[Link](128, activation='relu'), # Layer 1
[Link](64, activation='relu'), # Layer 2
[Link](10, activation='softmax') # Output
])
[Link](optimizer='adam', loss='categorical_crossentropy')
[Link]()

⚡ QUICK REVISION SUMMARY

→ Deep Learning = ML + many layers of calculations

→ Inspired by the human brain's neuron structure
→ Needs large data and high compute power
→ Best for images, text, audio, video
→ Each layer learns increasingly complex features

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm Key Difference

Traditional ML Needs manual feature engineering

Deep Learning Learns features automatically from raw data

Statistics Assumes data follows a distribution

Deep Learning Learns any distribution directly from data

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

CORE
2 Neural Networks (Artificial Neural Network)

📖 DEFINITION

A Neural Network is a computer system inspired by how our brain works. It has interconnected
'neurons' organized in layers — Input Layer, Hidden Layers, and Output Layer.

💡 EXPLANATION

Think of a neural network like a team of detectives working on a case.

Layer 1 (Input): Gets the clues (your data — like pixels of an image)
Hidden Layers: Each detective analyses the clue differently and passes notes to the next person
Output Layer: Final detective announces the verdict (cat or dog?)
Each connection between neurons has a weight — think of it as how much trust one detective puts in another's
notes. Training a neural network means adjusting all these weights until the answers become correct.
Example: A spam detector neural network. Input = email words → Hidden layers learn patterns like "free money,"
"click here" → Output = Spam or Not Spam.

📐 MATHEMATICAL FORMULA

Forward Pass:
z = W·X + b (linear combination)
a = activation(z) (non-linear activation)
Loss (MSE):
L = (1/n) sum(y_pred - y_true)^2

🎤 REAL INTERVIEW QUESTIONS

Q1 What are the different layers in a neural network?

Q2 What happens during forward propagation?

Q3 How does a neural network learn (adjust weights)?

Q4 What is the role of the output layer activation function?

⚠️ COMMON MISTAKES TO AVOID

✗ Not normalizing input data (always scale your inputs!)

✗ Using too many neurons with too little data — leads to overfitting
✗ Forgetting to set random seeds — results won't be reproducible

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

✓ Classification (spam, disease detection) ✗ Linear problems — just use linear regression
✓ Regression (price prediction) ✗ When you need to explain predictions to non-
✓ Pattern recognition tasks technical stakeholders

VISUAL / DIAGRAM DESCRIPTION

Draw 3 columns of circles. Left column = Input neurons (your features). Middle columns = Hidden
neurons. Right column = Output neurons. Lines connecting every circle = weights. Data flows left to right.

🐍 PYTHON CODE SNIPPET

import numpy as np

# Simple 2-layer neural net from scratch

X = [Link]([[0,0],[0,1],[1,0],[1,1]])
y = [Link]([[0],[1],[1],[0]]) # XOR problem

W1 = [Link](2,4) # weights layer 1

W2 = [Link](4,1) # weights layer 2

# Forward pass
h = [Link](X @ W1) # hidden layer
output = sigmoid(h @ W2) # output layer

⚡ QUICK REVISION SUMMARY

→ 3 parts: Input → Hidden → Output layers

→ Weights are the intelligence of the network
→ Forward pass = data flowing from input to output
→ Training = adjusting weights to reduce error
→ More hidden layers = deeper understanding (and more complexity)

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm Key Difference

Single Neuron (Perceptron) Can only solve linearly separable problems

Multi-layer ANN Can solve complex non-linear problems

Decision Tree Splits by rules, easy to interpret

ANN Learns patterns, hard to interpret but powerful

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

CORE
3 Activation Functions

📖 DEFINITION

An activation function decides whether a neuron should 'fire' (activate) or not. It adds non-linearity,
allowing the network to learn complex patterns — not just straight lines.

💡 EXPLANATION

Without activation functions, no matter how many layers you add, your whole neural network would just be doing
simple math that a single equation could replace.
Activation functions are the decision makers of each neuron. They squish, squeeze, or cut the signal before
passing it on.
ReLU (most popular): If the value is positive, pass it. If negative, send 0. Like a bouncer at a club — negative
vibes not allowed!
Sigmoid: Squishes any number into 0 to 1. Perfect for Yes/No outputs.
Softmax: For multi-class — converts numbers to probabilities that sum to 1. Like asking "which animal is this?"
and getting 70% dog, 20% cat, 10% wolf.

📐 MATHEMATICAL FORMULA

ReLU: f(x) = max(0, x)

Sigmoid: f(x) = 1 / (1 + e^(-x))
Tanh: f(x) = (e^x - e^(-x)) / (e^x + e^(-x))
Softmax: f(xi) = e^xi / sum(e^xj) for class i

🎤 REAL INTERVIEW QUESTIONS

Q1 Why do we need activation functions?

Q2 What is the dying ReLU problem?

Q3 When would you use Sigmoid vs Softmax?

Q4 What is the vanishing gradient problem and which activation causes it?

⚠️ COMMON MISTAKES TO AVOID

✗ Using Sigmoid in hidden layers — causes vanishing gradient problem

✗ Using ReLU for output layer in classification — use Softmax instead
✗ Not knowing WHEN to use which activation function

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

✓ ReLU: Hidden layers (default choice) ✗ Sigmoid or Tanh in deep hidden layers —
✓ Sigmoid: Binary classification output vanishing gradient
✓ Softmax: Multi-class output ✗ Linear activation in output for classification —
✓ Tanh: RNNs or when you need -1 to 1 range wrong predictions

VISUAL / DIAGRAM DESCRIPTION

Plot 4 graphs side by side. ReLU: flat at 0 for x<0, then diagonal line. Sigmoid: S-shaped curve from 0 to
1. Tanh: S-shaped from -1 to 1. Softmax: multiple inputs become bars that all sum to 100%.

🐍 PYTHON CODE SNIPPET

import numpy as np

x = [Link](-5, 5, 100)

relu = [Link](0, x)
sigmoid = 1 / (1 + [Link](-x))
tanh = [Link](x)

# In Keras -- just pass as string:

# Dense(64, activation='relu')
# Dense(1, activation='sigmoid') # binary output
# Dense(10, activation='softmax') # 10-class output

⚡ QUICK REVISION SUMMARY

→ Activation functions add non-linearity to neural networks

→ ReLU is the default choice for hidden layers
→ Sigmoid for binary output (0 or 1)
→ Softmax for multi-class output (probabilities sum to 1)
→ Avoid Sigmoid/Tanh in deep hidden layers to prevent vanishing gradients

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm Key Difference

ReLU Fast, simple, default — but can die (output always 0)

Leaky ReLU Fixes dying ReLU by allowing small negative values

Sigmoid Good for binary classification output

Softmax Good for multi-class classification output

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

TRAINING
4 Loss Functions & Optimization

📖 DEFINITION

A Loss Function measures how WRONG your model's predictions are. The optimizer is the
strategy that changes the model's weights to make it LESS wrong over time.

💡 EXPLANATION

Imagine you're playing darts blindfolded. After each throw, someone tells you "you missed by 30cm to the left."
That feedback is the Loss Function. You then adjust your throw — that's the Optimizer.
Loss Function = tells the model how bad your prediction was
Optimizer = tells the model how to change weights to be less bad
The most common optimizer is Adam (Adaptive Moment Estimation) — think of it as a smart GPS that adjusts
your speed AND direction based on the terrain.
Example: Predicting house prices. Your model says 50L, real answer is 80L. Loss = (80-50)^2 = 900. Now the
optimizer nudges all the weights slightly to make the next prediction closer to 80L.

📐 MATHEMATICAL FORMULA

Mean Squared Error (Regression):

MSE = (1/n) sum(y_pred - y_true)^2
Binary Cross-Entropy (Classification):
L = -[y * log(y_hat) + (1-y) * log(1 - y_hat)]
Gradient Descent Update Rule:
W_new = W_old - alpha * dL/dW
alpha = learning rate (step size)

🎤 REAL INTERVIEW QUESTIONS

Q1 What is the difference between loss and accuracy?

Q2 Why do we use Cross-Entropy instead of MSE for classification?

Q3 What happens if your learning rate is too high or too low?

Q4 What is Gradient Descent and how does it work?

⚠️ COMMON MISTAKES TO AVOID

✗ Using MSE for classification problems — use Cross-Entropy!

✗ Setting learning rate too high → model never converges
✗ Not monitoring loss curves — you won't know if training is failing
✗ Forgetting to reset gradients in PyTorch with optimizer.zero_grad()

✅ WHEN TO USE VS WHEN NOT TO USE

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

USE WHEN DON'T USE WHEN

✓ MSE: Regression (predicting numbers) ✗ MSE for classification — it does not penalize
✓ Binary Cross-Entropy: Binary classification confident wrong predictions enough
✓ Categorical Cross-Entropy: Multi-class
classification
✓ Adam optimizer: Almost always a safe default
choice

VISUAL / DIAGRAM DESCRIPTION

Draw a bowl shape (loss landscape). The ball (model parameters) starts at a random high point. Each
step, the ball rolls downhill (gradient descent) until it reaches the bowl's lowest point (minimum loss). Step
size = learning rate.

🐍 PYTHON CODE SNIPPET

from tensorflow import keras

[Link](
optimizer='adam', # smart optimizer
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)

history = [Link](X_train, y_train,

epochs=50,
validation_split=0.2,
batch_size=32
)

# Plot loss to monitor training

[Link]([Link]['loss'], label='Train')
[Link]([Link]['val_loss'], label='Val')

⚡ QUICK REVISION SUMMARY

→ Loss function = measures how wrong the model is

→ Optimizer = strategy to reduce the loss
→ Adam is the go-to optimizer for most problems
→ Learning rate = step size (too big → chaotic, too small → slow)
→ Always plot train vs val loss to catch overfitting

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm Key Difference

SGD Simple, slow convergence but very interpretable

Adam Fast, adaptive, works great out of the box

RMSprop Good for RNNs and sequential data

Adagrad Good for sparse data like NLP word counts

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

TRAINING
5 Backpropagation

📖 DEFINITION

Backpropagation is the algorithm that trains a neural network. It calculates how much EACH
weight contributed to the error, and then adjusts them all — going BACKWARDS from output to
input.

💡 EXPLANATION

Remember playing the blame game? Who caused this mistake? Backpropagation does exactly that — it traces
back through every weight in the network and assigns blame (gradient).
Step 1: Forward pass — data flows through the network, gives a prediction
Step 2: Calculate loss — how wrong was the prediction?
Step 3: Backward pass — calculate the gradient of the loss for each weight (using chain rule from calculus)
Step 4: Update weights — reduce each weight's contribution to the error
Example: Think of a relay race. The final runner (output) passes the baton (error signal) backwards to each
runner. Each runner adjusts their speed (weight update) based on how much they slowed the team.

📐 MATHEMATICAL FORMULA

Chain Rule (core of backprop):

dL/dW1 = dL/da2 * da2/dz2 * dz2/da1 * da1/dW1
Weight update:
W = W - alpha * dL/dW
Where:
alpha = learning rate
dL/dW = gradient of loss with respect to weight

🎤 REAL INTERVIEW QUESTIONS

Q1 Explain backpropagation in simple terms.

Q2 What is the chain rule and why is it used in backprop?

Q3 What is the vanishing gradient problem?

Q4 What is the difference between backpropagation and gradient descent?

⚠️ COMMON MISTAKES TO AVOID

✗ Confusing backpropagation with gradient descent (backprop COMPUTES gradients; gradient descent USES
them)
✗ Not understanding that backprop requires differentiable activation functions
✗ Thinking you need to code backprop yourself — frameworks do this automatically

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Used in EVERY neural network training — it's ✗ You never skip backprop in DL — but be aware
the core learning algorithm of its limitations with very deep networks
✓ Understanding it deeply helps debug
vanishing/exploding gradient problems

VISUAL / DIAGRAM DESCRIPTION

Draw a neural network with arrows going RIGHT (forward pass) and then dashed arrows going LEFT in
red (backward pass / backprop). At each node, show gradient values getting smaller as you go further left
= vanishing gradient.

🐍 PYTHON CODE SNIPPET

# PyTorch shows backprop explicitly

import torch

x = [Link]([2.0], requires_grad=True)
W = [Link]([3.0], requires_grad=True)
y_true = [Link]([10.0])

y_pred = W * x # forward pass

loss = (y_pred - y_true)**2 # compute loss

[Link]() # THIS IS BACKPROP!

print([Link]) # gradient of W

W = W - 0.01 * [Link] # gradient descent step

⚡ QUICK REVISION SUMMARY

→ Backprop = the learning algorithm of neural networks

→ Uses chain rule to compute gradients layer by layer
→ Goes BACKWARDS from output to input
→ Frameworks (TensorFlow, PyTorch) do this automatically
→ Vanishing gradient = gradients become tiny in early layers so they stop learning

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm Key Difference

Backpropagation Computes gradients (how much to adjust)

Gradient Descent Uses gradients to UPDATE the weights

Forward Propagation Makes predictions (input → output)

Backprop + GD Together Complete training loop of a neural network

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

REGULARIZATION
6 Overfitting, Underfitting & Regularization

📖 DEFINITION

Overfitting = model memorizes training data but fails on new data. Underfitting = model is too
simple to learn patterns. Regularization = techniques to find the sweet spot.

💡 EXPLANATION

Imagine studying for an exam.

Overfitting = you memorized all past papers word-for-word but can't answer new questions.
Underfitting = you barely studied and can't answer anything.
The goal is generalization — learning the concepts, not the exact questions.
Common regularization techniques:
Dropout: Randomly turn off neurons during training → forces the network to not rely on any single neuron
L2 Regularization (Weight Decay): Adds a penalty for large weights — keeps the model simple
Early Stopping: Stop training when validation loss starts increasing
Example: Like cross-validation in studying — test yourself on practice questions you haven't seen before.

📐 MATHEMATICAL FORMULA

L2 Regularization (Ridge):
L_total = L_original + lambda * sum(W^2)
lambda = regularization strength (hyperparameter)
Dropout:
Keep probability = p (typically 0.5-0.8)
During inference: multiply activations by p

🎤 REAL INTERVIEW QUESTIONS

Q1 What is the difference between overfitting and underfitting?

Q2 How does Dropout prevent overfitting?

Q3 What is the bias-variance tradeoff?

Q4 How do you know if your model is overfitting?

⚠️ COMMON MISTAKES TO AVOID

✗ Applying dropout during inference/testing (it should be OFF during testing)

✗ Setting dropout too high (above 0.5 on early layers) — model won't learn
✗ Not using a validation set — you won't detect overfitting!
✗ Regularizing too much → underfitting

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

✅ WHEN TO USE VS WHEN NOT TO USE

USE WHEN DON'T USE WHEN

✓ Dropout: When model is overfitting (val loss ✗ Don't use high dropout on small datasets — you
greater than train loss) need all the data you can get
✓ L2 Reg: When weights are becoming very large
✓ Early Stopping: Always! Monitor validation loss

VISUAL / DIAGRAM DESCRIPTION

Draw a graph with 3 curves: Training Loss always decreasing. Validation Loss: Underfitting = high
throughout. Good fit = decreases with training. Overfitting = decreases then INCREASES. The point
where val loss starts rising = stop training.

🐍 PYTHON CODE SNIPPET

from [Link] import layers, regularizers

model = [Link]([
[Link](256, activation='relu',
kernel_regularizer=regularizers.l2(0.001)),

[Link](0.5), # drop 50% neurons randomly

[Link](128, activation='relu'),
[Link](0.3),
[Link](10, activation='softmax')
])

# Early stopping callback

early_stop = [Link](
monitor='val_loss', patience=5, restore_best_weights=True)

⚡ QUICK REVISION SUMMARY

→ Overfitting = great on train data, terrible on test data

→ Underfitting = bad on both train and test data
→ Dropout: randomly kills neurons during training → prevents over-reliance
→ L2 regularization: penalizes large weights → keeps model simpler
→ Always use a validation set to detect overfitting early

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm Key Difference

Dropout Randomly disables neurons — acts like training many models

L1 Regularization Pushes some weights to exactly 0 → feature selection

L2 Regularization Shrinks all weights toward 0 (but not exactly 0)

Early Stopping Stops training before model starts memorizing noise

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

CNN
7 Convolutional Neural Networks (CNN) Basics

📖 DEFINITION

A CNN is a special type of neural network designed for images. Instead of connecting every pixel
to every neuron, it uses small filters to scan across the image and detect features like edges,
shapes, and textures.

💡 EXPLANATION

Imagine you're looking for Waldo in a Where's Waldo book. You don't look at the WHOLE page at once. You scan
small sections one by one. CNNs do the same!
Convolution Layer: A small filter (3x3 or 5x5) slides across the image looking for specific patterns (edges, corners)
Pooling Layer: Shrinks the image (keeps only the most important info) → faster computation
Flatten + Dense: Finally, flattens everything and uses regular neural network layers to classify
Example: Instagram's face filter. CNN detects where your eyes, nose, and mouth are → places the filter exactly
there. Without CNN, this would need millions of manually coded rules.

📐 MATHEMATICAL FORMULA

Convolution operation:
Feature Map(i,j) = sum sum Input(i+m, j+n) * Filter(m,n)
Output size after convolution:
Output = (Input - Filter + 2*Padding) / Stride + 1
Max Pooling (2x2):
Output = max of each 2x2 block

🎤 REAL INTERVIEW QUESTIONS

Q1 What is the purpose of the convolution layer?

Q2 Why do we use pooling layers in CNNs?

Q3 What is the difference between valid and same padding?

Q4 How does a CNN achieve translation invariance?

⚠️ COMMON MISTAKES TO AVOID

✗ Using fully connected networks for image data — CNNs are FAR more efficient
✗ Forgetting to normalize pixel values to 0-1 before feeding to CNN
✗ Not using data augmentation when you have limited image data
✗ Using too many pooling layers → losing too much spatial information

✅ WHEN TO USE VS WHEN NOT TO USE

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

USE WHEN DON'T USE WHEN

✓ Image classification, Object detection, Face ✗ Don't use CNNs for tabular/structured data —
recognition regular networks or trees work better
✓ Medical imaging (X-ray diagnosis) ✗ Don't use for sequential text — use RNNs or
✓ Any spatial data — satellite imagery, video Transformers
frames

VISUAL / DIAGRAM DESCRIPTION

Show a 6x6 image. Draw a 3x3 filter sliding across it (with arrows showing movement). Show the resulting
smaller feature map. Then show max pooling shrinking it further. Finally show flatten → dense → output.

🐍 PYTHON CODE SNIPPET

from [Link] import layers

cnn = [Link]([
# Convolution block 1
layers.Conv2D(32, (3,3), activation='relu', input_shape=(28,28,1)),
layers.MaxPooling2D((2,2)),

# Convolution block 2
layers.Conv2D(64, (3,3), activation='relu'),
layers.MaxPooling2D((2,2)),

# Classifier head
[Link](),
[Link](128, activation='relu'),
[Link](10, activation='softmax') # 10 classes
])

⚡ QUICK REVISION SUMMARY

→ CNNs are specialized for image and spatial data

→ Convolution layer = small filter sliding across image to find features
→ Pooling layer = shrinks feature maps to reduce computation
→ Translation invariant — can find a cat regardless of where it is in image
→ Key layers: Conv2D → MaxPool → Flatten → Dense → Softmax

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm Key Difference

Regular Dense Network on Millions of parameters, poor performance, no spatial awareness

Images

CNN Far fewer params, detects spatial patterns, state-of-art on images

ResNet/VGG (advanced CNNs) Deeper CNNs, pre-trained, used in transfer learning

Vision Transformer (ViT) New approach, no convolution, treats image as patches

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

NLP
8 Embeddings & Word Vectors

📖 DEFINITION

An Embedding converts words (or categories) into lists of numbers (vectors) so that computers
can process them mathematically. Similar words end up with similar numbers.

💡 EXPLANATION

Computers understand numbers, not words. So how do you feed a sentence to a neural network?
Bad approach: One-Hot Encoding — cat = [1,0,0,...], dog = [0,1,0,...]. Problem? No relationship between words.
King and Queen look totally unrelated!
Good approach: Word2Vec / Embeddings — king and queen are CLOSE in vector space. king - man + woman =
queen. The math CAPTURES meaning!
Each word becomes a vector of maybe 100-300 numbers. Words used in similar contexts end up near each other
in space.
Example: Google Translate uses embeddings. Happy in English is mapped near Feliz in Spanish because they
appear in similar contexts across millions of documents.

📐 MATHEMATICAL FORMULA

Word2Vec (Skip-gram objective):

Maximize: P(context word | center word)
P(wc | wt) = exp(vc dot vt) / sum exp(vj dot vt)
Cosine Similarity (measure word closeness):
similarity = (A dot B) / (||A|| * ||B||)
Range: -1 (opposite) to 1 (identical)

🎤 REAL INTERVIEW QUESTIONS

Q1 What is the difference between one-hot encoding and word embeddings?

Q2 What does it mean for two words to be close in embedding space?

Q3 What is Word2Vec and how does it work?

Q4 How are embeddings trained?

⚠️ COMMON MISTAKES TO AVOID

✗ Using one-hot encoding for NLP — doesn't capture word relationships

✗ Forgetting to handle out-of-vocabulary (OOV) words in production
✗ Not fine-tuning pre-trained embeddings when your domain is specific (medical, legal)

✅ WHEN TO USE VS WHEN NOT TO USE

Part 1: Basic Deep Learning | Page See footer

Deep Learning Notes — Part 1: Basic For Data Science Placements

USE WHEN DON'T USE WHEN

✓ NLP tasks: sentiment analysis, translation, ✗ Don't use embeddings for numerical/continuous
chatbots features — they're already numbers
✓ Recommendation systems (embed user/item ✗ Don't train embeddings from scratch when pre-
IDs) trained ones (GloVe, BERT) exist
✓ Any categorical variable with many unique
values

VISUAL / DIAGRAM DESCRIPTION

Draw 2D space with word dots. King and Queen are close. Cat and Dog are close. King and Cat are far
apart. Draw an arrow from King to Queen = same direction as Man to Woman. This shows embeddings
capture relationships.

🐍 PYTHON CODE SNIPPET

from [Link] import Embedding

import [Link] as api

# Option 1: Keras Embedding layer (learned from scratch)

embed_layer = Embedding(
input_dim=10000, # vocab size
output_dim=128, # embedding dimensions
input_length=50 # sequence length
)

# Option 2: Pre-trained Word2Vec

model = [Link]('word2vec-google-news-300')
model.most_similar('king') # → queen, prince...

# Famous analogy
result = model['king'] - model['man'] + model['woman']

⚡ QUICK REVISION SUMMARY

→ Embeddings convert words to dense numerical vectors

→ Similar words have similar embeddings (close in vector space)
→ Famous: king - man + woman = queen
→ Pre-trained embeddings (Word2Vec, GloVe, BERT) save training time
→ Cosine similarity = measure how close two word vectors are

🔄 COMPARISON WITH SIMILAR ALGORITHMS

Algorithm Key Difference

One-Hot Encoding Sparse, huge vectors, no semantic meaning between words

Word2Vec Dense, 300-dim, captures semantic relationships

GloVe Like Word2Vec but trained on word co-occurrence matrix

BERT Embeddings Context-aware — same word gets different vector in different sentences

Part 1: Basic Deep Learning | Page See footer