NLP Complete Study Guide

From Scratch to Advanced

Definitions · Google Colab Code · Real-World Examples

Level 1: Foundations | Level 2: Text Representation | Level 3: Core Tasks

Level 4: Deep Learning | Level 5: Advanced NLP & LLMs

Prepared for Data Science & NLP Learners

Covers All 5 NLP Levels with Code

Format Definition + Colab Code + Real Example

NLP Complete Study Guide | Page 1

Table of Contents

LEVEL 1 — FOUNDATIONS
• 1.1 Tokenization
• 1.2 Stemming
• 1.3 Lemmatization
• 1.4 Stop Words
• 1.5 Part-of-Speech (POS) Tagging

LEVEL 2 — TEXT REPRESENTATION

• 2.1 Bag of Words (BoW)
• 2.2 TF-IDF
• 2.3 N-grams
• 2.4 Word2Vec
• 2.5 GloVe
• 2.6 FastText

LEVEL 3 — CORE NLP TASKS

• 3.1 Sentiment Analysis
• 3.2 Named Entity Recognition (NER)
• 3.3 Text Classification
• 3.4 Machine Translation
• 3.5 Text Summarization
• 3.6 Dependency Parsing
• 3.7 Question Answering
• 3.8 Information Extraction

LEVEL 4 — DEEP LEARNING FOR NLP

• 4.1 RNN
• 4.2 LSTM
• 4.3 Attention Mechanism
• 4.4 Transformer Architecture
• 4.5 Positional Encoding

LEVEL 5 — ADVANCED NLP & LLMs

• 5.1 BERT
• 5.2 GPT / LLMs
• 5.3 Fine-tuning & PEFT (LoRA)
• 5.4 RAG
• 5.5 Prompt Engineering

NLP Complete Study Guide | Page 2

• 5.6 Semantic Search
• 5.7 RLHF
• 5.8 Multimodal NLP

NLP Complete Study Guide | Page 3

 LEVEL 1 — FOUNDATIONS

The foundation of any NLP pipeline is text preprocessing. Before a machine can understand language, raw
text must be cleaned and structured. These steps transform messy human text into a consistent format
that algorithms can work with.

1.1 Tokenization

Definition
Tokenization is the process of splitting a piece of text into smaller units called tokens. A token can be a
word, subword, character, or even a sentence. It is the very first step in almost every NLP pipeline. Without
tokenization, the model has no way to process raw text as individual meaningful units.

Types of Tokenization
• Word tokenization — splits text by spaces and punctuation (e.g. 'Hello world' → ['Hello','world'])
• Sentence tokenization — splits a paragraph into sentences
• Subword tokenization — breaks rare words into smaller known pieces (used in BERT, GPT)
• Character tokenization — each character is a token (used in some language models)

Google Colab Code

# Install NLTK (run once in Colab)

!pip install nltk

import nltk

[Link]('punkt')

[Link]('punkt_tab')

from [Link] import word_tokenize, sent_tokenize

text = 'Natural Language Processing is amazing. It helps computers understand human

s.'

# Word tokenization

words = word_tokenize(text)

print('Word Tokens:', words)

# Output: ['Natural', 'Language', 'Processing', 'is', 'amazing', '.', ...]

# Sentence tokenization

sentences = sent_tokenize(text)

print('Sentence Tokens:', sentences)

# Output: ['Natural Language Processing is amazing.', 'It helps...']

# Subword tokenization with HuggingFace

!pip install transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

NLP Complete Study Guide | Page 4

tokens = [Link]('unhappiness')

print('Subword tokens:', tokens)

# Output: ['un', '##happiness']

Real-World Example — Google Search

When you type 'running shoes under 500' in Google, the search engine tokenizes your query into
['running', 'shoes', 'under', '500'] to match relevant web pages. Each token is searched independently and
combined to rank results.

NLP Complete Study Guide | Page 5

 1.2 Stemming

Definition
Stemming is the process of reducing a word to its root or base form by removing suffixes and prefixes
using rule-based algorithms. The resulting stem may not be a real dictionary word. For example, 'running'
→ 'run', 'happily' → 'happili'. Stemming is fast but can be inaccurate.

Common Stemming Algorithms

• Porter Stemmer — most popular, works well for English
• Lancaster Stemmer — more aggressive than Porter
• Snowball Stemmer — improved version of Porter, supports multiple languages

Google Colab Code

import nltk

from [Link] import PorterStemmer, LancasterStemmer, SnowballStemmer

porter = PorterStemmer()

lancaster = LancasterStemmer()

snowball = SnowballStemmer('english')

words = ['running', 'happiness', 'studies', 'caring', 'flies', 'easily']

print(f'{'Word':<15} {'Porter':<15} {'Lancaster':<15} {'Snowball'}')

print('-' * 60)

for word in words:

print(f'{word:<15} {[Link](word):<15} {[Link](word):<15} {snowball

.stem(word)}')

# Sample Output:

# Word Porter Lancaster Snowball

# running run run run

# happiness happi happy happi

# studies studi study studi

Real-World Example — E-commerce Search

On Amazon, when you search 'buying', 'bought', and 'buy', stemming ensures all three map to the root
'buy', returning the same product listings. This improves recall in search engines without needing separate
indexes for every verb form.

1.3 Lemmatization

Definition
Lemmatization reduces a word to its dictionary base form (lemma) using vocabulary and morphological
analysis. Unlike stemming, it always returns a valid word. For example, 'better' → 'good', 'running' → 'run',
'geese' → 'goose'. It is more accurate but slower than stemming.

NLP Complete Study Guide | Page 6

Stemming vs Lemmatization — Quick Comparison

Feature Stemming Lemmatization

Output May not be a real word Always a valid word

Speed Fast Slower

Accuracy Lower Higher

Example: 'studies' studi study

Example: 'better' better good

Google Colab Code

import nltk

[Link]('wordnet')

[Link]('averaged_perceptron_tagger')

from [Link] import WordNetLemmatizer

from [Link] import wordnet

lemmatizer = WordNetLemmatizer()

# Basic lemmatization

words = ['running', 'studies', 'better', 'geese', 'caring', 'corpora']

for word in words:

print(f'{word} -> {[Link](word, pos="v")} (verb) / '

f'{[Link](word, pos="n")} (noun)')

# Output:

# running -> run (verb) / running (noun)

# studies -> study (verb) / study (noun)

# geese -> geese (verb) / goose (noun)

Real-World Example — Chatbot Understanding

A customer support chatbot receives messages like 'I am having issues', 'I had an issue', 'issues are
happening'. Lemmatization maps all forms of 'issue' to its base form so the bot correctly identifies all
messages as the same complaint category.

NLP Complete Study Guide | Page 7

 1.4 Stop Words

Definition
Stop words are common words that appear frequently but carry very little meaningful information
for NLP tasks. Words like 'the', 'is', 'at', 'which', 'on', 'a', 'an' are typically removed to reduce noise and
improve model efficiency. The list of stop words varies by language and task.

When to Remove vs Keep Stop Words

• Remove: text classification, sentiment analysis, topic modelling, TF-IDF search
• Keep: machine translation, question answering, grammar checking (meaning changes without them)

Google Colab Code

import nltk

[Link]('stopwords')

from [Link] import stopwords

from [Link] import word_tokenize

stop_words = set([Link]('english'))

print('Total stop words in English:', len(stop_words))

print('Sample:', list(stop_words)[:10])

text = 'The quick brown fox jumps over the lazy dog in the morning'

tokens = word_tokenize([Link]())

# Remove stop words

filtered = [w for w in tokens if w not in stop_words and [Link]()]

print('\nOriginal tokens:', tokens)

print('After stop word removal:', filtered)

# Output: ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog', 'morning']

# spaCy approach

!pip install spacy

!python -m spacy download en_core_web_sm

import spacy

nlp = [Link]('en_core_web_sm')

doc = nlp(text)

filtered_spacy = [[Link] for token in doc if not token.is_stop]

print('spaCy filtered:', filtered_spacy)

Real-World Example — Email Spam Filter

Gmail's spam detector ignores stop words like 'the', 'is', 'a' and focuses on meaningful words like 'lottery',
'prize', 'click', 'winner'. Removing stop words reduces computation and improves accuracy of the classifier.

1.5 Part-of-Speech (POS) Tagging

NLP Complete Study Guide | Page 8

Definition
Part-of-Speech (POS) tagging assigns a grammatical category (noun, verb, adjective, adverb, pronoun,
etc.) to each word in a sentence. It helps NLP models understand the grammatical structure and meaning
of text. For example: 'The cat sat on the mat' → [The/DT, cat/NN, sat/VBD, on/IN, the/DT, mat/NN].

Common POS Tags (Penn Treebank)

Tag Meaning Example

NN Noun, singular dog, city

NNS Noun, plural dogs, cities

VB Verb, base form run, eat

VBD Verb, past tense ran, ate

JJ Adjective big, happy

RB Adverb quickly, very

DT Determiner the, a, an

IN Preposition on, in, at

PRP Personal pronoun I, he, she

Google Colab Code

import nltk

[Link]('averaged_perceptron_tagger')

[Link]('punkt')

from nltk import pos_tag, word_tokenize

sentence = 'The quick brown fox jumps over the lazy dog'

tokens = word_tokenize(sentence)

pos_tags = pos_tag(tokens)

print('POS Tags:', pos_tags)

# [('The','DT'),('quick','JJ'),('brown','JJ'),('fox','NN'),

# ('jumps','VBZ'),('over','IN'),('the','DT'),('lazy','JJ'),('dog','NN')]

# Using spaCy for richer POS info

import spacy

nlp = [Link]('en_core_web_sm')

doc = nlp('Apple is looking at buying U.K. startup for 1 billion dollars')

for token in doc:

print(f'{[Link]:<15} {token.pos_:<8} {token.tag_:<8} {token.dep_}')

Real-World Example — Grammar Checker (Grammarly)

Grammarly uses POS tagging to detect grammar errors. When you write 'She run fast', POS tagging
identifies 'run' as VB (base verb) instead of VBZ (third-person singular), flagging it as incorrect and
suggesting 'runs'.

NLP Complete Study Guide | Page 9

NLP Complete Study Guide | Page 10
 LEVEL 2 — TEXT REPRESENTATION

After preprocessing, text must be converted into numerical form for machine learning algorithms. This level
covers classical and neural methods for representing text as numbers — from simple frequency counts to
dense semantic vectors.

2.1 Bag of Words (BoW)

Definition
Bag of Words is a text representation technique that counts how many times each word appears in a
document, ignoring grammar and word order. Each document becomes a vector of word counts across the
entire vocabulary. The 'bag' metaphor means word order is thrown away — only frequency matters.

How it works — Example

Documents: (1) 'I love NLP' (2) 'I love Python' (3) 'NLP and Python are great'
Vocabulary: [I, love, NLP, Python, and, are, great]

Document I love NLP Python and are great

Doc 1 1 1 1 0 0 0 0

Doc 2 1 1 0 1 0 0 0

Doc 3 0 0 1 1 1 1 1

Google Colab Code

from sklearn.feature_extraction.text import CountVectorizer

corpus = [

'I love NLP and machine learning',

'NLP is amazing and powerful',

'Machine learning helps solve real problems',

'I love solving problems with NLP',

vectorizer = CountVectorizer()

X = vectorizer.fit_transform(corpus)

print('Vocabulary:', vectorizer.get_feature_names_out())

print('\nBoW Matrix shape:', [Link])

print('BoW Matrix:\n', [Link]())

# Limitation demo: word order lost

v2 = CountVectorizer()

docs = ['dog bites man', 'man bites dog']

print('\nSame BoW for both:', v2.fit_transform(docs).toarray())

NLP Complete Study Guide | Page 11

# Both get identical vectors — BoW cannot distinguish meaning from order

Real-World Example — News Article Categorisation

BBC News uses Bag of Words to categorise articles. An article containing frequent words like 'goal',
'match', 'player', 'stadium' gets a high sports score, while one with 'election', 'vote', 'parliament' scores high
for politics.

NLP Complete Study Guide | Page 12

 2.2 TF-IDF (Term Frequency - Inverse Document Frequency)

Definition
TF-IDF is a statistical measure that evaluates how important a word is to a document relative to a
collection. It combines two components:
• TF (Term Frequency) — how often a word appears in a document. Common words in that doc get
high TF.
• IDF (Inverse Document Frequency) — penalises words that appear in many documents (like 'the').
Rare words get high IDF.
Formula: TF-IDF(t, d) = TF(t, d) × log(N / df(t)) where N = total docs, df = docs containing term t

Google Colab Code

from sklearn.feature_extraction.text import TfidfVectorizer

import pandas as pd

corpus = [

'the cat sat on the mat',

'the dog sat on the log',

'the cat chased the dog',

tfidf = TfidfVectorizer()

X = tfidf.fit_transform(corpus)

df = [Link]([Link](),

columns=tfidf.get_feature_names_out(),

index=['Doc1','Doc2','Doc3'])

print([Link](3))

# Notice: 'the' has very LOW score (appears in all docs)

# 'cat', 'log', 'chased' have HIGH scores (unique to specific docs)

# Find most important word per document

for i, doc in enumerate(corpus):

top_word = [Link][i].idxmax()

print(f'Doc {i+1} most important word: {top_word}')

Real-World Example — Google Search Ranking

Google uses TF-IDF concepts in its ranking. If you search 'transformer', pages that use the word
'transformer' frequently (high TF) but where it is rare across the web (high IDF) rank higher than pages
where it appears generically.

2.3 N-grams

Definition

NLP Complete Study Guide | Page 13

An N-gram is a contiguous sequence of N words from a text. Instead of treating words individually,
N-grams capture word combinations, preserving some word-order context. A 1-gram (unigram) is a single
word, 2-gram (bigram) is two consecutive words, 3-gram (trigram) is three.
Example: 'I love natural language processing'
• Unigrams (1-gram): ['I', 'love', 'natural', 'language', 'processing']
• Bigrams (2-gram): ['I love', 'love natural', 'natural language', 'language processing']
• Trigrams (3-gram): ['I love natural', 'love natural language', 'natural language processing']

Google Colab Code

from nltk import ngrams, word_tokenize

from collections import Counter

from sklearn.feature_extraction.text import CountVectorizer

text = 'I love natural language processing because NLP is powerful'

tokens = word_tokenize([Link]())

# Generate bigrams and trigrams with NLTK

bigrams = list(ngrams(tokens, 2))

trigrams = list(ngrams(tokens, 3))

print('Bigrams:', bigrams[:4])

print('Trigrams:', trigrams[:3])

# Frequency of bigrams in a corpus

corpus = ['I love NLP', 'NLP is great', 'I love Python', 'NLP and Python']

vec = CountVectorizer(ngram_range=(2,2)) # bigrams only

X = vec.fit_transform(corpus)

print('\nBigram vocabulary:', vec.get_feature_names_out())

# Most common bigrams

all_bg = list(ngrams(word_tokenize(' '.join(corpus).lower()), 2))

print('Most common bigrams:', Counter(all_bg).most_common(5))

Real-World Example — Autocomplete / Predictive Text

When you type 'Happy Birth' on your phone keyboard, the autocomplete suggests 'day' using
bigram/trigram probabilities. The model learned that 'Birthday' follows 'Happy Birth' very frequently in
training data.

NLP Complete Study Guide | Page 14

 2.4 Word2Vec

Definition
Word2Vec is a neural network-based word embedding technique that maps words to dense numerical
vectors in a continuous vector space. Words with similar meanings end up close to each other in this
space. It captures semantic relationships — famously: King - Man + Woman = Queen.

Two Architectures
• CBOW (Continuous Bag of Words) — predicts a target word from surrounding context words
• Skip-gram — predicts surrounding context words from a target word. Better for rare words.

Google Colab Code

!pip install gensim

from [Link] import Word2Vec

from [Link] import word_tokenize, sent_tokenize

import nltk; [Link]('punkt')

# Training corpus

text = '''

King rules the kingdom. Queen is the wife of the king.

Man works hard every day. Woman works equally hard.

Doctor treats patients in the hospital.

Engineer builds software and systems.

Python is a programming language. Java is also a language.

'''

# Tokenize into sentences then words

sentences = [word_tokenize([Link]()) for s in sent_tokenize(text)]

# Train Word2Vec model

model = Word2Vec(sentences, vector_size=100, window=5,

min_count=1, workers=4, epochs=100)

# Vector for a word

print('Vector for king (first 5 dims):', [Link]['king'][:5])

# Similar words

print('Words similar to king:', [Link].most_similar('king', topn=3))

# Analogy: king - man + woman = ?

result = [Link].most_similar(positive=['king','woman'], negative=['man'])

print('King - Man + Woman =', result[0][0]) # Expected: queen

# Similarity score

print('Similarity king-queen:', [Link]('king','queen'))

print('Similarity king-python:', [Link]('king','python'))

NLP Complete Study Guide | Page 15

 Real-World Example — Recommendation Systems
Netflix uses word2vec-style embeddings on user watch history. Movies are treated as 'words' and viewing
sessions as 'sentences'. Movies watched together frequently end up close in vector space, enabling
'customers who watched X also liked Y' recommendations.

2.5 GloVe (Global Vectors for Word Representation)

Definition
GloVe is an unsupervised learning algorithm for word embeddings developed by Stanford. Unlike
Word2Vec which uses local context windows, GloVe uses global word co-occurrence statistics from
the entire corpus. It builds a word-word co-occurrence matrix and factorises it to get embeddings. GloVe
typically outperforms Word2Vec on word analogy tasks.

Google Colab Code

# Download pre-trained GloVe embeddings (run in Colab)

!wget -q [Link]

!unzip -q [Link]

import numpy as np

# Load GloVe vectors

def load_glove(path):

embeddings = {}

with open(path, encoding='utf-8') as f:

for line in f:

values = [Link]()

word = values[0]

vector = [Link](values[1:], dtype='float32')

embeddings[word] = vector

return embeddings

glove = load_glove('[Link]') # 50-dimensional vectors

print('Vocabulary size:', len(glove))

print('Vector for "computer":', glove['computer'][:5])

# Cosine similarity function

def cosine_sim(a, b):

return [Link](a, b) / ([Link](a) * [Link](b))

print('Sim(king, queen):', cosine_sim(glove['king'], glove['queen']))

print('Sim(king, banana):', cosine_sim(glove['king'], glove['banana']))

Real-World Example — Document Similarity in Legal Tech

Legal tech companies use GloVe embeddings to find similar legal documents. A contract about 'property
lease' is represented as the average of its word vectors, and cosine similarity finds similar clauses or
precedents automatically.

NLP Complete Study Guide | Page 16

NLP Complete Study Guide | Page 17
 2.6 FastText

Definition
FastText, developed by Facebook AI, extends Word2Vec by treating each word as a bag of character
n-grams. Instead of learning vectors for whole words, it learns vectors for character substrings (e.g.,
'eating' → 'ea', 'eat', 'ati', 'tin', 'ing', 'ting'). This makes FastText excellent for handling rare and
out-of-vocabulary words and works well for morphologically rich languages.

Google Colab Code

!pip install fasttext-wheel

# Alternative: use gensim's FastText

from [Link] import FastText

from [Link] import word_tokenize, sent_tokenize

corpus = '''

NLP stands for Natural Language Processing.

Deep learning improves NLP dramatically.

Transformers revolutionized language models.

Python is widely used for NLP tasks.

'''

sentences = [word_tokenize([Link]()) for s in sent_tokenize(corpus)]

# Train FastText model

ft_model = FastText(sentences, vector_size=100, window=5,

min_count=1, epochs=50, min_n=2, max_n=6)

# Works with unseen / misspelled words!

print('Vector for "NLP":', ft_model.wv['nlp'][:3])

print('Handles misspelling - "Pythoon":', ft_model.wv['pythoon'][:3])

# ^ FastText generates a vector even for unseen words via character n-grams

print('Similar to "learning":', ft_model.wv.most_similar('learning', topn=3))

Real-World Example — Multilingual NLP at Facebook

Facebook uses FastText for language identification and content moderation across 170+ languages.
Because it handles subword units, it correctly processes words with typos, regional spellings, and
morphological variations common in user-generated social media content.

NLP Complete Study Guide | Page 18

 LEVEL 3 — CORE NLP TASKS

This level covers the main real-world problems NLP solves. These are the tasks that drive products you
use every day — from detecting spam emails to translating languages.

3.1 Sentiment Analysis

Definition
Sentiment Analysis (also called Opinion Mining) is the use of NLP to identify and extract subjective
information from text — primarily whether the expressed opinion is positive, negative, or neutral. It can
also detect emotions like joy, anger, fear, and surprise.

Google Colab Code

!pip install transformers torch vaderSentiment

# Method 1: VADER (rule-based, good for social media)

from [Link] import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()

reviews = [

'This product is absolutely amazing! I love it!',

'Terrible experience, never buying again.',

'It is okay, nothing special.',

'Best purchase I have made this year!!!'

for review in reviews:

score = analyzer.polarity_scores(review)

sentiment = 'POSITIVE' if score['compound'] > 0.05 else \

'NEGATIVE' if score['compound'] < -0.05 else 'NEUTRAL'

print(f'{sentiment}: {review[:40]}...')

print(f' Scores: {score}\n')

# Method 2: Transformer-based (more accurate)

from transformers import pipeline

sentiment_pipe = pipeline('sentiment-analysis',

model='distilbert-base-uncased-finetuned-sst-2-english')

results = sentiment_pipe(reviews)

for r, res in zip(reviews, results):

print(f'{res["label"]} ({res["score"]:.2f}): {r[:45]}')

NLP Complete Study Guide | Page 19

 Real-World Example — Brand Monitoring (Twitter/X)
Companies like Nike monitor millions of tweets daily using sentiment analysis. After a product launch, they
automatically classify tweets as positive/negative, track sentiment trends hour by hour, and alert
marketing teams when negative sentiment spikes above a threshold.

3.2 Named Entity Recognition (NER)

Definition
Named Entity Recognition is an NLP task that identifies and classifies named entities in text into
predefined categories such as person names, organisations, locations, dates, monetary values,
percentages, and more. It extracts structured information from unstructured text.

Common Entity Types

• PERSON — 'Elon Musk', 'Sachin Tendulkar'
• ORG — 'Google', 'Tata Motors', 'United Nations'
• GPE (Geo-political entity) — 'India', 'Chennai', 'USA'
• DATE — 'January 15', '2024', 'yesterday'
• MONEY — '$1 billion', 'Rs 500 crore'

Google Colab Code

!pip install spacy

!python -m spacy download en_core_web_sm

import spacy

nlp = [Link]('en_core_web_sm')

text = '''

Elon Musk, CEO of Tesla and SpaceX, announced on Monday that the company

will invest $5 billion in India by 2025. The meeting took place in New Delhi

with Prime Minister Narendra Modi.

'''

doc = nlp(text)

print('Named Entities Found:')

print(f'{'Entity':<25} {'Label':<12} {'Description'}')

print('-' * 60)

for ent in [Link]:

print(f'{[Link]:<25} {ent.label_:<12} {[Link](ent.label_)}')

# Output:

# Elon Musk PERSON People, including fictional

# Tesla ORG Companies, agencies, institutions

# SpaceX ORG Companies, agencies, institutions

# Monday DATE Absolute or relative dates

NLP Complete Study Guide | Page 20

# $5 billion MONEY Monetary values

# India GPE Countries, cities, states

# 2025 DATE Absolute or relative dates

# New Delhi GPE Countries, cities, states

Real-World Example — Financial News Analysis (Bloomberg)

Bloomberg Terminal uses NER to extract company names, stock tickers, financial figures, and dates from
thousands of news articles per minute. This structured data is used to automatically update market
dashboards and trigger trading alerts.

NLP Complete Study Guide | Page 21

 3.3 Text Classification

Definition
Text Classification is the task of assigning predefined categories or labels to text documents. It is one
of the most widely used NLP tasks. Applications include spam detection, language detection, topic
classification, and intent detection.

Google Colab Code

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.naive_bayes import MultinomialNB

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import train_test_split

from [Link] import classification_report

# Sample dataset

texts = [

'Win a million dollars click here now', 'Congratulations you won a prize',

'Buy cheap medication online now', 'Free iPhone click this link',

'Meeting tomorrow at 10am', 'Can you review my code pull request',

'Project deadline is Friday EOD', 'Lunch at 1pm conference room B',

labels = [1,1,1,1,0,0,0,0] # 1=spam, 0=ham

X_train, X_test, y_train, y_test = train_test_split(

texts, labels, test_size=0.3, random_state=42)

# TF-IDF + Logistic Regression

tfidf = TfidfVectorizer(ngram_range=(1,2))

X_train_vec = tfidf.fit_transform(X_train)

X_test_vec = [Link](X_test)

clf = LogisticRegression()

[Link](X_train_vec, y_train)

y_pred = [Link](X_test_vec)

print(classification_report(y_test, y_pred, target_names=['Ham','Spam']))

# Test on new message

new_msg = ['Click to claim your free gift now!!!']

print('Prediction:', 'SPAM' if [Link]([Link](new_msg))[0] else 'HAM')

Real-World Example — Gmail Spam Filter

Gmail processes over 100 billion emails per day and classifies each as spam or not-spam using text
classification. The model considers word patterns, sender reputation, and historical feedback from millions
of users who clicked 'Report Spam'.

NLP Complete Study Guide | Page 22

 3.4 Machine Translation

Definition
Machine Translation (MT) is the automatic translation of text from one natural language to another using
computational methods. Modern MT uses sequence-to-sequence (Seq2Seq) neural networks with
attention, or Transformer models. Google Translate and DeepL are prominent examples.

Google Colab Code

!pip install transformers sentencepiece sacremoses

from transformers import MarianMTModel, MarianTokenizer

# Load a pre-trained translation model (English to French)

model_name = 'Helsinki-NLP/opus-mt-en-fr'

tokenizer = MarianTokenizer.from_pretrained(model_name)

model = MarianMTModel.from_pretrained(model_name)

sentences = [

'Natural Language Processing is a field of AI.',

'I love learning machine learning every day.',

'The transformer model changed everything.',

# Tokenize and translate

inputs = tokenizer(sentences, return_tensors='pt', padding=True)

translated = [Link](**inputs)

results = tokenizer.batch_decode(translated, skip_special_tokens=True)

for src, tgt in zip(sentences, results):

print(f'EN: {src}')

print(f'FR: {tgt}\n')

# Google Translate API alternative

!pip install googletrans==4.0.0rc1

from googletrans import Translator

translator = Translator()

result = [Link]('Hello, how are you?', dest='ta') # Tamil

print('Tamil:', [Link])

Real-World Example — Google Translate (200+ Languages)

Google Translate handles 100 billion words per day across 133 languages. When a Tamil-speaking
customer contacts an international company's support portal, MT instantly translates the query to English
for the agent and translates the response back to Tamil.

NLP Complete Study Guide | Page 23

 3.5 Text Summarization

Definition
Text Summarization automatically produces a short, coherent summary of a longer document. There
are two main approaches:
• Extractive summarization — selects and returns the most important sentences from the original text
as-is
• Abstractive summarization — generates new sentences that paraphrase and condense the original
(uses seq2seq models)

Google Colab Code

!pip install transformers sumy

# Abstractive summarization with BART

from transformers import pipeline

summarizer = pipeline('summarization', model='facebook/bart-large-cnn')

article = '''

Artificial intelligence is rapidly transforming industries worldwide.

From healthcare where AI assists in diagnosis, to finance where it detects fraud,

the applications are endless. Natural language processing, a subset of AI,

enables machines to read and understand human language. Companies like Google,

Microsoft, and OpenAI invest billions in NLP research. Recent advances in

transformer models have achieved human-level performance on many benchmarks.

The future of AI looks extremely promising with breakthroughs happening every month
.

'''

summary = summarizer(article, max_length=80, min_length=30, do_sample=False)

print('ABSTRACTIVE SUMMARY:')

print(summary[0]['summary_text'])

# Extractive summarization with sumy

from [Link] import PlaintextParser

from [Link] import Tokenizer

from [Link].lex_rank import LexRankSummarizer

parser = PlaintextParser.from_string(article, Tokenizer('english'))

summarizer_lex = LexRankSummarizer()

extractive = summarizer_lex([Link], 2) # 2 sentences

print('\nEXTRACTIVE SUMMARY:')

for sentence in extractive:

print(str(sentence))

NLP Complete Study Guide | Page 24

 Real-World Example — News Aggregator Apps (Inshorts, Flipboard)
Inshorts (Indian news app) summarises every news article in exactly 60 words using text summarization
models. Reuters uses abstractive summarization to generate headlines automatically from long wire
reports before human editors review them.

3.6 Dependency Parsing

Definition
Dependency Parsing analyses the grammatical structure of a sentence by identifying relationships
(dependencies) between words. Each word is connected to a head word by a labelled arc showing the
syntactic relationship — subject, object, modifier, etc. The result is a dependency tree.

Google Colab Code

import spacy

from spacy import displacy

nlp = [Link]('en_core_web_sm')

sentence = 'The clever student solved the difficult problem quickly'

doc = nlp(sentence)

print(f'{'Token':<15} {'Head':<15} {'Dep Relation':<15} {'POS'}')

print('-' * 60)

for token in doc:

print(f'{[Link]:<15} {[Link]:<15} {token.dep_:<15} {token.pos_}')

# Output structure:

# student -> solved (nsubj) — student is subject of solved

# problem -> solved (dobj) — problem is direct object of solved

# clever -> student (amod) — clever modifies student

# Extract subject and object programmatically

for token in doc:

if token.dep_ == 'nsubj': print(f'Subject: {[Link]}')

if token.dep_ == 'dobj': print(f'Object: {[Link]}')

if token.dep_ == 'ROOT': print(f'Main verb: {[Link]}')

Real-World Example — Customer Query Understanding

When a user asks 'Cancel my last order from Chennai', dependency parsing identifies 'Cancel' as the root
verb, 'order' as the object, and 'Chennai' as a location modifier. This helps chatbots understand both the
action and its target accurately.

NLP Complete Study Guide | Page 25

 3.7 Question Answering (QA)

Definition
Question Answering is an NLP task where a system automatically answers questions posed in natural
language. Two main types:
• Extractive QA — finds and extracts the answer span directly from a given passage (BERT-based
models)
• Generative QA — generates a new answer using language models (GPT, T5)

Google Colab Code

from transformers import pipeline

# Extractive QA with BERT

qa_pipeline = pipeline('question-answering',

model='distilbert-base-cased-distilled-squad')

context = '''

Natural Language Processing (NLP) is a subfield of linguistics, computer science,

and artificial intelligence concerned with the interactions between computers and

human language. The goal of NLP is to enable computers to process, understand,

and generate human language in a way that is both meaningful and useful.

Key NLP tasks include text classification, named entity recognition, and translatio
n.

BERT, developed by Google in 2018, is a landmark model in NLP history.

'''

questions = [

'What is NLP?',

'What are key NLP tasks?',

'Who developed BERT and when?',

for q in questions:

result = qa_pipeline(question=q, context=context)

print(f'Q: {q}')

print(f'A: {result["answer"]} (confidence: {result["score"]:.2f})')

print()

Real-World Example — Customer Support Chatbots

Flipkart's customer chatbot uses QA models trained on FAQs and policy documents. When a customer
asks 'What is the return policy for electronics?', the model scans the policy document and extracts the
exact answer span in milliseconds.

3.8 Information Extraction

NLP Complete Study Guide | Page 26

Definition
Information Extraction (IE) is the task of automatically extracting structured information from
unstructured text. This includes extracting named entities, relations between entities, events, and facts.
IE converts free-form text into structured data (tables, knowledge graphs).

Google Colab Code

import spacy

from spacy import displacy

nlp = [Link]('en_core_web_sm')

# Relation extraction: who works for whom?

texts = [

'Sundar Pichai is the CEO of Google.',

'Elon Musk founded Tesla and SpaceX.',

'Sam Altman is the CEO of OpenAI.',

print('Extracted Relations:')

for text in texts:

doc = nlp(text)

persons = [e for e in [Link] if e.label_ == 'PERSON']

orgs = [e for e in [Link] if e.label_ == 'ORG']

if persons and orgs:

for p in persons:

for o in orgs:

print(f' {[Link]} ---[associated_with]---> {[Link]}')

# Event extraction

text2 = 'Google acquired YouTube for $1.65 billion in 2006.'

doc2 = nlp(text2)

print('\nEvent Extraction:')

print('Entities:', [([Link], e.label_) for e in [Link]])

Real-World Example — Medical Record Processing

Hospitals use IE to extract diagnoses, medications, and dosages from doctors' free-text notes. A note like
'Patient was prescribed Metformin 500mg twice daily for Type 2 Diabetes' is automatically structured into
database fields: Drug=Metformin, Dose=500mg, Frequency=BID, Condition=T2DM.

NLP Complete Study Guide | Page 27

 LEVEL 4 — DEEP LEARNING FOR NLP

Deep learning transformed NLP. This level covers neural architectures — from recurrent networks that
process sequences, to the Transformer that powers modern AI systems.

4.1 Recurrent Neural Networks (RNN)

Definition
An RNN is a neural network designed to handle sequential data. Unlike feedforward networks, RNNs
have a hidden state (memory) that is updated at each timestep, allowing information from previous
tokens to influence the current prediction. Used for text generation, language modelling, and sentiment
analysis.
Problem: RNNs suffer from the vanishing gradient problem — gradients become very small during
backpropagation through many time steps, making it hard to learn long-range dependencies.

Google Colab Code

import torch

import [Link] as nn

import numpy as np

# Simple RNN for sentiment classification

class SimpleRNN([Link]):

def __init__(self, vocab_size, embed_dim, hidden_dim, output_dim):

super().__init__()

[Link] = [Link](vocab_size, embed_dim)

[Link] = [Link](embed_dim, hidden_dim, batch_first=True)

[Link] = [Link](hidden_dim, output_dim)

def forward(self, x):

embedded = [Link](x) # (batch, seq, embed)

out, hidden = [Link](embedded) # hidden: (1, batch, hidden)

return [Link]([Link](0)) # (batch, output)

# Hyperparameters

VOCAB_SIZE = 10000

EMBED_DIM = 64

HIDDEN_DIM = 128

OUTPUT_DIM = 2 # positive / negative

model = SimpleRNN(VOCAB_SIZE, EMBED_DIM, HIDDEN_DIM, OUTPUT_DIM)

print('RNN Model:')

print(model)

print(f'Trainable parameters: {sum([Link]() for p in [Link]()):,}')

NLP Complete Study Guide | Page 28

 # Forward pass with dummy data

dummy_input = [Link](0, VOCAB_SIZE, (4, 20)) # batch=4, seq_len=20

output = model(dummy_input)

print('Output shape:', [Link]) # (4, 2)

Real-World Example — Language Modelling for Autocomplete

Early autocomplete systems in smartphones (pre-2017) used RNNs trained on billions of text messages.
The hidden state carried context from previous words to predict the next word in your sentence.

4.2 Long Short-Term Memory (LSTM)

Definition
LSTM is an advanced RNN architecture with gating mechanisms that solve the vanishing gradient
problem. It has three gates:
• Forget gate — decides what information to discard from cell state
• Input gate — decides what new information to store in cell state
• Output gate — decides what to output based on cell state
LSTMs can learn long-range dependencies (e.g., linking a pronoun to its antecedent 50 words earlier).
They were the dominant NLP architecture before Transformers.

Google Colab Code

import torch

import [Link] as nn

class LSTMClassifier([Link]):

def __init__(self, vocab_size, embed_dim, hidden_dim, n_layers, output_dim, dro

pout=0.3):

super().__init__()

[Link] = [Link](vocab_size, embed_dim, padding_idx=0)

[Link] = [Link](embed_dim, hidden_dim,

num_layers=n_layers,

batch_first=True,

bidirectional=True,

dropout=dropout)

[Link] = [Link](dropout)

[Link] = [Link](hidden_dim * 2, output_dim) # *2 for bidirectional

def forward(self, x):

embedded = [Link]([Link](x))

output, (hidden, cell) = [Link](embedded)

# Concatenate final forward and backward hidden states

hidden = [Link]([hidden[-2], hidden[-1]], dim=1)

return [Link]([Link](hidden))

NLP Complete Study Guide | Page 29

model = LSTMClassifier(

vocab_size=10000, embed_dim=128, hidden_dim=256,

n_layers=2, output_dim=2, dropout=0.3

print(model)

dummy = [Link](0, 10000, (8, 50)) # batch=8, seq=50

print('Output:', model(dummy).shape) # (8, 2)

Real-World Example — Speech Recognition (Siri, Alexa)

Amazon Alexa originally used Bidirectional LSTMs for converting speech to text. The bidirectional nature
means the model reads your spoken sentence both forwards and backwards, understanding context from
both directions before transcribing any word.

NLP Complete Study Guide | Page 30

 4.3 Attention Mechanism

Definition
The Attention Mechanism allows neural networks to focus on the most relevant parts of the input when
producing each output token. Instead of compressing the entire input into one fixed vector (as Seq2Seq
did), attention creates a weighted sum of all encoder hidden states at each decoding step. Originally
proposed by Bahdanau et al. (2015) for machine translation.
Self-Attention extends this so each word in the input attends to all other words in the same sequence —
the foundation of Transformers.

Google Colab Code

import torch

import [Link] as nn

import [Link] as F

import [Link] as plt

class BahdanauAttention([Link]):

def __init__(self, hidden_dim):

super().__init__()

self.W1 = [Link](hidden_dim, hidden_dim)

self.W2 = [Link](hidden_dim, hidden_dim)

self.V = [Link](hidden_dim, 1)

def forward(self, query, values):

# query: (batch, hidden), values: (batch, seq, hidden)

query = [Link](1) # (batch, 1, hidden)

score = self.V([Link](self.W1(query) + self.W2(values))) # (batch, seq

, 1)

weights = [Link](score, dim=1) # attention weights

context = [Link]([Link](1,2), values) # weighted sum

return [Link](1), [Link](-1)

# Self-Attention (simplified Transformer style)

class SelfAttention([Link]):

def __init__(self, embed_dim, n_heads=4):

super().__init__()

[Link] = [Link](embed_dim, n_heads, batch_first=True)

def forward(self, x):

out, weights = [Link](x, x, x) # Q=K=V=x (self-attention)

return out, weights

self_attn = SelfAttention(embed_dim=64, n_heads=4)

x = [Link](2, 10, 64) # batch=2, seq=10, embed=64

out, w = self_attn(x)

NLP Complete Study Guide | Page 31

 print('Output shape:', [Link]) # (2, 10, 64)

print('Weight shape:', [Link]) # (2, 10, 10) — each token attends to all

Real-World Example — Google Neural Machine Translation

When translating 'The animal didn't cross the street because it was too tired' to French, the attention
mechanism correctly identifies that 'it' refers to 'animal' (not 'street') by assigning high attention weight
from the token 'it' to 'animal'.

4.4 Transformer Architecture

Definition
The Transformer (Vaswani et al., 2017 — 'Attention Is All You Need') is an architecture that replaces
RNNs entirely with self-attention and feedforward layers. It processes all tokens in parallel (not
sequentially), enabling massive speedup and the ability to train on huge datasets. All modern LLMs
(BERT, GPT, T5, LLaMA) are built on this architecture.

Key Components
• Multi-Head Self-Attention — each token attends to all others; multiple heads capture different
relationship types
• Feedforward Network — applied to each position independently after attention
• Layer Normalisation — stabilises training
• Positional Encoding — injects order information since attention is order-agnostic
• Encoder — processes input (used in BERT for understanding tasks)
• Decoder — generates output (used in GPT for generation tasks)

Google Colab Code

import torch

import [Link] as nn

class TransformerBlock([Link]):

def __init__(self, embed_dim, n_heads, ff_dim, dropout=0.1):

super().__init__()

[Link] = [Link](embed_dim, n_heads, batch_first=True

)

self.norm1 = [Link](embed_dim)

self.norm2 = [Link](embed_dim)

[Link] = [Link](

[Link](embed_dim, ff_dim),

[Link](),

[Link](dropout),

[Link](ff_dim, embed_dim),

[Link](dropout),

NLP Complete Study Guide | Page 32

 [Link] = [Link](dropout)

def forward(self, x):

# Multi-head self-attention + residual + norm

attn_out, _ = [Link](x, x, x)

x = self.norm1(x + [Link](attn_out))

# Feedforward + residual + norm

ff_out = [Link](x)

x = self.norm2(x + ff_out)

return x

# Stack multiple transformer blocks

class MiniTransformer([Link]):

def __init__(self, vocab_size, embed_dim, n_heads, ff_dim, n_layers, n_classes)

:

super().__init__()

[Link] = [Link](vocab_size, embed_dim)

[Link] = [Link]([

TransformerBlock(embed_dim, n_heads, ff_dim) for _ in range(n_layers)])

[Link] = [Link](embed_dim, n_classes)

def forward(self, x):

x = [Link](x)

for layer in [Link]:

x = layer(x)

return [Link]([Link](dim=1)) # mean pooling over sequence

model = MiniTransformer(10000, 128, 4, 256, 3, 2)

out = model([Link](0, 10000, (4, 50)))

print('Output:', [Link]) # (4, 2)

Real-World Example — ChatGPT / Google Bard

ChatGPT is a Transformer model with 175 billion parameters (GPT-3). Every response you receive is
generated token by token using the Transformer decoder, where each new word attends to all previous
words in the conversation context.

NLP Complete Study Guide | Page 33

 4.5 Positional Encoding

Definition
Since Transformers process all tokens in parallel (no sequential order), they have no inherent sense of
word position. Positional Encoding injects position information into token embeddings using sinusoidal
functions of different frequencies. Each position gets a unique pattern of sine and cosine values that the
model can use to determine word order.
Formula: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)); PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Google Colab Code

import torch

import [Link] as nn

import math

import [Link] as plt

import numpy as np

class PositionalEncoding([Link]):

def __init__(self, d_model, max_len=512, dropout=0.1):

super().__init__()

[Link] = [Link](dropout)

pe = [Link](max_len, d_model)

position = [Link](0, max_len).unsqueeze(1).float()

div_term = [Link]([Link](0, d_model, 2).float() *

-([Link](10000.0) / d_model))

pe[:, 0::2] = [Link](position * div_term) # even dimensions

pe[:, 1::2] = [Link](position * div_term) # odd dimensions

self.register_buffer('pe', [Link](0)) # (1, max_len, d_model)

def forward(self, x):

x = x + [Link][:, :[Link](1)]

return [Link](x)

# Visualise positional encodings

pe = PositionalEncoding(d_model=64, max_len=50)

dummy = [Link](1, 50, 64)

pos_enc = [Link][0].detach().numpy()

[Link](figsize=(12, 5))

[Link](pos_enc, aspect='auto', cmap='RdBu')

[Link]()

[Link]('Embedding Dimension')

[Link]('Position in Sequence')

[Link]('Positional Encoding Heatmap')

[Link]('positional_encoding.png', dpi=100, bbox_inches='tight')

NLP Complete Study Guide | Page 34

[Link]()

print('PE shape:', pos_enc.shape) # (50, 64)

Real-World Example — Sentence Order Understanding

'The cat chased the mouse' and 'The mouse chased the cat' have the same words but opposite
meanings. Positional encoding ensures the Transformer knows 'cat' is at position 1 and 'mouse' at
position 4, correctly interpreting who chased whom.

NLP Complete Study Guide | Page 35

 LEVEL 5 — ADVANCED NLP & LLMs

This is where modern AI products live. Large Language Models, fine-tuning, RAG, and prompt engineering
are the skills driving the current AI industry revolution.

5.1 BERT (Bidirectional Encoder Representations from Transformers)

Definition
BERT (Google, 2018) is a pre-trained Transformer encoder that reads text bidirectionally — it considers
both left and right context simultaneously for every token. Pre-trained on masked language modelling
(predict masked words) and next sentence prediction on 3.3 billion words. BERT revolutionised NLP
benchmarks and is fine-tuned for classification, NER, QA, and more.

Google Colab Code

!pip install transformers torch

from transformers import BertTokenizer, BertForSequenceClassification

from transformers import pipeline

import torch

# Zero-shot with BERT pipeline (sentiment)

classifier = pipeline('sentiment-analysis',

model='nlptown/bert-base-multilingual-uncased-sentiment')

texts = [

'This NLP course is absolutely excellent!',

'I found the material very confusing and unhelpful.',

'The content is okay, nothing exceptional.',

for text in texts:

result = classifier(text)[0]

print(f'{result["label"]} | {result["score"]:.3f} | {text[:50]}')

# BERT Embeddings for semantic similarity

from transformers import BertModel, BertTokenizer

import [Link] as F

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

bert = BertModel.from_pretrained('bert-base-uncased')

def get_embedding(text):

inputs = tokenizer(text, return_tensors='pt', max_length=128, truncation=True)

with torch.no_grad():

output = bert(**inputs)

return output.last_hidden_state[:, 0, :] # [CLS] token embedding

NLP Complete Study Guide | Page 36

s1 = get_embedding('I love playing cricket')

s2 = get_embedding('Cricket is my favourite sport')

s3 = get_embedding('The weather is very hot today')

print('Sim(s1,s2):', F.cosine_similarity(s1, s2).item())

print('Sim(s1,s3):', F.cosine_similarity(s1, s3).item())

Real-World Example — Google Search (2019 Update)

In October 2019, Google deployed BERT in its search engine, calling it 'the biggest leap forward in the
past five years'. BERT helped Google understand search queries like "can you get medicine for someone
pharmacy" correctly interpreting 'for someone' as the key context — improving 10% of all English
searches.

NLP Complete Study Guide | Page 37

 5.2 GPT and Large Language Models (LLMs)

Definition
GPT (Generative Pre-trained Transformer) is a Transformer decoder trained autoregressively —
predicting the next token given all previous tokens. Unlike BERT (bidirectional encoder), GPT reads
left-to-right. Scaled up dramatically: GPT-1 (117M params) → GPT-2 (1.5B) → GPT-3 (175B) → GPT-4
(estimated 1T+). These are the foundation of ChatGPT, Claude, and Gemini.

Google Colab Code

from transformers import GPT2LMHeadModel, GPT2Tokenizer

import torch

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

model = GPT2LMHeadModel.from_pretrained('gpt2')

[Link]()

def generate_text(prompt, max_new_tokens=100, temperature=0.8, top_p=0.9):

inputs = [Link](prompt, return_tensors='pt')

with torch.no_grad():

outputs = [Link](

inputs,

max_new_tokens=max_new_tokens,

temperature=temperature,

top_p=top_p,

do_sample=True,

pad_token_id=tokenizer.eos_token_id

return [Link](outputs[0], skip_special_tokens=True)

# Generate continuations

prompts = [

'Natural Language Processing is',

'The future of artificial intelligence will',

for prompt in prompts:

print(f'Prompt: {prompt}')

print(f'Generated: {generate_text(prompt, max_new_tokens=60)}')

print()

Real-World Example — ChatGPT (OpenAI)

ChatGPT, based on GPT-4, crossed 100 million users in just 2 months — the fastest growing app in
history. It uses the Transformer decoder to generate responses token-by-token, sampling from a
probability distribution over the vocabulary at each step, shaped by temperature and top-p sampling.

NLP Complete Study Guide | Page 38

 5.3 Fine-tuning & PEFT (LoRA)

Definition
Fine-tuning takes a pre-trained model and continues training it on a smaller task-specific dataset,
adapting its weights for a specific application. Full fine-tuning updates all parameters (expensive). PEFT
(Parameter-Efficient Fine-Tuning) methods like LoRA (Low-Rank Adaptation) add small trainable
matrices to frozen layers, reducing trainable parameters by 99% while achieving similar performance.

Google Colab Code

!pip install transformers peft datasets

from transformers import AutoModelForSequenceClassification, AutoTokenizer

from peft import LoraConfig, get_peft_model, TaskType

# Load pre-trained BERT

model_name = 'bert-base-uncased'

model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2

)

tokenizer = AutoTokenizer.from_pretrained(model_name)

# Apply LoRA — only train small adapter matrices

lora_config = LoraConfig(

task_type=TaskType.SEQ_CLS,

r=8, # rank of the low-rank matrices

lora_alpha=32, # scaling factor

lora_dropout=0.1,

target_modules=['query', 'value'] # which layers to adapt

lora_model = get_peft_model(model, lora_config)

# Compare parameter counts

total = sum([Link]() for p in lora_model.parameters())

trainable = sum([Link]() for p in lora_model.parameters() if p.requires_grad)

print(f'Total parameters: {total:,}')

print(f'Trainable parameters: {trainable:,}')

print(f'Reduction: {100*(1 - trainable/total):.1f}% fewer parameters to train')

# Typically shows 99%+ reduction in trainable params

Real-World Example — Domain-Specific AI Assistants

A hospital fine-tunes BERT on medical records using LoRA. Instead of training 110M parameters, only
~1M adapter weights are trained. The resulting model understands medical jargon like 'myocardial
infarction' and 'HbA1c' at near-specialist level, deployed on a hospital laptop without GPU clusters.

NLP Complete Study Guide | Page 39

 5.4 RAG (Retrieval-Augmented Generation)

Definition
RAG combines information retrieval with language generation. Instead of relying solely on the LLM's
trained knowledge (which has a cutoff date), RAG first retrieves relevant documents from an external
knowledge base, then passes them as context to the LLM to generate a grounded answer. This reduces
hallucinations and enables up-to-date responses.

RAG Pipeline
• Step 1 (Indexing) — documents are split into chunks, embedded, and stored in a vector database
• Step 2 (Retrieval) — user query is embedded; top-k similar chunks are retrieved
• Step 3 (Generation) — retrieved chunks + query are given to LLM as context to generate answer

Google Colab Code

!pip install transformers sentence-transformers faiss-cpu

from sentence_transformers import SentenceTransformer

from transformers import pipeline

import numpy as np, faiss

# ■■ Knowledge Base (your documents) ■■

documents = [

'Python was created by Guido van Rossum and released in 1991.',

'NLP stands for Natural Language Processing, a field of AI.',

'BERT was developed by Google and published in 2018.',

'Transformers use self-attention to process sequences in parallel.',

'GPT-4 is a large language model created by OpenAI in 2023.',

'LLMs are trained on massive text datasets using self-supervised learning.',

# ■■ Step 1: Encode documents into vectors ■■

embedder = SentenceTransformer('all-MiniLM-L6-v2')

doc_embeddings = [Link](documents, convert_to_numpy=True)

# ■■ Step 2: Build FAISS vector index ■■

dim = doc_embeddings.shape[1]

index = faiss.IndexFlatL2(dim)

[Link](doc_embeddings)

def retrieve(query, top_k=2):

q_vec = [Link]([query], convert_to_numpy=True)

_, idxs = [Link](q_vec, top_k)

return [documents[i] for i in idxs[0]]

# ■■ Step 3: RAG — retrieve + generate ■■

qa = pipeline('question-answering',

NLP Complete Study Guide | Page 40

 model='distilbert-base-cased-distilled-squad')

query = 'When was BERT developed and by whom?'

retrieved = retrieve(query)

context = ' '.join(retrieved)

print('Retrieved context:', context)

answer = qa(question=query, context=context)

print('Answer:', answer['answer'])

Real-World Example — Enterprise Chatbots (Internal Knowledge Base)

Infosys built an internal RAG system where employees ask questions like 'What is the HR leave policy for
paternity?' The system retrieves relevant HR policy PDF chunks, feeds them to an LLM, and generates a
precise answer — always citing the source document, preventing hallucinations.

NLP Complete Study Guide | Page 41

 5.5 Prompt Engineering

Definition
Prompt Engineering is the practice of designing and optimising input text (prompts) to guide LLMs to
produce desired outputs. Since LLMs are sensitive to how questions are phrased, careful prompt design
can dramatically improve output quality without changing model weights.

Key Techniques
• Zero-shot prompting — ask directly without examples: 'Classify this review as positive or negative:
...'
• Few-shot prompting — provide 2-5 examples before your actual request
• Chain-of-Thought (CoT) — instruct the model to 'think step by step' for reasoning tasks
• Role prompting — 'You are an expert data scientist. Explain...'
• Output format control — 'Respond only in JSON with keys: sentiment, confidence'

Google Colab Code

# Use OpenAI API or HuggingFace for this demo

# Here we demonstrate prompt patterns with a local pipeline

from transformers import pipeline

# Load a text generation model

gen = pipeline('text-generation', model='gpt2', max_new_tokens=80)

# ■■ Zero-shot prompt ■■

zero_shot = 'Classify the sentiment (positive/negative): "I love this product!" Sen
timent:'

print('Zero-shot:', gen(zero_shot)[0]['generated_text'])

# ■■ Few-shot prompt ■■

few_shot = '''

Review: 'Great quality, fast delivery!' -> positive

Review: 'Broke after one day, very disappointing' -> negative

Review: 'Absolutely love it, buying again!' -> '''

print('Few-shot:', gen(few_shot)[0]['generated_text'][-20:])

# ■■ Chain-of-thought prompt ■■

cot = '''Solve step by step.

Problem: A customer bought 3 items at Rs 250 each and paid Rs 800.

How much change should they receive?

Step 1:'''

print('CoT:', gen(cot, max_new_tokens=100)[0]['generated_text'])

# ■■ Role prompt ■■

role = '''You are an expert NLP engineer. Explain tokenization in one sentence:'''

print('Role:', gen(role)[0]['generated_text'])

NLP Complete Study Guide | Page 42

 Real-World Example — GitHub Copilot
GitHub Copilot uses prompt engineering under the hood. When you write a Python function signature,
Copilot constructs a prompt including your code context, file language, and surrounding functions, then
sends it to an LLM (Codex) to generate the function body.

5.6 Semantic Search & Vector Databases

Definition
Semantic Search finds documents based on meaning rather than keyword matching. Text is converted
into dense vectors (embeddings), stored in a vector database, and queries are matched using cosine or
dot-product similarity. 'Cars' can match 'automobiles' and 'vehicles' even if the exact word doesn't appear.

Google Colab Code

!pip install sentence-transformers faiss-cpu

from sentence_transformers import SentenceTransformer

import numpy as np, faiss

model = SentenceTransformer('all-MiniLM-L6-v2')

# Document corpus

docs = [

'How to cook pasta at home',

'Python programming for beginners',

'Machine learning model evaluation metrics',

'Italian food recipes and ingredients',

'Deep learning with PyTorch tutorial',

'Car maintenance and engine repair guide',

'Neural network training best practices',

# Encode and index

doc_vecs = [Link](docs, convert_to_numpy=True)

index = [Link](doc_vecs.shape[1]) # Inner Product = cosine sim

faiss.normalize_L2(doc_vecs)

[Link](doc_vecs)

def semantic_search(query, top_k=3):

q_vec = [Link]([query], convert_to_numpy=True)

faiss.normalize_L2(q_vec)

scores, idxs = [Link](q_vec, top_k)

return [(docs[i], scores[0][j]) for j, i in enumerate(idxs[0])]

# Test: semantically different query words

queries = ['automobile repair', 'noodle dish preparation', 'AI model training']

for q in queries:

NLP Complete Study Guide | Page 43

 print(f'Query: {q}')

for doc, score in semantic_search(q):

print(f' {score:.3f} | {doc}')

print()

Real-World Example — LinkedIn Job Search

LinkedIn's job recommendation engine uses semantic search. When you search 'data analysis role
Chennai', it finds postings containing 'business intelligence analyst' and 'data insights specialist' that don't
contain your exact words but are semantically similar. Vector databases like Pinecone and Weaviate
power this at scale.

NLP Complete Study Guide | Page 44

 5.7 RLHF (Reinforcement Learning from Human Feedback)

Definition
RLHF is a training technique that aligns LLMs with human preferences by using human feedback as a
reward signal. It was used to train ChatGPT and Claude. The process has three stages:
• Stage 1 — Supervised Fine-Tuning (SFT): Fine-tune the base LLM on high-quality human-written
demonstrations
• Stage 2 — Reward Model Training: Train a model to predict human preference scores from pairs of
model outputs that humans ranked
• Stage 3 — PPO Optimisation: Use Proximal Policy Optimisation (PPO) RL to fine-tune the LLM to
maximise the reward model's score

Google Colab Code (Conceptual Demo)

# RLHF full implementation requires significant compute.

# This demo shows the reward model concept using preference data.

!pip install transformers datasets trl

from trl import RewardTrainer, RewardConfig

from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Simulated preference dataset

# Each item: (prompt, chosen_response, rejected_response)

preference_data = [

'prompt': 'Explain NLP',

'chosen': 'NLP is Natural Language Processing — teaching computers to understa

nd human language.',

'rejected': 'NLP means things. Language stuff happens.'

},

'prompt': 'What is BERT?',

'chosen': 'BERT is a bidirectional Transformer model pre-trained by Google in

2018.',

'rejected': 'BERT is some kind of language model I think.'

},

# In real RLHF, human annotators compare model outputs and rank them.

# The reward model learns to predict: which response would a human prefer?

# Training signal for PPO:

# reward = reward_model(prompt + chosen) > reward_model(prompt + rejected)

# The LLM is then fine-tuned with PPO to maximise reward_model score,

# making it more helpful, harmless, and honest (HHH alignment).

NLP Complete Study Guide | Page 45

 print('RLHF stages:')

print('1. SFT — fine-tune on human demonstrations')

print('2. RM — train reward model on preference pairs')

print('3. PPO — optimise LLM using reward model as signal')

Real-World Example — ChatGPT Alignment (OpenAI)

OpenAI hired human contractors to rate thousands of GPT outputs for helpfulness and safety. These
ratings trained a reward model, which was then used to guide PPO training. The result was ChatGPT —
an LLM that refuses harmful requests and gives more helpful, structured responses than the raw GPT-3.

5.8 Multimodal NLP (Vision + Language)

Definition
Multimodal NLP combines text with other modalities — images, audio, video — enabling models to
understand and generate content across multiple formats. Models like CLIP (image-text matching),
DALL-E (text-to-image), and GPT-4V (vision) process both language and visual information
simultaneously.

Google Colab Code

!pip install transformers torch Pillow requests

from transformers import CLIPProcessor, CLIPModel

from PIL import Image

import requests, torch

# CLIP: match images to text descriptions

model = CLIPModel.from_pretrained('openai/clip-vit-base-patch32')

processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')

# Load a sample image from URL

url = '[Link]

image = [Link]([Link](url, stream=True).raw)

# Candidate text descriptions

texts = ['a photo of cats', 'a photo of dogs',

'a photo of a car', 'people playing football']

# Process and compute similarity

inputs = processor(text=texts, images=image,

return_tensors='pt', padding=True)

with torch.no_grad():

outputs = model(**inputs)

# Softmax over text-image similarity scores

probs = outputs.logits_per_image.softmax(dim=1)

print('Image-Text Match Probabilities:')

for text, prob in zip(texts, probs[0]):

NLP Complete Study Guide | Page 46

 print(f' {prob:.3f} — {text}')

# Highest probability = best matching description

Real-World Example — Google Lens / Pinterest Visual Search

Google Lens uses multimodal NLP: you point your camera at a plant and it understands both the visual
features (leaf shape, colour, texture) and cross-references them with text descriptions from botanical
databases, identifying the species and providing cultivation tips.

NLP Complete Study Guide | Page 47

 QUICK REFERENCE — All Concepts at a Glance

Level Concept Key Idea Library / Tool

1 Tokenization Split text into tokens NLTK, HuggingFace

1 Stemming Cut to root form (rough) NLTK PorterStemmer

1 Lemmatization Dictionary base form NLTK WordNetLemmatizer

1 Stop Words Remove noise words NLTK, spaCy

1 POS Tagging Label parts of speech NLTK, spaCy

2 Bag of Words Word count vectors sklearn CountVectorizer

2 TF-IDF Weighted word importance sklearn TfidfVectorizer

2 N-grams Word sequence combos NLTK ngrams

2 Word2Vec Dense semantic vectors gensim Word2Vec

2 GloVe Global co-occurrence vectors Stanford GloVe

2 FastText Subword char n-gram vecs gensim FastText

3 Sentiment Analysis Positive / Negative opinion VADER, HuggingFace

3 NER Find entities in text spaCy, HuggingFace

3 Text Classification Assign labels to docs sklearn, transformers

3 Machine Translation Translate languages MarianMT, googletrans

3 Summarisation Shorten long text BART, sumy

3 Dependency Parsing Sentence structure tree spaCy

3 Question Answering Auto-answer from context DistilBERT SQuAD

3 Info Extraction Structured facts from text spaCy, OpenIE

4 RNN Sequential neural memory PyTorch [Link]

4 LSTM Gated long-range memory PyTorch [Link]

4 Attention Weighted focus on input PyTorch MultiheadAttn

4 Transformer Parallel self-attention arch PyTorch, HuggingFace

4 Positional Encoding Inject sequence order Custom / built-in

5 BERT Bidirectional encoder LM HuggingFace BERT

5 GPT / LLMs Autoregressive generation HuggingFace GPT2

5 Fine-tuning / LoRA Adapt pre-trained model HuggingFace PEFT

5 RAG Retrieve + generate FAISS + transformers

5 Prompt Engineering Guide LLMs with text Any LLM API

5 Semantic Search Meaning-based retrieval SentenceTransformers

5 RLHF Align LLM with humans TRL library

5 Multimodal NLP Vision + language CLIP, GPT-4V

NLP Complete Study Guide | Page 48