1. What is Natural Language Processing?

Natural Language Processing is a field of artificial intelligence that sits at the intersection of
computer science, linguistics, and machine learning. Its goal is to give computers the ability
to understand, interpret, and manipulate human language (both text and speech) in a way that
is meaningful.
Human language is inherently messy, ambiguous, and full of complex rules, idioms, and
metaphors, making NLP one of the most challenging areas of AI.
2. The Core NLP Pipeline (Text Preprocessing)
Before an AI model can analyze text, the raw strings must be cleaned and broken down into
structured, machine-readable formats. This is called preprocessing.
●​ Tokenization: Breaking down a continuous string of text into individual units, called
tokens (words, characters, or sub-words).
○​ Example: "I love NLP" ➔ ["I", "love", "NLP"]
●​ Lowercasing: Converting all text to lowercase so the model treats words like "Machine"
and "machine" identically.
●​ Stop Word Removal: Stripping out common words that carry little unique meaning or
semantic weight (e.g., "and", "the", "is", "at").
●​ Stemming & Lemmatization: Reducing words to their base or root form.
○​ Stemming: Uses crude rules to chop off word endings (e.g., "running", "runs" ➔ "run";
"studies" ➔ "studi").
○​ Lemmatization: Uses a dictionary to find the linguistically correct root word, or lemma
(e.g., "better" ➔ "good"; "studies" ➔ "study").
●​ Part-of-Speech (POS) Tagging: Identifying the grammatical role of each token in a
sentence (e.g., labeling a word as a noun, verb, adjective, etc.).
3. Text Representation (Vectorization)
Machines cannot read words; they can only process numbers. Vectorization is the process of
converting text into numerical vectors.
A. Bag-of-Words (BoW)

A simple approach that counts how many times each word appears in a document. It
completely ignores word order and grammar, treating a document as a literal "bag" of individual
words.
B. TF-IDF (Term Frequency-Inverse Document Frequency)

An improvement over Bag-of-Words. It evaluates how important a word is to a specific

Advanced techniques that map words into a multi-dimensional vector space. Words with
similar meanings or contexts are placed close together in this mathematical space.
●​ Famous intuition: $\text{Vector}(\text{"King"}) - \text{Vector}(\text{"Man"}) +
\text{Vector}(\text{"Woman"}) \approx \text{Vector}(\text{"Queen"})$
4. Key NLP Tasks and Applications
NLP powers dozens of applications we use every day, categorized by their underlying tasks:
●​ Sentiment Analysis: Determining whether a piece of text expresses a positive, negative,
or neutral opinion (e.g., analyzing product reviews or social media trends).
●​ Named Entity Recognition (NER): Locating and classifying key entities in text into
predefined categories such as names of people, organizations, locations, dates, or
monetary values.
●​ Machine Translation: Automatically translating text or speech from one human language
to another (e.g., Google Translate).
●​ Text Summarization: Condensing a long document into a short paragraph or bulleted list
while retaining key information.
●​ Intent Detection & Chatbots: Figuring out what a user is trying to accomplish from their
query to trigger the correct automated response.
5. Evolution of NLP Models
The algorithms used to process natural language have evolved dramatically:
1.​ Rule-Based & Statistical Models (Pre-2010s): Relied on hardcoded linguistic rules,
regular expressions, and statistical probabilities (like Naive Bayes).
2.​ Recurrent Neural Networks & LSTMs (2010s): Introduced deep learning to NLP.
Because text is sequential, RNNs and LSTMs were used to process text word-by-word,
keeping track of memory from earlier parts of a sentence.
3.​ Transformers & Large Language Models (LLMs) (2017–Present): The modern era. Built
on the Attention Mechanism, Transformers process entire blocks of text simultaneously
rather than word-by-word. This breakthrough allowed models to scale massively, leading
to modern systems like GPT-4, Claude, and Gemini, which can generate human-like
prose, write code, and reason through complex text.