What is NLP (Natural Language Processing)?
Natural Language Processing (NLP) is a field of artificial intelligence (AI) that
focuses on enabling computers to understand, interpret, generate, and interact with
human language (natural language) in a meaningful way.
Goals of NLP
Understand human language (text or speech)
Extract meaning from text or audio
Generate human-like text or responses
Facilitate communication between humans and machines
Common NLP Tasks
Task Description Examples
Breaking text into words or Splitting "I like AI."
Tokenization
sentences → ["I", "like", "AI"]
Part-of-Speech Identifying grammar roles of Nouns, verbs,
Tagging words adjectives
Named Entity Detecting names, places, Extracting "New
Recognition (NER) dates, etc. York" from text
Determining the sentiment
Sentiment Analysis Reviews, tweets
(positive/negative) of text
Translating text from one
Machine Translation Google Translate
language to another
Creating concise summaries News article
Text Summarization
of long text summaries
Converting spoken language Voice assistants
Speech Recognition
to text (Siri, Alexa)
Chatbots / Engaging in human-like Customer service
Conversational AI dialogue bots
How NLP Works
1. Text Preprocessing: Clean and prepare text (remove punctuation, lowercase,
remove stop words).
2. Feature Extraction: Convert text into numerical format (e.g., Bag of Words, TF-
IDF, word embeddings).
3. Modeling: Use machine learning or deep learning models to understand or
generate language.
4. Post-processing: Interpret and present the results.
Popular NLP Tools & Libraries
NLTK (Natural Language Toolkit)
spaCy
Transformers (by Hugging Face)
Gensim
Stanford NLP
Why NLP is Important
Bridges the gap between human communication and computer understanding.
Powers applications like virtual assistants, automatic translation, sentiment
analysis, and more.
Enables large-scale analysis of text data (e.g., social media, customer feedback).
What is Tokenization in NLP?
Tokenization is the first and most fundamental step in Natural Language
Processing (NLP).
It means breaking down text into smaller units called tokens, such as words,
subwords, or sentences.
These tokens are then used as the basic units for further analysis like parsing, part-
of-speech tagging, or feeding into machine learning models.
Why is Tokenization Important?
Computers can't understand raw text — it must be broken down.
Tokenization turns unstructured text into structured, processable data.
It's the foundation for most NLP tasks like text classification, translation,
sentiment analysis, etc.
Types of Tokenization
Type Description Example (on "I love AI!")
Word Tokenization Splits text into words ['I', 'love', 'AI', '!']
Sentence
Splits text into sentences ["I love AI!"]
Tokenization
Type Description Example (on "I love AI!")
Subword Splits into meaningful ['lov', '##ing'] (in deep
Tokenization sub-units learning)
Character Splits into individual ['I', ' ', 'l', 'o', 'v',
Tokenization characters 'e']
Challenges in Tokenization
Punctuation: "Let’s go." → should “Let’s” be one or two tokens?
Abbreviations: “Dr. Smith” – don’t split at the period.
Languages without spaces (e.g., Chinese, Japanese): Need advanced
tokenizers.
Popular Tokenizers / Libraries
Tool Use
NLTK Basic tokenization
spaCy Fast and accurate NLP tokenization
Hugging Face Tokenizers For transformer models (BERT, GPT, etc.)
SentencePiece / Byte-Pair Encoding For subword tokenization
Feature Extraction in NLP
1. One-Hot Encoding
Definition: Represents each word in the vocabulary as a binary vector.
How it works:
For a vocabulary of size V, each word is assigned a vector of size V, with a 1 at
the index of the word and 0s elsewhere.
Example:
Vocabulary: ["cat", "dog", "mouse"]
One-hot for "dog": [0, 1, 0]
Pros: Simple, fast
Cons: Doesn't capture meaning or similarity between words; very sparse for
large vocabularies.
2. Bag of Words (BOW)
Definition: Counts the frequency of each word in a document, ignoring grammar
and order.
How it works:
For a document, count how many times each word in the vocabulary appears.
Example:
Document: "dog bites man"
Vocabulary: ["dog", "bites", "man", "cat"]
BOW vector: [1, 1, 1, 0]
Pros: Easy to implement
Cons: Ignores word order and semantics
3. TF-IDF (Term Frequency–Inverse Document Frequency)
Definition: Weighs word frequencies by how unique or rare they are across all
documents.
Formula:
TF-IDF(t,d)=TF(t,d)×IDF(t)\text{TF-IDF}(t,d) = \text{TF}(t,d) \times \text{IDF}(t)TF-
IDF(t,d)=TF(t,d)×IDF(t)
TF (Term Frequency): How often the term t appears in document d
IDF (Inverse Document Frequency): Penalizes common words across many
documents
Example:
Common words like "the" get low scores; rare but relevant words get high scores.
Pros: Better than BOW for filtering out unimportant words
Cons: Still ignores word order
4. N-Grams
Definition: Contiguous sequences of n items (usually words) from a given text.
Types:
Unigrams: single words → "the", "cat"
Bigrams: 2-word pairs → "the cat", "cat sat"
Trigrams: 3-word groups → "the cat sat"
Example:
Sentence: "the cat sat on the mat"
Bigrams: [("the", "cat"), ("cat", "sat"), ("sat", "on"), ...]
Pros: Captures some context and word order
Cons: Gets very large and sparse for higher n; still limited in understanding
semantics
Summary Table:
Captures Captures Handles
Method Sparsity
Frequency Order Meaning
One-Hot No No No High
BOW Yes No No High
TF-IDF Yes (weighted) No Some (via rarity) High
N-
Yes (with context) Yes (partial) No Higher
Grams
Word Embeddings
Definition: Word embeddings are dense vector representations of words in a
continuous vector space where semantically similar words are close together.
Key Idea:
Unlike one-hot or BOW (which are sparse and ignore semantics), embeddings
learn from context — i.e., "king" and "queen" end up with vectors close to each
other.
Properties:
Low-dimensional (e.g., 100–300 dimensions)
Learned from large corpora
Preserve semantic and syntactic relationships
Example:
king - man + woman ≈ queen
Word vectors reflect real-world relationships
Visualization Example:
Imagine a 2D plot of word embeddings (after dimensionality reduction):
Here, you can see that king → queen is similar to man → woman, showing how
Word2Vec captures analogy relationships.
Word2Vec
Definition: A popular algorithm to generate word embeddings, developed by
Google in 2013.
Two main architectures:
CBOW (Continuous Bag of Words):
Predicts a word from its surrounding context
Example: Given "the __ sat on the mat", predict "cat"
Skip-gram:
Predicts context from a single word
Example: Given "cat", predict "the", "sat", "on", "mat"
Training Objective:
Use a shallow neural network to learn word vectors that maximize the
probability of context words.
Pros:
Captures semantic relationships
Efficient to train
Results in meaningful vector space
Cons:
Ignores subword information
Fixed vocabulary (words unseen during training are out-of-vocabulary)
Comparison with Other Techniques:
Captures Captures
Technique Vector Type Dimensionality
Meaning Order
Sparse,
One-Hot ❌ ❌ High
binary
Sparse,
BOW ❌ ❌ High
count
Sparse,
TF-IDF (rarity-based) ❌ High
weighted
Sparse,
N-Grams ❌ ✅ (some) Very High
context
Dense, ✅ (via
Embeddings ✅ Low
learned context)