UNIT 1: Introduction to NLP and Text Processing
What is Natural Language Processing (NLP)?
Natural Language Processing (NLP) is a branch of Artificial Intelligence that enables computers
to understand, interpret, and generate human language in a meaningful way. It combines
concepts from linguistics, computer science, and machine learning.
NLP allows machines to process text and speech data just like humans. It is widely used in
applications such as chatbots, language translation, sentiment analysis, and voice assistants.
History and Applications of NLP
Initially, NLP systems were rule-based, where linguistic rules were manually defined. These
systems were limited in handling complex language structures. Later, machine learning
techniques improved NLP by learning patterns from data. In recent years, deep learning models
such as BERT have revolutionized NLP by understanding context more effectively.
Applications of NLP include:
• Machine translation (e.g., English to Hindi)
• Sentiment analysis (positive/negative reviews)
• Chatbots and virtual assistants
• Text summarization
• Speech recognition
Structure of NLP System
An NLP system typically follows these steps:
1. Input – Raw text or speech data
2. Preprocessing – Cleaning and preparing text
3. Feature Extraction – Converting text into numerical form
4. Model Processing – Applying machine learning algorithms
5. Output – Final result (classification, translation, etc.)
Challenges in NLP
NLP faces several challenges due to the complexity of human language:
• Ambiguity – Words can have multiple meanings
• Context Understanding – Same sentence may mean different things
• Sarcasm Detection – Difficult for machines to understand
• Language Diversity – Different languages and dialects
• Grammar Variations
Text Preprocessing Techniques
1. Tokenization
Tokenization is the process of splitting text into smaller units called tokens (words or sentences).
Example:
“I love AI” → [“I”, “love”, “AI”]
2. Stopword Removal
Stopwords are common words like “is”, “the”, “and” that do not add much meaning and are
removed to reduce noise.
3. Stemming
Stemming reduces words to their root form by removing suffixes.
Example:
“running”, “runner” → “run”
4. Lemmatization
Lemmatization converts words into their base or dictionary form considering context.
Example:
“better” → “good”
UNIT 2: Syntax and Parsing
Parts of Speech (POS) Tagging
POS tagging assigns grammatical categories to words such as noun, verb, adjective, etc.
Example:
“AI is powerful”
AI (Noun), is (Verb), powerful (Adjective)
Context-Free Grammar (CFG)
CFG is a set of rules used to describe sentence structure.
Example rule:
S → NP + VP
(Noun Phrase + Verb Phrase)
Parsing
Parsing is the process of analyzing the grammatical structure of a sentence.
Types of Parsing:
1. Constituency Parsing
Breaks sentence into sub-parts (phrases)
2. Dependency Parsing
Shows relationship between words (who depends on whom)
NLP Libraries
Popular NLP libraries include:
• NLTK – for basic NLP tasks
• spaCy – for fast and advanced processing
UNIT 3: Semantics and Word Representation
Bag of Words (BoW)
BoW represents text as a collection of word frequencies without considering order.
Example:
“I love AI” → {I:1, love:1, AI:1}
TF-IDF (Term Frequency – Inverse Document Frequency)
TF-IDF measures the importance of a word in a document relative to a collection of documents.
Common words get lower weight, rare words get higher weight.
Word Embeddings
Word embeddings convert words into numerical vectors that capture semantic meaning.
Types:
• Word2Vec
• GloVe
These models place similar words close to each other in vector space.
BERT and Transformer Models
Modern NLP models like BERT use transformer architecture to understand context better.
Example:
Word “bank”
• River bank
• Money bank
BERT identifies correct meaning based on context.
UNIT 4: Text Classification and Sequence Models
Applications
• Sentiment analysis
• Spam detection
• News classification
Classification Algorithms
1. Naive Bayes
A probabilistic classifier based on Bayes’ theorem. It assumes independence between features.
2. Logistic Regression
A statistical model used for binary classification (e.g., spam vs not spam).
Sequence Models
RNN (Recurrent Neural Network)
Processes sequential data by remembering previous inputs.
LSTM (Long Short-Term Memory)
An advanced RNN that can remember long-term dependencies.
Attention Mechanism
Attention helps models focus on important parts of input text instead of processing everything
equally.
UNIT 5: Advanced Topics and Applications
Named Entity Recognition (NER)
NER identifies entities such as names, locations, and dates.
Example:
“Zunaid lives in Delhi”
Zunaid → Person
Delhi → Location
Text Summarization
Summarization reduces large text into shorter meaningful content.
Machine Translation
Automatically converts text from one language to another.
Example:
English → Hindi
Chatbots and Conversational AI
Chatbots simulate human conversation using NLP techniques. They are used in customer
support, virtual assistants, etc