UNIT 1: Introduction to NLP & Text Processing
What is NLP?
NLP = Teaching computers to understand, process, and generate human language
Examples:
• Chatbots
• Google Translate
• Spam detection
History (short for exam)
• Early → Rule-based systems
• Later → Machine Learning
• Now → Deep Learning (like BERT)
Structure of NLP System
1. Input text
2. Preprocessing
3. Feature extraction
4. Model
5. Output
Challenges in NLP
• Ambiguity (same word, different meaning)
• Context understanding
• Sarcasm
• Multiple languages
Text Preprocessing (VERY IMPORTANT)
1. Tokenization
Split sentence into words
from [Link] import word_tokenize
text = "I love AI"
print(word_tokenize(text))
2. Stopword Removal
Remove common words (is, the, and)
from [Link] import stopwords
words = ["I", "love", "AI"]
filtered = [w for w in words if w not in [Link]('english')]
print(filtered)
3. Stemming
Cut words to root
from [Link] import PorterStemmer
ps = PorterStemmer()
print([Link]("running")) # run
4. Lemmatization
Convert to meaningful root
from [Link] import WordNetLemmatizer
lm = WordNetLemmatizer()
print([Link]("running", pos='v'))
UNIT 2: Syntax & Parsing
Parts of Speech (POS Tagging)
Identify grammar roles
import nltk
text = "AI is amazing"
print(nltk.pos_tag(nltk.word_tokenize(text)))
Context-Free Grammar (CFG)
Rules to form sentences
Example:
S → NP + VP
Parsing Types
• Dependency Parsing → Relationship between words
• Constituency Parsing → Sentence structure tree
Tools
Using spaCy
import spacy
nlp = [Link]("en_core_web_sm")
doc = nlp("AI is powerful")
for token in doc:
print([Link], token.dep_)
UNIT 3: Semantics & Word Representation
Bag of Words (BoW)
Counts words frequency
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()
print(cv.fit_transform(["I love AI"]).toarray())
TF-IDF (IMPORTANT)
Measures importance of words
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer()
print(tfidf.fit_transform(["I love AI"]).toarray())
Word Embeddings
Convert words → vectors
Word2Vec / GloVe
• Capture meaning
• Similar words = similar vectors
BERT & Transformers
• Understand context deeply
• Used in ChatGPT-like systems
Example:
“bank” (river vs money) → BERT understands context
UNIT 4: Text Classification & Sequence Models
Applications
• Sentiment Analysis
• Spam Detection
Models
Naive Bayes
Simple probability model
from sklearn.naive_bayes import MultinomialNB
Logistic Regression
Binary classification
from sklearn.linear_model import LogisticRegression
RNN & LSTM
• Used for sequence data
• LSTM remembers long context
Attention Mechanism
Focus on important words
Example:
“I didn’t like the movie”
Focus on → “didn’t like”
UNIT 5: Advanced Topics
Named Entity Recognition (NER)
Find entities like:
• Names
• Places
• Dates
for ent in [Link]:
print([Link], ent.label_)
Text Summarization
Convert long text → short summary
Machine Translation
Language conversion
English → Hindi
Chatbots & Conversational AI
• Rule-based
• AI-based
Example:
Customer support bot
FINAL REVISION (VERY IMPORTANT)
Must remember:
• NLP definition
• Preprocessing steps
• POS tagging
• BoW vs TF-IDF
• Word2Vec, BERT
• Naive Bayes
• RNN, LSTM
• NER
Easy Memory Trick
"TSL-LC"
• Tokenization
• Stopwords
• Lemmatization
• Learning models
• Classification