Text Classification
• Text Classification (Text Categorization) is the task of assigning a label or
categorization category to an entire text or document.
Text classification can be defined as follows:
Input:
• a document d
• a fixed set of classes C = {c1, c2 ,…, cn}
Output:
• a predicted class c € C
• Some of common text categorization tasks are:
– Sentiment analysis
• extraction of sentiment, positive or negative orientation that writer expresses
toward an object.
– Spam detection
• binary classification task of assigning an email to one of the two classes
spam or not-spam.
– Authorship identification
• determining a text’s author.
– Age/gender identification
• determining a text’s author characteristics like gender and age.
– Language Identification
• finding the language of a text.
Naïve Bayes Classifier for Text Classification
Naive Bayes is a machine learning classification algorithm that predicts the
category of a data point using probability. It assumes that all features are
independent of each other. Naive Bayes performs well in many real-world
applications such as spam filtering, document categorisation and sentiment
analysis.
What is Naïve Bayes?
Naïve Bayes is a baseline classifier for text (spam filtering, sentiment analysis,
topic categorization) — simple yet surprisingly effective in practice.
A probabilistic classifier based on Bayes’ Theorem.
Assumes conditional independence between features (hence “naïve”).
Widely used for text classification because it works well with high-
dimensional data (e.g., thousands of words).
P(meeting|NotSpam), P(project|NotSpam), P(report|NotSpam)
5. Advantages
Simple and fast.
Works well for large vocabulary, sparse data.
Requires less training data.
6. Limitations
Assumes independence of words (not always true).
Can perform poorly if features are highly correlated.
Doesn’t capture semantic meaning of words.
TRAINING AND OPTIMIZATION FOR SENTIMENT
ANALYSIS
Sentiment Analysis:
Sentiment analysis is the process of analysing textual data to determine the
emotional tone expressed in it. It classifies text as positive, negative or neutral and can
also detect more amotions like happy, sad, angry or frustrated. It is widely used to
analyse large volumes of text such as customer reviews, social media posts, Emails &
chat transcripts, survey responses.
Training sentiment analysis involves preparing clean data, selecting an
appropriate model, and optimizing it with proper hyperparameter tuning, feature
engineering, and regularization to achieve strong generalization.
1. Training Pipeline
Step 1: Data Preparation
Collect labeled dataset → e.g., tweets, reviews, or news headlines with
Positive, Negative, Neutral tags.
Preprocess:
o Tokenization
o Stopword removal
o Lemmatization/Stemming
o Handling emojis/emoticons (e.g., ●•^•^- = positive)
o Converting text → numerical features (Bag-of-Words, TF-IDF,
Word2Vec, BERT embeddings).
Step 2: Model Selection
Classical ML Models: Naïve Bayes, Logistic Regression, SVM.
Deep Learning Models: CNNs, RNNs (LSTM, GRU),
Transformers (BERT, RoBERTa).
Step 3: Training
Split dataset → Train (70%), Validation (15%), Test (15%).
Feed text features into the model.
Model learns to map inputs → sentiment labels.
Loss functions:
o Cross-Entropy Loss (for classification).
o Optimized using algorithms like SGD, Adam, RMSprop.
2. Optimization Techniques
It involves method to improve model performance.
A. Feature Engineering
Use TF-IDF instead of raw word counts.
Use n-grams (bigrams, trigrams) to capture context (e.g., “not good” vs
“good”).
Pretrained embeddings (Word2Vec, GloVe, BERT) improve semantic
understanding.
B. Hyperparameter Tuning
It involves selecting a set of optimal hyper parameter for a learning
algorithm by selecting early stopping to prevent overfitting.
Adjust learning rate, batch size, regularization, dropout rate.
Use Grid Search, Random Search, or Bayesian Optimization.
C. Regularization
It is used to prevent overfitting by adding a penalty to the model’s complexity.
Overfitting in Machine Learning occurs when a model learns training data too
well, capturing noise and random fluctuations rather than just the underlying
patterns.
D. Data Augmentation
Involves word replacement (synonym), back translation (translate to another
language and back), paraphrasing.
Helps when labeled data is small.
E. Class Imbalance Handling
Addresses situations where a dataset has more positives than negatives:
The following techniques can be used
Oversample minority class (SMOTE).
Undersample majority class.
Class-weight adjustment in loss function.
3. Evaluation Metrics
Measures used to assess a model’s performance
Accuracy (not enough if imbalanced) - Ratio of correct predictions to the total
no. of predictions.
Precision, Recall, F1-score → better indicators.
Confusion Matrix to analyze errors.
4. Example Workflow
1. Input: "This movie was amazing, I loved it!"
2. Preprocessing: tokens = [“movie”, “amazing”, “loved”]
3. Features: TF-IDF vector or BERT embedding.
4. Training: Model learns that “amazing”, “loved” → Positive class.
5. Optimization: Adjust learning rate + dropout until best validation F1-score.
6. Output: Positive Sentiment ⬛