Machine Learning — Classification: Explanations
1. Machine Learning and Real-world Applications (Use-Case: Classification)
Overview: Classification is a supervised machine learning task where the model assigns discrete
labels (classes) to inputs. It is widely used in real-world applications such as sentiment analysis,
medical diagnosis, spam detection, and image recognition. Below are explanations of the
specific items listed.
A. Analyzing the Sentiment of Reviews (Sentiment Analysis)
What it is:
Sentiment analysis uses classification techniques to determine whether a piece of text expresses
positive, negative, or neutral sentiment.
How it works (high-level):
1. Data collection: Gather labeled reviews (e.g., movie reviews labeled positive/negative).
2. Preprocessing: Clean text (lowercase, remove punctuation), tokenize, remove stop words, and
optionally lemmatize/stem.
3. Feature extraction: Convert text to numeric features (Bag-of-Words, TF-IDF, or embeddings
such as Word2Vec/BERT).
4. Model training: Use a classifier (e.g., logistic regression, SVM, or a deep model like an
LSTM/BERT fine-tune) to learn mapping from features to sentiment labels.
5. Evaluation: Measure performance with accuracy, precision, recall, F1-score, and confusion
matrix.
Practical tips: Use domain-specific data (restaurant vs. product reviews), handle class imbalance,
and consider model explainability for business use.
B. What is an Intelligent Restaurant Review System?
Definition:
An intelligent restaurant review system automatically analyzes customer reviews to provide
insights such as overall sentiment, common complaints, feature-level ratings (food, service,
ambiance), and suggestions.
Components:
1. Ingestion: Collect reviews from platforms (Google, Yelp, app feedback).
2. NLP pipeline: Clean and preprocess text, detect language, and extract entities (menu items,
staff names).
3. Sentiment & aspect analysis: Classify sentiment overall and per aspect (e.g., food vs. service).
4. Dashboard & alerts: Visualize trends (weekly sentiment), highlight urgent issues (food safety
complaints), and suggest actions.
Use-cases:
- Prioritize issues for managers, tailor marketing, and auto-respond to customers.
C. Examples of Classification Tasks
- Binary classification: Spam vs. Not-spam, Positive vs. Negative sentiment.
- Multiclass classification: Classifying images into multiple object categories (cat, dog, car), or
classifying news into topics.
- Multilabel classification: Assigning multiple tags to one example (an article may belong to
"sports" and "health").
Key considerations:
- Choose algorithms based on dataset size, feature type, and performance requirements.
- Evaluate using appropriate metrics (accuracy for balanced classes; precision/recall/F1 for
imbalanced).
D. Linear Classifiers
What they are:
Linear classifiers make predictions using a linear combination of input features. The decision is
made by checking which side of a hyperplane the input falls on.
Common examples:
- Logistic Regression (for probabilities in binary classification)
- Linear Support Vector Machine (SVM)
- Perceptron
Characteristics:
- Fast to train, easy to interpret when features are meaningful.
- May underperform if the true decision boundary is non-linear unless features are transformed
(polynomial features, kernels).
When to use:
- Use as a baseline, for high-dimensional sparse data (text), or when interpretability is
important.
E. Decision Boundary
Definition:
A decision boundary is the surface (line in 2D, plane in 3D, hyperplane in higher dimensions)
that separates regions of different predicted classes.
Interpretation:
- For linear classifiers, the decision boundary is a straight hyperplane.
- For non-linear models (decision trees, neural networks, kernel SVMs), the boundary can be
complex and curved.
Visualization tip:
Plot two selected features and color regions by predicted class to inspect the shape and margin
of the boundary.
2. Evaluating Classification
Overview: Evaluating classifiers ensures they generalize well to unseen data and that their
performance meets the problem requirements. Key concepts follow.
A. Training and Evaluating a Classifier
Steps:
1. Split data: Train / validation / test splits (common ratios: 70/15/15 or 80/10/10). Alternatively
use cross-validation (k-fold).
2. Train: Fit the model on training data while tuning hyperparameters on validation data.
3. Evaluate: Use the test set (kept separate) for a final unbiased performance estimate.
4. Cross-validation: Use k-fold CV to reduce variance in performance estimates and for
hyperparameter selection.
Best practices:
- Avoid leaking test information into training.
- Use stratified splits for imbalanced classes.
- Track training and validation metrics over epochs to detect overfitting.
B. What is a Good Accuracy?
Accuracy definition:
Accuracy = (Number of correct predictions) / (Total predictions).
Is it good?
- Depends on the problem and the class balance. For example, 95% accuracy on a dataset where
95% of examples belong to one class is not useful.
- For imbalanced problems, prefer metrics like precision, recall, F1-score, or area under ROC
(AUC).
Rule of thumb:
- Compare to a meaningful baseline (random chance, majority-class classifier, or business
requirement).
- Consider costs of different errors (false positives vs false negatives) to decide acceptable
accuracy.
C. False Positive and False Negative (and the Confusion Matrix)
Confusion Matrix layout (binary):
Predicted Positive | Predicted Negative
Actual Positive | True Positive (TP) | False Negative (FN)
Actual Negative | False Positive (FP)| True Negative (TN)
Definitions:
- False Positive (FP): Model predicts positive, actual is negative (Type I error).
- False Negative (FN): Model predicts negative, actual is positive (Type II error).
Derived metrics:
- Precision = TP / (TP + FP) — how many predicted positives are correct.
- Recall (Sensitivity) = TP / (TP + FN) — how many actual positives were found.
- Specificity = TN / (TN + FP) — how many actual negatives were correctly found.
- F1-score = 2 * (Precision * Recall) / (Precision + Recall) — harmonic mean of precision and
recall.
Business impact:
- Medicine: False negatives (missed disease) are often worse.
- Spam detection: False positives (classifying a valid email as spam) may be more costly to users.
D. Learning Curves
What they show:
Learning curves plot model performance (e.g., error or accuracy) on training and validation sets
as a function of training set size or training epochs.
How to interpret:
- High training accuracy & low validation accuracy → overfitting (model too complex).
- Low training accuracy & low validation accuracy → underfitting (model too simple).
- If validation performance improves with more data, collecting more labeled data can help.
Use:
- Diagnose whether to focus on regularization, more data, or a bigger/smaller model.
E. Class Prediction
Meaning:
Class prediction is the final output of a classifier — the label assigned to an input. Some
classifiers also provide probabilities or confidence scores for each class.
Notes:
- Use probability thresholds to trade off precision and recall (e.g., require a higher probability to
call a positive prediction).
- For multi-class problems, the predicted class is often the one with the highest predicted
probability (argmax).
- Calibrate probabilities (Platt scaling, isotonic regression) if you need reliable probabilistic
outputs.
References & Further Reading:
- "Pattern Recognition and Machine Learning" (Bishop)
- "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow" (A. Géron)
- Scikit-learn documentation and tutorials