Machine Learning Theory Assignment

Question 1: Data Preprocessing at InsightX Labs

A. Importance of Preprocessing

Data preprocessing cleans and transforms raw data into a suitable format for machine learning models.
Without it, models learn from noise instead of real patterns, leading to poor performance.

Two problems with raw data:

1. Biased Predictions from Missing/Noisy Data: Missing values or errors cause the model to learn
incorrect patterns, producing unreliable outputs.

2. Feature Scale Imbalance: Features like "Annual Income" (20,000–200,000) dominate "Age" (18–65),
causing algorithms like KNN or SVM to give unfair importance to larger-scale features.

B. Handling Missing Values and Duplicates

Two techniques for missing customer ages:

1. Mean/Median Imputation: Replace missing values with the column's mean or median. Median is
preferred for skewed age data as it is less affected by outliers.

2. Predictive Imputation: Use a regression model with other features (income, location) to predict and fill
missing age values more accurately.

Removing duplicate transaction IDs:

Drop Duplicates by Primary Key: Identify rows sharing the same Transaction ID and keep only the first
occurrence, dropping all subsequent duplicates to ensure each transaction is counted once.

C. Feature Selection vs Feature Extraction

Aspect Feature Selection Feature Extraction

Approach Selects existing features Creates new features

Original Data Preserved Transformed

Example Technique Correlation filter PCA

Example from customer demographics (age, gender, income, location):

Feature Selection: Remove "location" if it shows near-zero correlation with the target variable, retaining
only age, gender, and income.

Feature Extraction: Combine age and income into a new "Financial Maturity Index" feature that captures
more predictive power than either variable alone.

D. Normalization and Standardization

Features with different scales cause gradient-based algorithms to converge slowly and inaccurately.
Scaling ensures every feature contributes equally to model learning.
Real-World Example — Predicting Diabetes: A medical dataset contains Blood Glucose (70–200), BMI
(15–50), and Insulin Level (0–900). Without scaling, Insulin Level dominates all other features. After
standardization (mean=0, std=1), the model learns balanced and accurate decision boundaries.

E. Encoding Categorical Data — "Payment Method"

Label Encoding:

Payment Method Encoded Value

Cash 0

Credit Card 1

Online 2

Limitation: Implies a false ordinal relationship (Online > Cash).

One-Hot Encoding:

Cash Credit Card Online

1 0 0

0 1 0

0 0 1

Preferred for nominal data as no false ranking is introduced.

F. Feature Scaling

Min-Max Scaling:
X' = (X - X_min) / (X_max - X_min) [Rescales values to range 0-1]
Z-Score Standardization:
X' = (X - mu) / sigma [Rescales to mean=0, std=1]

Situation Preferred Technique

No significant outliers, bounded range Min-Max Scaling

Outliers present, Gaussian distribution Z-Score Standardization

Neural networks, image data Min-Max Scaling

SVM, PCA, Logistic Regression Z-Score Standardization

G. Dimensionality Reduction

High-dimensional data causes overfitting, slow training, and the Curse of Dimensionality. Dimensionality
reduction removes redundant features while preserving meaningful information.

How PCA helps: PCA transforms correlated features into ranked uncorrelated Principal Components. For
example, 20 customer features can be reduced to 5 components retaining 95% of variance. This reduces
training time, eliminates noise, and improves performance of distance-based models like KNN and SVM.

H. Advanced Preprocessing — Data Integration

Benefits: Combining sales data (what customers buy) with feedback data (why they complain) creates a
richer dataset, enabling models to detect patterns like dissatisfaction leading to churn — invisible in either
source alone.

Key Challenges:

Challenge Description

Schema Mismatch Different column names for same field (Cust_ID vs CustomerID)

Inconsistent Formats Different date formats across sources

Duplicate Records Same customer appearing multiple times after merging

Missing Join Keys Not all records have a matching entry in both sources

I. End-to-End Preprocessing Pipeline — Predicting Customer Churn

Step Action Details

1 Data Collection & Integration Merge CRM, transaction, and feedback data using Customer ID as the common key.

2 Data Cleaning Handle missing values via imputation, remove duplicates, cap outliers using IQR analy

3 Feature Engineering & Encoding Create new features like "Days Since Last Purchase." Apply One-Hot Encoding to cate

4 Feature Scaling Apply Z-Score Standardization to numerical features like "Monthly Charges" and "Acco

5 Feature Selection Use correlation analysis and Variance Threshold to remove low-predictive features.

6 Train-Test Split Split final dataset into 80% training and 20% testing for unbiased model evaluation.

J. Feature Selection Methods

1. Correlation-Based Selection: Measures the statistical relationship between each feature and the
target. Features with near-zero correlation are dropped. Highly correlated feature pairs are also identified,
and one from each pair is removed to eliminate redundancy.
2. Variance Threshold Method: Removes features with very low variance (near-constant values). For
example, if 98% of customers share the same "Country," that feature adds no discriminative value and is
removed.

How both reduce overfitting: Removing irrelevant and redundant features simplifies the model, forcing it
to learn only genuine patterns. This improves generalization to unseen data and directly reduces
overfitting.