Data Preprocessing and Feature Engineering (Exam Notes)
1. Data Preprocessing
Definition: Data preprocessing is the process of converting raw data into a clean, structured, and meaningful
format suitable for analysis and machine learning.
Explanation: In real-world applications, data is often incomplete, noisy, and inconsistent because it is collected
from multiple sources such as sensors, databases, and user inputs. Data preprocessing ensures that such issues
are handled before applying any algorithm. It improves data quality, reduces errors, and increases the efficiency of
machine learning models. Proper preprocessing also helps in faster convergence and better accuracy. Without
preprocessing, the model may give incorrect or misleading results.
Steps Involved:
• Data Cleaning
• Data Transformation (Scaling/Normalization)
• Data Reduction (Feature Selection & Extraction)
Example: In a wireless communication dataset: missing signal strength is filled using mean, different units (Hz,
kHz) are converted, and irrelevant features are removed.
2. Data Cleaning
Definition: Data cleaning is the process of detecting and correcting missing, noisy, duplicate, or inconsistent data.
Explanation: Raw data may contain errors due to sensor faults, transmission errors, or human mistakes. Data
cleaning ensures that the dataset is accurate, complete, and consistent. It is one of the most important steps in
preprocessing because poor-quality data leads to poor model performance. Clean data improves decision-making
and reliability of results.
(a) Handling Missing Values (Mean Imputation)
Mean formula: x■ = (1/n) Σ x■
Example: Marks: {40, 50, ?, 60}. Mean = (40 + 50 + 60)/3 = 50, so the missing value is replaced with 50.
(b) Noise Reduction
Regression model: y = mx + c
Example (Binning): Data {10, 12, 14, 50, 52, 54} → bins {10,12,14} → 12 and {50,52,54} → 52. Smoothed data
becomes {12,12,12,52,52,52}.
(c) Outlier Detection
Z-score: z = (x − µ) / σ
Example: In {10, 12, 14, 100}, the value 100 is far from others and treated as an outlier.
(d) Removing Duplicates & Inconsistencies
Example: 'M', 'Male', 'male' are standardized to 'Male'. Duplicate rows are removed.
Advantages: Improves data quality, reduces errors, and increases accuracy.
3. Data Normalization and Scaling
Definition: Normalization and scaling bring all features to a common scale.
Explanation: Different features have different ranges, which may cause bias in models. Scaling ensures that all
features contribute equally. It improves convergence speed and model performance, especially in algorithms like
SVM and gradient descent.
(a) Min-Max Normalization
x_norm = (x − x_min) / (x_max − x_min)
Example: Range 20,000 to 120,000; value 70,000 → normalized value = 0.5.
(b) Z-Score Standardization
z = (x − µ) / σ
Example: Mean = 50, Std Dev = 10, value 70 → z = 2.
Advantages: Prevents dominance of large values, improves accuracy, and speeds up learning.
4. Feature Selection
Definition: Selecting relevant features and removing unnecessary ones.
Explanation: Not all features are useful for prediction. Removing redundant features reduces complexity and
prevents overfitting. It improves model performance and reduces training time.
(a) Filter Method (Correlation)
r = Σ[(x■ − x■)(y■ − ■)] / sqrt(Σ(x■ − x■)² Σ(y■ − ■)²)
Example: Study hours vs marks shows high correlation → keep; height vs marks shows low correlation → remove.
(b) Wrapper Method
Example: Compare feature sets: {Age, Salary} → 90% accuracy vs {Height, Weight} → 60% → choose best set.
(c) Embedded Method (Lasso)
J(θ) = MSE + λ Σ |θ■|
Example: If a coefficient becomes 0, the feature is removed automatically.
Advantages: Reduces overfitting, improves accuracy, reduces computation.
5. Feature Extraction
Definition: Transforming original features into a new reduced set.
Explanation: New features are created by combining existing ones. It reduces dimensionality and captures
important information. It is useful for large datasets and improves efficiency and visualization.
(a) PCA
X_new = X V_k
Example: (Height, Weight, BMI) → converted to (PC1, PC2).
(b) LDA
J(w) = (w■ S_B w) / (w■ S_W w)
Example: Email classification into spam and not spam.
Advantages: Reduces dimensionality, removes redundancy, improves performance.
6. Conclusion
Data preprocessing and feature engineering are essential steps in machine learning. They improve data quality,
reduce complexity, and enhance model performance. Techniques like cleaning, scaling, selection, and extraction
ensure accurate and efficient results.
Difference Between Feature Selection and Feature
Extraction
No. Feature Selection Feature Extraction
1 Selects the most important features from the dataset. Creates new features from existing data.
2 Uses original data without transformation. Transforms data into a new form.
3 Keeps original meaning of features. Changes the meaning of features.
4 Removes irrelevant and redundant features. Combines features to reduce dimensionality.
5 Easy to understand and interpret. More complex and harder to interpret.
6 Has high interpretability. Has low interpretability.
7 Does not compress data. Compresses data into fewer features.
8 Requires less computational power. Requires more computational power.
9 Less complex. More mathematically complex.
10 Keeps actual features like age and salary. Produces features like principal components.
11 May lose useful information if done incorrectly. Retains most important information.
12 Works well with smaller datasets. Useful for large/high-dimensional datasets.
13 Improves performance by removing noise. Improves performance by reducing dimensions.
14 Uses methods like correlation and Lasso. Uses methods like PCA and LDA.
15 Example: Selecting Age and Salary. Example: Converting to principal components.