Data Preprocessing and Feature
Engineering
1. Data Preprocessing
Definition:
Data preprocessing is the process of converting raw data into a clean, structured, and
meaningful format suitable for analysis and machine learning.
Explanation:
In real-world applications, data is often incomplete, noisy, and inconsistent because it is
collected from multiple sources such as sensors, databases, and user inputs. Data prepro-
cessing ensures that such issues are handled before applying any algorithm. It improves
data quality and reduces errors, which directly affects model accuracy. It also helps in
reducing computational complexity and improves the efficiency of algorithms. Without
preprocessing, even advanced models may produce incorrect or misleading results.
Steps:
• Data Cleaning
• Data Transformation (Scaling/Normalization)
• Data Reduction (Feature Selection & Extraction)
Example: Missing signal values are filled, units are standardized, and irrelevant
features are removed.
2. Data Cleaning
Definition:
Data cleaning is the process of detecting and correcting missing, noisy, duplicate, or
inconsistent data.
Explanation:
Raw data may contain errors due to sensor faults, transmission issues, or human mistakes.
Data cleaning ensures that the dataset is accurate, complete, and consistent. It improves
the reliability of the data and helps in better decision-making. Clean data reduces the
1
chances of errors during model training and testing. It is considered one of the most
important steps in data preprocessing.
(a) Handling Missing Values
n
1X
x̄ = xi
n i=1
Extra Theory: Missing values can degrade model performance if not handled prop-
erly. Imputation methods assume that missing data follows the same distribution as
available data. Choosing the correct method depends on the dataset type and missing
percentage. Improper handling may introduce bias.
Example: Marks = {40, 50, ?, 60}
Mean = 50 ⇒ Missing value = 50
(b) Noise Reduction
y = mx + c
Extra Theory: Noise refers to random errors or unwanted variations in data. Noise
can hide important patterns and reduce model accuracy. Smoothing techniques help
reveal underlying trends. Removing noise improves stability and consistency of results.
Example:
Data = {10,12,14,50,52,54}
Smoothed = {12,12,12,52,52,52}
(c) Outlier Detection
x−µ
z=
σ
Extra Theory: Outliers are extreme values that differ significantly from other ob-
servations. They can distort statistical measures such as mean and variance. Detecting
outliers helps in improving model accuracy. However, sometimes outliers contain useful
information and should be handled carefully.
Example: 100 in {10,12,14,100} is an outlier.
(d) Removing Duplicates
Extra Theory: Duplicate data leads to biased results and incorrect analysis. Inconsis-
tent formats reduce data quality and make processing difficult. Standardization ensures
uniform representation of data. This step improves data integrity and usability.
Example: “M”, “Male”, “male” → standardized to “Male”
2
3. Data Normalization and Scaling
Definition:
Normalization and scaling are techniques used to bring all features to a common range.
Explanation:
Different features have different units and ranges, which can lead to biased model re-
sults. Scaling ensures that each feature contributes equally to the model. It improves
convergence speed and numerical stability. It is especially important for distance-based
and gradient-based algorithms. Proper scaling leads to better performance and faster
learning.
(a) Min-Max Normalization
x − xmin
xnorm =
xmax − xmin
Extra Theory: This method rescales data into a fixed range, usually between 0 and
1. It preserves relationships between values. However, it is sensitive to outliers. It is
widely used in neural networks.
Example: 70,000 in range 20,000–120,000 ⇒ 0.5
(b) Z-score Standardization
x−µ
z=
σ
Extra Theory: This method transforms data to have zero mean and unit variance.
It is less affected by outliers compared to Min-Max scaling. It is commonly used in
statistical models. It helps compare values across different distributions.
Example: Mean = 50, Std = 10, Value = 70 ⇒ z = 2
4. Feature Selection
Definition:
Feature selection is the process of selecting important features and removing irrelevant
ones.
Explanation:
Not all features contribute equally to the output. Some features may be redundant or
irrelevant. Removing such features reduces complexity and prevents overfitting. It also
improves model accuracy and reduces training time. Feature selection helps in building
efficient and interpretable models.
3
(a) Filter Method
P
(xi − x̄)(yi − ȳ)
r = pP P
(xi − x̄)2 (yi − ȳ)2
Extra Theory: Filter methods are independent of machine learning models. They
are fast and computationally efficient. These methods use statistical measures to rank
features. However, they may ignore interactions between features.
Example: High correlation ⇒ keep feature
(b) Wrapper Method
Extra Theory: Wrapper methods use a predictive model to evaluate feature subsets.
They provide better accuracy but are computationally expensive. They are suitable for
smaller datasets. These methods consider feature interactions.
Example: Best feature set gives highest accuracy
(c) Embedded Method
X
J(θ) = MSE + λ |θj |
Extra Theory: Embedded methods perform feature selection during model training.
They combine advantages of filter and wrapper methods. They are efficient and reduce
overfitting. Regularization techniques like Lasso are commonly used.
Example: Coefficient = 0 ⇒ feature removed
5. Feature Extraction
Definition:
Feature extraction transforms original features into a new reduced set.
Explanation:
Instead of selecting features, new features are created by combining existing ones. It
reduces dimensionality and removes redundancy. It helps in capturing important patterns
in data. Feature extraction improves model efficiency and visualization. It is especially
useful for high-dimensional datasets.
(a) PCA
Xnew = XVk
Extra Theory: PCA is an unsupervised technique that maximizes variance. It
converts correlated variables into uncorrelated components. It reduces dimensionality
while preserving most information. It is widely used in image and signal processing.
Example: (Height, Weight) → (PC1, PC2)
4
(b) LDA
wT SB w
J(w) =
w T SW w
Extra Theory: LDA is a supervised method used mainly for classification. It maxi-
mizes class separability. It works well when classes are clearly separated. It is commonly
used in pattern recognition.
Example: Spam vs Not Spam classification
6. Conclusion
Data preprocessing and feature engineering are essential steps in machine learning. They
improve data quality, reduce complexity, and enhance model performance. Proper use of
these techniques ensures accurate and efficient systems.