Data
Pre-processing
Data Pre-processing Techniques are:
Data Cleaning – Fixes missing, noisy, or
inconsistent data to improve quality.
Data Integration – Combines data from multiple
sources into a unified view.
Data Transformation – Converts data into suitable
format (scaling, encoding, normalization).
Data Reduction – Reduces data size while keeping
key information (feature selection, PCA)
Step What it Does Student Example
Fix errors, missing, Correct wrong marks, fill
Cleaning
inconsistencies missing attendance
Merge from multiple Combine LMS scores +
Integration
sources attendance machine data
Shrink data size while Store average attendance
Reduction
keeping info instead of daily logs
Scale, encode, construct Normalize marks, encode
Transformation
features branch, add total marks
Method How it works Example (Students) When to Use
Group values into Marks 51,52,54,55
For 1D noisy
Binning intervals and → all replaced with
numeric data
smooth 53
Attendance trend For time-series /
Fit a line/curve and
Regression 80,82,83,120 → continuous data
replace noisy points
replace 120 with 84 with trends
Group similar GPA+Attendance →
For multi-
Clustering points; outliers = (2,100) marked as
dimensional data
noise noise
Method Concept Student Example
Box Plot (IQR) Values outside 1.5×IQR Student with 300 marks
when max is 100
Scatter Plot Outliers visually far away Low attendance but GPA
9.8
Z-Score Values > 3σ from mean 120 marks when mean =
70, SD = 10
Expectation-Maximization Data not fitting any Mark = 2 in group with avg
Gaussian cluster 50 & 80
Linear Correlation Breaking a correlation Studies 50 hrs but GPA = 2
trend
Cluster Density (DBSCAN) Point in sparse region Student not fitting any
GPA-attendance cluster
Nearest Neighbor Distance to k-NN too high 60 library visits vs peers
with 5–10
IQR Values outside bounds Q1– 50 study hours/day vs class
1.5×IQR, Q3+1.5×IQR avg 2–4
Method Example When to Use
Drop missing values Remove Student 102 Few missing rows only
Constant replacement Branch = “Unknown” Categorical data
Mean/Median/Mode Attendance = 85 Numerical values
Forward/Backward Fill Attendance logs Time-series data
Model-based Imputation Predict GPA Complex dependencies
Technique to Example (Student
Issue Explanation
Resolve Data)
- Schema Matching
Attendance dataset
Same concept (map Roll_No =
→ Student_ID
stored under Student_ID) - Data
Schema Exam dataset →
different attribute Transformation
Integration Roll_No → Map
names or formats (standardize
both as
in multiple sources. formats, e.g., date
Student_ID.
or units)
Library dataset has
- Entity Resolution
Mira Sharma
Same entity (Record Linkage) -
Attendance dataset
Redundancy appears multiple Deduplication -
has M. Sharma
(Duplicates) times across Use string similarity
Both map to the
datasets. or probabilistic
same student
matching
record.
- Conflict
Attendance dataset
Resolution Rules
says Student 101 =
(choose most
85% Biometric
Same record has reliable source,
system says
Data Value Conflict conflicting values in average, or latest
Student 101 = 87%
different sources. timestamp) -
→ Choose
Domain
biometric as more
Knowledge for
reliable.
preference
Example (Student
Technique What it Does When to Use Dataset)
Removes features When a column is Scholarship Amount
Missing Value Ratio with too many mostly empty column has 70%
missing values missing → drop it
Removes features When a feature is Uniform Color =
Low Variance Filter with little or no nearly constant same for 99% of
variation across all records students → drop it
Removes When two features Total Marks vs
High Correlation redundant features contain almost the Percentage → keep
Filter that are highly
correlated same information only one
Combines features When there are
PCA (Principal into fewer many related Math, Physics,
Component “principal variables, and you Chemistry →
Analysis) components” that want fewer combined into one
capture most dimensions Science Score
variance
Scaling
Definition: Resizing data values so they fall within a fixed
range, often without changing their relative differences.
Why: Some ML models (like KNN, SVM, Neural Networks) are
sensitive to large ranges.
Example:
Student exam marks range: 0–100
Lab attendance marks range: 0–10
After scaling, both could be mapped to 0–1 so neither
dominates the model.
1. Ordinal Encoding
Used when categories have a natural order or ranking.
Assigns integer values in order.
👉 Example (Student Grades):
Grades = {Poor, Average, Good, Excellent}
Ordinal encoding:
Poor → 1
Average → 2
Good → 3
Excellent → 4
✅ Works because grades have a clear hierarchy.
⚠️Limitation: It assumes equal distance (difference
between “Poor” and “Average” = “Good” and
“Excellent”), which might not always be true.
2. Nominal Encoding
Used when categories have no order.
Main techniques:
(a) One-Hot Encoding (OHE)
Creates binary columns for each category.
👉 Example (Student Branch):
{CSE, ECE, ME}
CSE → [1, 0, 0]
ECE → [0, 1, 0]
ME → [0, 0, 1]
(b) Label Encoding
Assigns arbitrary numbers, but order doesn’t matter.
👉 Example:
CSE → 1, ECE → 2, ME → 3
⚠️Issue: Model may mistakenly assume ME > ECE > CSE, so one-hot is
usually preferred.
Training and Testing Sets?
When we build a machine learning model, we need data. But we don’t use the
entire dataset for training, because then the model might just memorize
everything and fail to generalize on new unseen data.
So, we split the dataset into different parts:
Training Set
Used to train the model (adjust weights/parameters).
Model "learns patterns" from this data.
Testing Set
Used to evaluate the model’s performance.
Checks how well the model works on new/unseen data.
Helps detect overfitting (when the model memorizes training data but fails
on new data).
🔹 Train-Test Split Ratio
Commonly used ratios:
70% training – 30% testing
80% training – 20% testing
75% training – 25% testing
👉 The choice depends on dataset size:
Large datasets → 80–20 split is good.
Small datasets → may keep more for training (e.g., 90–
10).
Covariance
Definition:
Covariance measures the direction of the relationship between two
variables.
Positive covariance → variables increase together.
Negative covariance → when one increases, the other decreases.
Close to zero → no clear linear relationship.
Formula:
Problem:
Covariance values depend on the scale of data.
Example: If you measure height in cm vs meters, covariance will change.
So it’s hard to compare across datasets.
Correlation
Definition:
Correlation is a standardized version of covariance.
It tells us both direction and strength of the relationship.
Formula:
Range: Always between –1 and +1.
+1 → perfect positive relationship.
–1 → perfect negative relationship.
0 → no linear relationship.
Aspect Covariance Correlation
Tells us Direction only Direction + Strength
Not standardized (depends
Scale Standardized (–1 to +1)
on units)
Range –∞ to +∞ –1 to +1
Final measure of
Use case Intermediate step
relationship