📊 Data Preprocessing Steps with Simple Examples
1. Data Collection
Definition: Gathering raw data from various sources.
Example
Suppose you're building a house-price prediction model. You collect data from:
CSV file containing house prices
SQL database containing neighborhood information
API giving interest rates
Source Columns Collected
house_data.csv price, size, rooms
[Link] crime_rate, school_rating
api.interest_rate current_rate
2. Data Integration
Definition: Combining data from multiple sources into a single dataset.
Example
You merge the house_data with the neighborhood_data using a common key such as
house_id.
Python
merged_data = house_data.merge(neighborhood_data, on="house_id")
Now you have one dataset with:
price, size, rooms, crime_rate, school_rating
3. Data Cleaning
Fixing errors, missing values, outliers, and inconsistencies.
3a. Handling Missing Values
Example
You find missing values in the "school_rating" column:
school_rating: [8, 9, NaN, 7, NaN]
Fill with mean:
Python
df['school_rating'].fillna(df['school_rating'].mean(), inplace=True)
3b. Handling Outliers
Example
You find a house with size = 20,000 sq ft while others range from 800–3000.
Outlier removal (e.g., keeping sizes less than 6000 sq ft):
Python
df = df[df['size'] < 6000]
3c. Fixing Inconsistencies
Example
You see inconsistent category values for a city column:
"New York", "new york", "NY", "N.Y."
You standardize them (e.g., to lowercase):
Python
df['city'] = df['city'].[Link]()
4. Data Transformation
4a. Encoding Categorical Variables
Example
Column: color = [red, blue, green]
One-hot encoding:
Python
pd.get_dummies(df['color'])
Output:
red blue green
1 0 0
0 1 0
0 0 1
4b. Feature Scaling
Example
Feature: house size
Original values: [1000, 1500, 2000]
Standardization (Z-score normalization):
Python
from [Link] import StandardScaler
df['size_scaled'] = StandardScaler().fit_transform(df[['size']])
Output (approx):
[-1.22, 0.0, 1.22]
4c. Feature Engineering
Example
You have a date column: "2023-05-14"
You extract useful features:
Python
df['year'] = df['date'].[Link]
df['month'] = df['date'].[Link]
5. Data Reduction
5a. Dimensionality Reduction (PCA)
Example
You have 50 features that are highly correlated. You apply Principal Component Analysis
(PCA) to reduce it to 10 components:
Python
from [Link] import PCA
pca = PCA(n_components=10)
df_reduced = pca.fit_transform(df)
5b. Feature Selection
Example
You drop irrelevant features like:
house_color
owner_name
because they don’t affect the target variable (house price).
6. Data Splitting
Example
You split the dataset into training and testing sets to evaluate the model's performance on unseen
data.
Python
from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.2, random_state=42)
Splits into:
80% training set
20% testing set
7. Handling Imbalanced Data
Example
In a fraud detection dataset:
98% transactions → not fraud (Majority Class)
2% transactions → fraud (Minority Class)
You apply SMOTE (Synthetic Minority Over-sampling Technique) to oversample the
minority class:
Python
from imblearn.over_sampling import SMOTE
X_resampled, y_resampled = SMOTE().fit_resample(X, y)
✨ Final Summary with Real Examples
Step Example
Data Collection Load CSV + SQL + API data
Data Integration Merge tables on house_id
Cleaning (Missing) Fill missing school ratings with the mean
Cleaning (Outliers) Remove unrealistic house sizes (> 6000 sq ft)
Step Example
Encoding Convert color (e.g., red) → one-hot encoded vector
Scaling Normalize size values using Standardization
Feature Engineering Split date into year/month features
Reduction (PCA) Use PCA to reduce 50 correlated features to 10
Splitting 80% train / 20% test data split
Class Imbalance Use SMOTE for rare classes (e.g., fraud)