Practical Lab Lecture – Data Pre-processing in
Machine Learning
By the end of this lab, students will be able to:
1. Understand the importance of data cleaning.
2. Handle missing values, noisy data, and inconsistent formats.
3. Perform data integration and transformation.
4. Apply feature scaling (normalization, standardization, robust scaling).
5. Use feature selection and discretization.
6. Implement data reduction techniques (PCA).
7. Prepare categorical data for ML models.
8. Use Pandas, NumPy, and Scikit-Learn for data pre-processing.
Theory Recap
1. Data Cleaning – Removing missing, noisy, or inconsistent data.
2. Data Integration – Combining multiple sources into one dataset.
3. Data Transformation – Aggregation, scaling, and encoding.
4. Feature Scaling – Normalization, standardization, robust scaling.
5. Feature Selection – Choosing the most informative variables.
6. Discretization – Converting continuous values into categorical bins.
7. Data Reduction – PCA, sampling, clustering.
Part 1: Data Cleaning
Exercise 1: Handle missing values in a dataset.
import pandas as pd
import numpy as np
# Sample dataset with missing values
data = {'Name': ['Ali', 'Sara', 'Ahmed', 'Lina'],
'Age': [25, [Link], 30, 28],
'Salary': [5000, 6000, [Link], 7000]}
df = [Link](data)
print("Original Data:\n", df)
# Fill missing values with mean
df['Age'].fillna(df['Age'].mean(), inplace=True)
df['Salary'].fillna(df['Salary'].mean(), inplace=True)
print("\nCleaned Data:\n", df)
Part 2: Handling Noisy & Inconsistent Data
Exercise 2: Smooth noisy data using moving average.
import pandas as pd
# Simulated noisy sensor readings
data = {'Sensor': [10, 12, 15, 100, 14, 13, 12]}
df = [Link](data)
print("Original Data:\n", df)
# Moving average smoothing
df['Smoothed'] = df['Sensor'].rolling(window=3).mean()
print("\nSmoothed Data:\n", df)
Part 3: Data Transformation & Integration
Exercise 3: Aggregate daily sales into monthly totals.
import pandas as pd
# Daily sales data
dates = pd.date_range('2025-01-01', periods=10, freq='D')
sales = [100, 120, 130, 150, 200, 180, 170, 160, 190, 210]
df = [Link]({'Date': dates, 'Sales': sales})
print("Daily Sales:\n", df)
# Aggregate by month
df['Month'] = df['Date'].[Link]
monthly_sales = [Link]('Month')['Sales'].sum()
print("\nMonthly Sales:\n", monthly_sales)
Part 4: Feature Scaling
Exercise 4: Apply normalization and standardization.
from [Link] import MinMaxScaler, StandardScaler
import numpy as np
data = [Link]([[100], [200], [300], [400], [500]])
# Normalization (Min-Max Scaling)
scaler = MinMaxScaler()
normalized = scaler.fit_transform(data)
# Standardization (Z-score)
std_scaler = StandardScaler()
standardized = std_scaler.fit_transform(data)
print("Original:\n", [Link]())
print("Normalized:\n", [Link]())
print("Standardized:\n", [Link]())
Part 6: Data Reduction (PCA)
Exercise 6: Apply PCA for dimensionality reduction.
from [Link] import PCA
from [Link] import StandardScaler
import pandas as pd
# Sample dataset with 3 features
data = [Link]({
'Feature1': [2, 4, 5, 6, 8],
'Feature2': [1, 3, 5, 7, 9],
'Feature3': [2, 2, 3, 4, 5]
})
# Standardize
scaler = StandardScaler()
scaled_data = scaler.fit_transform(data)
# PCA to reduce to 2 components
pca = PCA(n_components=2)
reduced = pca.fit_transform(scaled_data)
print("Original Data:\n", data)
print("\nReduced Data (2D):\n", reduced)