Lesson 4: Data Cleaning and preprocessing
Why Data Cleaning Matters
Dirty data leads to incorrect insights
and poor model performance.
Cleaning improves reliability,
reproducibility, and stakeholder trust.
Reasons for data cleaning
Improved decision-making: Clean and accurate data leads to
trustworthy insights, enabling organizations to make confident,
strategic decisions.
Increased efficiency: It streamlines processes by reducing the time
analysts and data scientists spend correcting errors and validating
data, allowing them to focus on generating value.
Enhanced data quality and consistency: Data cleaning standardizes
formats, corrects errors, and removes duplicates, making the data
more consistent and easier to work with.
Better customer relationships: Accurate data allows for more
personalized marketing and communication, which improves customer
engagement and trust.
Cont’
Risk mitigation and compliance: Cleaning data helps businesses
comply with regulations like GDPR and avoid risks associated with
using incomplete, inaccurate, or outdated information, such as legal
fines or security vulnerabilities.
Reliable analytics: Clean data is a prerequisite for the success of
data analysis, business intelligence, and machine learning models,
as dirty data can lead to inaccurate or biased results.
Cost savings: Addressing data quality issues proactively through
cleaning prevents wasted resources, failed marketing campaigns,
and costly mistakes that can arise from bad data.
Common Data Quality Issues
Missing values, duplicates,
inconsistent formats, outliers, wrong
types.
Also: irrelevant columns, typos, and
mixed units.
Typical Preprocessing Workflow
Inspect Feature
Clean Transform Validate Save
data engineer
Loading Data: CSV/Excel/JSON
• Use pandas to load different formats reliably.
import pandas as pd
df = pd.read_csv('[Link]')
df = pd.read_excel('[Link]')
df = pd.read_json('[Link]')
Initial Exploration: head(), info(),
describe()
• head() shows sample rows, info() shows dtypes and non-null counts,
describe() gives stats.
print([Link]())
print([Link]())
print([Link](include='all'))
Detecting Missing Values
• Use isnull(), notnull(), and sum to quantify missingness per column.
• Visualize with heatmaps for patterns.
missing = [Link]().sum()
print(missing)
Types of Missingness
MCAR: Missing Completely at Random
MAR: Missing at Random
MNAR: Missing Not at Random
Understand mechanism to choose imputation strategy.
Basic Strategies: Drop or Impute
• Dropping rows/cols ok if few missing or not-critical columns.
• Impute with mean/median/mode or model-based imputation.
[Link](subset=['important_col'], inplace=True)
df['col'].fillna(df['col'].mean(), inplace=True)
Advanced Imputation Methods
• KNN-imputation, IterativeImputer (MICE), or model predictions.
• Use [Link] or fancyimpute for complex cases.
from [Link] import KNNImputer
imputer = KNNImputer(n_neighbors=5)
df[['A','B']] =
imputer.fit_transform(df[['A','B']])
Handling Duplicates
• Identify duplicates with duplicated(), remove with drop_duplicates().
• Be careful with subset and keep parameters.
dupes = [Link]()
print([Link]())
df.drop_duplicates(inplace=True)
Correcting Data Types
• Convert numeric strings, parse dates, and convert categories.
• Avoid using object dtype for numeric columns.
df['date'] = pd.to_datetime(df['date'])
df['value'] = pd.to_numeric(df['value'],
errors='coerce')
Parsing Dates & Times
• Use pd.to_datetime and set as index for time series.
• Extract features: year, month, weekday, hour.
df['date']=pd.to_datetime(df['date'])
df['year']=df['date'].[Link]
df.set_index('date', inplace=True)
Standardizing Text Data
• Lowercase, strip whitespace, unify encodings, fix typos.
• Useful before categorical encoding or NLP tasks.
df['city'] =
df['city'].[Link]().[Link]()
Dealing with Inconsistent Categories
• Map synonyms to canonical values, e.g., 'US'/'United States'.
• Use mapping dictionaries or fuzzy matching for typos.
mapping = {'usa':'United
States','us':'United States'}
df['country']=df['country'].map(mappin
g).fillna(df['country'])
Outlier Detection: Visual & Statistical
• Boxplots, z-score, and IQR are common methods.
• Visual inspection often complements statistical tests.
from scipy import stats
z =
[Link]([Link](df['value'].dropna())
)
outliers = [Link](z>3)
Outlier Handling Strategies
• Investigate cause: measurement error vs true extreme.
• Options: remove, cap (winsorize), or transform (log).
df['val_clipped'] =
df['value'].clip(lower, upper)
Dealing with Skewness and Transformations
• Log, sqrt, or Box-Cox transforms reduce right skew and stabilize
variance.
df['log_val'] = np.log1p(df['value'])
Scaling: Why and When
• Scaling ensures features contribute equally to distance-based models.
• Two main methods: StandardScaler and MinMaxScaler.
from [Link] import
StandardScaler
scaler = StandardScaler()
df[['x','y']] =
scaler.fit_transform(df[['x','y']])
Normalization (Min-Max Scaling)
• Rescales features to [0,1], useful for neural nets and visualization.
from [Link] import
MinMaxScaler
mms = MinMaxScaler()
df[['a','b']]=mms.fit_transform(df[['a
','b']])
Standardization (Z-score)
• Centers data to mean 0 and std 1; preferred for many ML algorithms.
from [Link] import
StandardScaler
scaler=StandardScaler()
df['z']=scaler.fit_transform(df[['value
']])
Encoding Categorical Variables
• Label encoding for ordinal categories; one-hot for nominal variables.
• Watch out for high-cardinality causing many columns.
pd.get_dummies(df, columns=['category'])
Ordinal Encoding vs One-Hot
• Ordinal: map ranks to ints. One-Hot: create binary columns.
• Choose based on model and variable nature.
from [Link] import
OrdinalEncoder
enc = OrdinalEncoder()
df['ord'] =
enc.fit_transform(df[['ord_col']])
Handling High-Cardinality Categories
• Use target encoding, hashing, or grouping rare categories into 'Other'.
• Cross-validate target encoding to avoid leakage.
counts = df['cat'].value_counts()
rare = counts[counts<10].index
df['cat']=df['cat'].replace(rare,'Other')
Feature Engineering Basics
• Create meaningful features: ratios, interactions, date parts.
• Good features often beat complex models.
df['pm_ratio'] = df['PM2.5']/df['PM10']
Scaling and normalization: Adjusting the range of
feature values to prevent features with larger values
from dominating the model.
Encoding: Converting non-numeric data (like text or
categories) into a numerical format that a model
can process, such as one-hot encoding.
Imputation: Handling missing data by filling in
Techniques: values, for example, using the median or mean.
Creation: Generating new features from existing
ones, such as calculating a new feature like "price
per square foot" from "price" and "square footage".
Feature selection: Choosing the most relevant
features and removing redundant or irrelevant
ones.
Text Preprocessing
• Lowercase, remove punctuation, tokenize, remove stopwords,
lemmatize.
• Use sklearn or nltk/spaCy for pipelines.
import re // regular
s = [Link](r'[^a-z0-9 ]','',
[Link]())
Imbalanced Data Handling
• For classification: oversampling (SMOTE), undersampling, or class
weights.
• Always validate on untouched test set.
from imblearn.over_sampling import SMOTE
sm = SMOTE()
X_res, y_res = sm.fit_resample(X, y)
Pipelines: Reproducible Preprocessing
• Use sklearn Pipeline to chain preprocessing and modeling steps.
• Helps avoid data leakage and keeps code clean.
from [Link] import Pipeline
pipe = Pipeline([('impute',
SimpleImputer()),('scaler',
StandardScaler())])
ColumnTransformer for heterogeneous
preprocessing
• Apply different transformers to numeric and categorical columns.
• Useful in production-ready preprocessing.
from [Link] import
ColumnTransformer
ct = ColumnTransformer([('num',
num_pipe, num_cols),('cat', cat_pipe,
cat_cols)])
Feature Selection Techniques
• Filter methods (corr threshold), wrapper (RFE), and embedded (Lasso).
• Select features to reduce noise and overfitting.
from sklearn.feature_selection import
SelectKBest, f_classif
skb = SelectKBest(k=10)
X_new = skb.fit_transform(X, y)
Dimensionality Reduction: PCA
• Principal Component Analysis reduces dimensionality with variance
preservation.
• Scale before PCA.
from [Link] import PCA
pca = PCA(n_components=5)
X_p = pca.fit_transform(X_scaled)
Time Series Specific Cleaning
• Resample, handle missing timestamps, rolling aggregates, detect
seasonality.
• Careful with forward/backward filling to avoid leakage.
df =
df.set_index('date').resample('D').mea
n()
Validation & Sanity Checks
• Check ranges, uniqueness, nulls, and simple aggregations to validate
transformations.
• Compare before/after stats.
print([Link]())
print([Link]().sum())
Saving Cleaned Data
• Export to CSV, parquet or database depending on size and usage.
• Parquet is efficient for large datasets.
df.to_csv('[Link]', index=False)
df.to_parquet('[Link]')
Versioning and Reproducibility
• Save preprocessing scripts, use git, and store versions of cleaned
datasets.
• Document steps and random seeds.
import joblib
[Link](pipe, '[Link]')
Automating with Scripts & Notebooks
• Notebooks for exploration; scripts for production. Parameterize and
schedule ETL jobs.
Use papermill or Apache Airflow for
automation
Common Pitfalls & Best Practices
• Never peek at test labels, beware of leakage, log transformations
carefully.
• Keep a data-cleaning checklist.
Data Cleaning Checklist
1. Inspect data
2. Handle missing
3. Remove dupes
4. Correct types
5. Handle outliers
6. Encode categories
7. Scale features
8. Save artifacts
Mini Project: Air Quality Preprocessing (assignment)
• Task: Given 'air_quality_raw.csv' perform full cleaning pipeline and
produce 'air_quality_clean.parquet'.
• Deliverables: cleaned data, notebook, summary report with before/after
statistics.
Lesson 5: Data visualization