Data Mining
Data
Preprocessing
Christopher J Kawishe
MIS, BSc CS, ICDL
Author, Researcher, Trainer, Consultant
Cyber Security, System Dev, and AI
1
Lab Outcome
By the end of this lab, students will be able to:
• Understand key preprocessing steps
• Clean raw datasets
• Integrate multiple data sources
• Transform and normalize attributes
• Handle missing and inconsistent values
• Apply dimensionality reduction techniques
Prepared by Christopher James 2
What Is Data Preprocessing?
Data preprocessing is the first stage of the • Preparing raw data for analysis
Knowledge Discovery in Databases (KDD) • Improves data quality
process. It converts raw, messy data into a clean, • Ensures consistency and accuracy
consistent format ready for analysis. • First stage of the KDD process
This critical step involves detecting and correcting errors, filling
gaps, and harmonizing formats. By ensuring high-quality input, it
enables data-mining algorithms to produce more accurate and
From Dirty to
reliable patterns .
Diamond
Why Preprocessing Matters
The Problem: Dirty Data The Solution: Preprocessing
• Incomplete student registries, mobile- • Acts as a quality-assurance gate
money logs • Removes inaccuracies & standardizes
• Noisy hospital records with errors formats
• Inconsistent formats and typos • Safeguards the validity of analytics
Result: Misleading patterns & costly Result: Confident, data-driven strategies.
decisions.
Data Cleaning
Identify Outliers
Detect anomalies like impossible GPAs
(e.g., 100) for review.
Remove Noise
Target and remove random errors and
irrelevant data points.
Fix Inconsistencies Resolve Duplicates
Standardize formats, e.g., 'Male' vs. 'M', to Delete or merge repeated records to
ensure uniformity. create a single source of truth.
The Origins of Dirty Data
Human Entry Errors Sensor Malfunctions
Manual form-filling introduces typos and Failing weather stations or IoT devices
inconsistencies. deliver null readings.
System Integration Issues Intentional Omissions
Merging legacy systems clashes different Incomplete surveys or non-mandatory
date or number formats. fields lead to gaps in data.
Data Integration Strategies
Source A Source B
• Combine data from multiple sources
• Resolve naming conflicts Resolve & Align
• Detect redundant records
• Produce unified datasets
Example:
Joining registrar tables (using 'Student_ID') with finance
Unified Dataset
data (using 'SID') requires careful mapping to create a
single, consistent student record.
Typical Integration Hurdles
Different Attribute Names Different Units
One system uses 'Customer_ID', another uses Merging data in TZS, USD, and EUR requires
'CustNum'. conversion.
Duplicate Entities Conflicting Formats
The same customer is recorded twice with slight One dataset stores 'Arusha' as 'AR', another as
name variations. 'Arusha'.
Resolving these requires domain dictionaries, fuzzy matching, and manual oversight .
Data Transformation Techniques
Aggregation Attribute Construction Categorical Encoding
Roll-up detailed data, like Create new, derived Convert text labels (e.g.,
converting monthly sales attributes, like 'AgeGroup' Male/Female) to binary (0/1)
into quarterly totals. from a raw 'Age'. for model processing.
Normalization Fundamentals
Normalization scales numeric attributes to a
common range, preventing features with large
magnitudes from dominating distance-based
algorithms .
Min-Max Scaling:
Scales data to a 0-1 interval.
Z-Score Standardization:
Produces values with a mean of 0 and a standard
deviation of 1.
Handling Missing Values
Remove Records:
If data is few, delete affected rows.
Missing data is a common challenge. The chosen
Fill with Mean/Median:
tactic depends on the missingness pattern: random For numeric data, use a central tendency.
absence tolerates imputation, while systematic gaps
may require collection redesign. Most Frequent Value:
For categorical data, use the mode.
Predictive Imputation:
Use models to predict missing values.
Dimensionality Reduction
High-dimensional datasets (with many features)
slow computation and risk overfitting. Reduction
techniques create a leaner representation that
retains essential patterns, accelerating training and
improving model generalization .
Remove irrelevant features
Decrease storage needs
Expose latent structure
Key Reduction Methods
Feature Selection Principal Component Analysis (PCA)
Picks only the most informative attributes, Transforms attributes into fewer, uncorrelated
discarding the rest. linear combinations.
Attribute Grouping Sampling
Merges related variables into a single, Shrinks dataset size while maintaining its
composite feature. overall structure.
The Tangible Payoff of Preprocessing
Higher Accuracy Faster Execution Trustworthy Decisions
Clean data leads to more precise Optimized datasets speed up Reduces operational risk and
and reliable models. algorithm training and analysis. increases confidence in strategies.
[Link]
THANK YOU
©2025 Christopher has over decade years of experience in IT infrastructure, cybersecurity, and education, with
proven track record in designing, implementing, and supporting digital learning environments. His professional
journey includes roles as a tutorial assistant, computer engineer, and assistant system analyst, where he
enhanced institutional capabilities by developing custom web-based systems and providing training to over
10,000 users. In consultancy, he have led workshops on eLearning, cybersecurity, and artificial Intelligence.
15