Data Preprocessing

Session No.:17
Course Name: Data Warehouse and Data Mining
Course Code:E1UA514C
Duration:50 Minutes
Instructor Name: Mr. Manish Kumar Maurya

1
 Session Outline

• Introduction
• Data Processing
• Processing Techniques
• Student Learning Activity 1
• Activity 2
• Conclusion & Q&A

2
 Recap of previous Session

Integration of a Data Mining System with a Data Warehouse -issues

3
 Learning Outcome

After completion lecture student would be able to do following

LO-1 Apply data preprocessing techniques such as normalization, scaling,

and encoding.

4
 Opening Question
"What’s Wrong with This Data?"
Customer Dataset
Age Income ($) Location Purchased
25 30,000 Urban Yes
- 50,000 Suburban No
30 - ??? Yes
40 45k Rural Y
28 60,000 urban 1

1. What problems do you see?

2. How might these affect a model?
5
This dataset contains deliberate issues to
highlight preprocessing needs:
 Missing Values:
1. Age is missing in row 2 (denoted by "-").
2. Income is missing in row 3.
3. Location is missing in row 3 (denoted by "???").
 Inconsistent Formats:
1. Income has a mix of formats (e.g., "30,000" vs. "45k").
2. Location has inconsistent capitalization ("Urban" vs. "urban").
 Inconsistent Labels:
• Purchased is recorded as "Yes"/"No", "Y", and "1", which needs
standardization.
 Categorical Data:
• Location (Urban, Suburban, Rural) requires encoding for machine learning
models.
 Data Quality: Why Preprocess the
Data?
"Garbage In, Garbage Out" – models fail with poor-
quality data.
Measures for data quality: A multidimensional view
 Accuracy: correct or wrong, accurate or not
 Completeness: not recorded, unavailable, …
 Consistency: some modified but some not, dangling, …
 Timeliness: timely update?
 Believability: how trustable the data are correct?
 Interpretability: how easily the data can be understood?
7
 Data Preprocessing

In machine learning/data mining, preprocessing prepares raw data

for model training.

This includes handling missing values, normalizing features,

encoding categorical variables, and splitting datasets into training
and testing sets to improve model performance and accuracy.

8
 Major Tasks in Data Preprocessing

Data cleaning
Fill in missing values, smooth noisy data, identify or remove outliers, and resolve
inconsistencies
Data integration
Integration of multiple databases, data cubes, or files
Data reduction
Dimensionality reduction
Numerosity reduction
Data compression
Data transformation and data discretization
Normalization
Scaling
Standardization 9
 Data Cleaning
Data in the Real World Is Dirty: Lots of potentially incorrect data, e.g., instrument faulty, human or
computer error, transmission error
• incomplete: lacking attribute values, lacking certain attributes of interest, or containing only
aggregate data
• e.g., Occupation=“ ” (missing data)
• noisy: containing noise, errors, or outliers
• e.g., Salary=“−10” (an error)
• inconsistent: containing discrepancies in codes or names, e.g.,
• Age=“42”, Birthday=“03/07/2010”
• Was rating “1, 2, 3”, now rating “A, B, C”
• discrepancy between duplicate records
• Intentional (e.g., disguised missing data)
• Jan. 1 as everyone’s birthday?
10
 Incomplete (Missing) Data
Data is not always available
E.g., many tuples have no recorded value for several attributes, such as customer
income in sales data
Missing data may be due to
1. equipment malfunction
2. inconsistent with other recorded data and thus deleted
3. data not entered due to misunderstanding
4. certain data may not be considered important at the time of entry
5. not register history or changes of the data
Missing data may need to be inferred

11
 How to Handle Missing Data?
• Ignore the tuple: usually done when class label is missing (when doing classification)
—not effective when the % of missing values per attribute varies considerably
• Fill in the missing value manually: tedious + infeasible?
• Fill in it automatically with
1. a global constant : e.g., “unknown”, a new class?!
2. the attribute mean
3. the attribute mean for all samples belonging to the same class: smarter
4. the most probable value: inference-based such as Bayesian formula or decision
tree

12
 Noisy Data
Noise: random error or variance in a measured variable
Incorrect attribute values may be due to
faulty data collection instruments
data entry problems
data transmission problems
technology limitation
inconsistency in naming convention
Other data problems which require data cleaning
duplicate records
incomplete data
inconsistent data

13
 How to Handle Noisy Data?
• Binning
• First sort data and partition into (equal-frequency) bins
• Then one can smooth by bin means, smooth by bin median, smooth by bin
boundaries, etc.
• Regression
• Smooth by fitting the data into regression functions
• Clustering
• Detect and remove outliers
• Combined computer and human inspection
• Detect suspicious values and check by human (e.g., deal with possible outliers)

14
 Encoding Categorical Data
Methods: Label Encoding: Assign numbers (e.g., Urban=0, Suburban=1, Rural=2).

One-Hot Encoding: Create binary columns (e.g., Urban: 1/0, Suburban: 1/0).

Urban Suburban Rural

1 0 0
1 0 1

15
 Normalization
Normalization is a key preprocessing technique used to
rescale numerical features to a common scale, typically
between [0, 1] or to have a mean of 0 and standard
deviation of 1. This ensures that no single feature
dominates due to differences in magnitude (e.g., age vs.
salary).
Normalization: Scaled to fall within a smaller, specified range
1. min-max normalization
2. z-score normalization
3. normalization by decimal scaling
16
 Normalization/Scaling
• Min-max normalization: to [new_minA, new_maxA]
v  minA
v'  (new _ maxA  new _ minA)  new _ minA
maxA  minA
• Ex. Let income range $12,000 to $98,000 normalized to [0.0, 1.0].
73,600  12,000
(1.0  0)  0 0.716
Then $73,000 is mapped to 98,000  12,000

• Z-score normalization (μ: mean, σ: standard deviation): v  A

v' 
 A
• Ex. Let μ = 54,000, σ = 16,000. Then 73,600  54,000
1.225
• Normalization by decimal scaling 16,000

v
v'  j Where j is the smallest integer such that Max(|ν’|) < 1
10
17
 Discretization/Binning
• Purpose: Convert continuous data into bins (e.g., age groups).
Methods:
• Equal-Width Binning: Fixed range (e.g., 0-10, 10-20).
• Equal-Frequency Binning: Same number of samples per bin.

18
 Activity 1 : Clean and Encode"
Task: "Fix missing values and prepare ‘Location’ for a model."

Age Income ($) Location Purchased

25 30,000 Urban Yes
- 50,000 Suburban No
30 - ??? Yes
40 45k Rural Y
28 60,000 urban 1

Methods:
•Imputation: Fill missing values (e.g., use mean for Age, mode for Location).
•Removal: Drop rows/columns with too many missing values (if appropriate
19
 Activity 1 : Clean and Encode"

Methods:

Imputation: Fill missing values (e.g., use mean for Age, mode for Location).
Removal: Drop rows/columns with too many missing values (if appropriate).

Example:
Missing Age (row 2): Replace with mean Age (31.7).
Missing Income (row 3): Replace with mean Income (41,667).

20
 Activity 2: "Standardizing Student Heights"
Dataset: Heights (in cm) of 5 students: [160, 170, 175, 180, 190]

Tasks:
[Link] the mean (μ) and standard deviation (σ).
Answer: μ= 175 σ≈ 10.8
[Link] Z-scores for each student manually.

Example: Z160=(160−175)/10.8 ≈−1.39

[Link] results:

21
 Activity 2: "Standardizing Student Heights"
Z-Scores: [-1.39, -0.46, 0.0, 0.46, 1.39]
Z-Scores: [-1.39, -0.46, 0.0, 0.46, 1.39]

Dataset: Heights (in cm) of 5 students: [160, 170, 175, 180, 190]

Tasks:
[Link] the mean (μ) and standard deviation (σ).
Answer: μ=175, σ≈10.8
[Link] Z-scores for each student manually.
Example: Z-Scores: [ - 1 . 3 9 , - 0 . 4 6 , 0 . 0 , 0 . 4 6 , 1 . 3 9 ]
[Link] results:
1. "A Z-score of -1.39 means 160 cm is 1.39 standard deviations below the mean."

22
 Conclusion and Q & A

Preprocessing cleans and formats data for machine learning.

Techniques: Handle missing data (imputation), encode categories (one-hot),
scale features (standardization/min-max).
Pros:
1. Improves model accuracy and speed.
2. Handles real-world messy data.

Cons:
1. Risk of data leakage
2. Adds computational cost or complexity.

Question: "What’s one preprocessing step you’d prioritize for a messy

dataset?"
23
Next Session

Revision of Unit 2

24