West Yangon Technological University
Department of Information Technology Engineering
Topic:3
Data Preprocessing
Dr. Honey Htun
Associate Professor
Information Technology Dept.
West Yangon Technological University
2
3
Lecture Objectives
• To introduce
- What Data Pre-processing is used for?
- Data Cleaning
- Data Integration
- Data Transformation
- Data Reduction
4
What Data Pre-processing is used for?
• Data is in the real world is dirty
- incomplete: lacking attribute values, lacking certain attributes of interest,
or containing only aggregate data
- noisy: containing errors, duplicate or outliers
- inconsistent: containing discrepancies in codes or names
• No quality data, no quality mining results
Therefore, to overcome the above problems, data pre-processing step is
applied to make the data more suitable for analysing the data
5
Data Pre-processing Tasks
6
Major Tasks in Data Pre-processing
• Data cleaning
▫ Fill in missing values, smooth noisy data, identify or remove outliers,
and resolve inconsistencies
• Data integration
▫ Integration of multiple databases, data cubes, files, or notes
• Data transformation
▫ Normalization (scaling to a specific range)
▫ Aggregation
• Data reduction
▫ Obtains reduced representation in volume but produces the same or
similar analytical results
7
Data Cleaning
• Data cleaning tasks
▫ Handling missing values (Filling or Ignoring)
➢ Filling ➔ Regression, Clustering
▫ Identify outliers and smooth out noisy data
▫ Correct inconsistent data
8
Missing Values
• Reasons for missing values
▫ Information is not collected
(e.g., people decline to give their age and weight)
▫ Attributes may not be applicable to all cases
(e.g., annual income is not applicable to children)
• Handling missing values
▫ Eliminate Data Objects
▫ Estimate Missing Values
▫ Ignore the Missing Value During Analysis
▫ Replace with all possible values (weighted by their probabilities)
9
Outliers
• Outliers are data objects that are considerably values of an attribute
that are unusual with respect to the typical values for that attribute.
Outliers
• Handle by Clustering
▫ detect and remove outliers
• Handle by using Semi-automated method (combined computer and
human inspection)
▫ detect suspicious values and check manually
• Handle by Regression
▫ smooth by fitting the data into regression functions
10
Correct Inconsistent Data
• Handling Correct Data
▫ Manual correction using external references
• Semi-automatic using various tools
▫ detect violation of known functional dependencies and data
constraints
▫ correct redundant data
11
Data Integration
• Combining two or more attributes (or objects) into a single attribute (or object)
• Purpose
▫ Data reduction
Reduce the number of attributes or objects
▫ Change of scale
Cities aggregated into regions, states, countries, etc
▫ More “stable” data
Aggregated data tends to have less variability
Data Transformation
• Attribute Selection: Creating new attribute from existing attributes for
more accuracy
• Normalization: scaled to fall within a specified range
▫ min-max normalization (scaled to a fixed range, usually [0,1] or [−1,1])
▫ z-score normalization/ Standardization (scaling data to have a mean of 0
and a standard deviation of 1)
▫ normalization by decimal scaling (moving the decimal point, dividing each
value by 10j)
13
Data Reduction
• Obtains a reduced representation of the data set that is much smaller in
volume but produces the same (or almost the same) analytical results.
• Purpose:
▫ Avoid curse of dimensionality
▫ Reduce amount of time and memory required by data mining algorithms
▫ Allow data to be more easily visualized
▫ May help to eliminate irrelevant features or reduce noise
14
Data Reduction
• Aggregation: summarization (Instead of storing daily sales data, aggregate it to
weekly or monthly sales data)
• Dimensionality Reduction (Reducing the number of attributes (features) in the
dataset)
• Data Compression (Representing data in a compressed format)
• Data Discretization and Binning (Reducing the range of data by dividing it into
bins or categories)
➢ Eg., Grouping ages into ranges like 0–18, 19–35, 36–60, etc
15
Filling Missing Value: Data Cleaning
• Simple Linear Regression:
16
Eg: Simple Linear Regression:
17
Eg: Simple Linear Regression:
18
Data Transformation
• Eg-1:
Use Manhattan Distance:
19
Data Transformation
20
• Eg-2 :
Nominal
Data
(0, 1)
Ordinal Data (salesperson [0],
Assistance-Manager [1], Manager
[2])
21
• Eg-2 :
22
• Eg-2 :
Problem: Large Difference values
23
Standardization and Normalization
24
Standardization [For Age]
25
Standardization [For Age]
For Age = 45:
26
Standardization [For Salary]
27
Standardization [For Salary]
For Salary = 8:
Ex: Calculate all the remaining values (age & salary) using standardization.
29
Next Week Lecture
• Supervised Learning:
Classification with Nearest Neighbor Classifier (KNN)
Thank You