Data Mining
Data Preprocessing
(Data Cleaning)
Dr. Saumendra Pattnaik
Assistant Professor
Department of Computer Science and Engineering
ITER, Siksha ‘O’ Anusandhan University.
Content
Data cleaning
• Missing Value
• Noisy Data
• Data Cleaning as a Process
Summary
11/1/2024 Data Preprocessing (Data Cleaning) 2
Data Cleaning
• Data cleaning tasks
• Fill in missing values
• Identify outliers and smooth out noisy data
• Correct inconsistent data
11/1/2024 Data Preprocessing (Data Cleaning) 3
Missing Data
• Data is not always available
• E.g., many tuples have no recorded value for several attributes, such as customer
income in sales data
• Missing data may be due to
• equipment malfunction
• inconsistent with other recorded data and thus deleted
• data not entered due to misunderstanding
• certain data may not be considered important at the time of entry
• not register history or changes of the data
11/1/2024 Data Preprocessing (Data Cleaning) 4
How to Handle Missing Data?
• Missing data may need to be inferred.
• Ignore the tuple:
• usually done when class label is missing (assuming the tasks in classification—not
effective when the percentage of missing values per attribute varies considerably.
• Fill in the missing value manually:
• tedious + infeasible.
• Use a global constant to fill in the missing value:
• Replace all missing attribute values by the same constant, such as a label like
“Unknown” a new class.
11/1/2024 Data Preprocessing (Data Cleaning) 5
How to Handle Missing Data? [Cont..]
• Use the attribute mean to fill in the missing value
• Use the attribute mean for all samples belonging to the same class to fill in the
missing value:
• smarter
• Use the most probable value to fill in the missing value:
• inference-based such as Bayesian formula or decision tree.
11/1/2024 Data Preprocessing (Data Cleaning) 6
Noisy Data
Noise: random error or variance in a measured variable
Incorrect attribute values may due to
• faulty data collection instruments
• data entry problems
• data transmission problems
• technology limitation
• inconsistency in naming convention
Other data problems which requires data cleaning
• duplicate records
• incomplete data
• inconsistent data
11/1/2024 Data Preprocessing (Data Cleaning) 7
How to Handle Noisy Data?
Binning method:
• First sort data and the sorted values are distributed into a number of “buckets,”
or bins.
• Then one can smooth by bin means, smooth by bin median, smooth by bin
boundaries, etc.
Smoothing by bin means
• Each value in a bin is replaced by the mean value of the bin.
Smoothing by bin medians
• Each bin value is replaced by the bin median.
Smoothing by bin boundaries
• The minimum and maximum values in a given bin are identified as the bin boundaries.
• Each bin value is then replaced by the closest boundary value.
11/1/2024 Data Preprocessing (Data Cleaning) 8
Simple Discretization Methods: Binning
Equal-width (distance) partitioning:
• It divides the range into N intervals of equal size: uniform grid
• if A and B are the lowest and highest values of the attribute, the width of
intervals will be: W = (B-A)/N.
• The most straight forward
• But outliers may dominate presentation
• Skewed data is not handled well.
Equal-depth (frequency) partitioning:
• It divides the range into N intervals, each containing approximately same
number of samples
• Good data scaling
• Managing categorical attributes can be tricky.
11/1/2024 Data Preprocessing (Data Cleaning) 9
Binning Methods for Data Smoothing
* Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
Partition into (equi-depth/equal-frequency) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
11/1/2024 Data Preprocessing (Data Cleaning) 10
Binning Methods for Data Smoothing [Cont..]
* Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
Smoothing by bin medians:
- Bin 1: 8.5, 8.5, 8.5, 8.5
- Bin 2: 22.5, 22.5, 22.5, 22.5
- Bin 3: 28.5, 28.5, 28.5, 28.5
Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
11/1/2024 Data Preprocessing (Data Cleaning) 11
Binning Methods for Data Smoothing [Cont..]
* Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
Partition into (equi-width) bins:
• Lowest value A=4
• Highest value B=34
• Width of Interval W=34-4/3=10
- Bin 1: Range(4 to 14) : 4, 8, 9
- Bin 2: Range(15 to 25) : 15, 21, 21, 24
- Bin 3: Range(26 to 34) :25, 26, 28, 29, 34
11/1/2024 Data Preprocessing (Data Cleaning) 12
Clustering & Regression
Clustering
• detect and remove outliers
Regression
• smooth by fitting the data into regression functions
Linear regression
• Involves finding the “best” line to fit two attributes (or variables), so that one attribute can
be used to predict the other.
Multiple linear regression
• Is an extension of linear regression, where more than two attributes are involved and the
data are fit to a multidimensional surface.
11/1/2024 Data Preprocessing (Data Cleaning) 13
Data Cleaning as a Process
• We have two steps for data cleaning as a process
Step-1: Discrepancy Detection
• Discrepancies can be caused by several factors:
• poorly designed data entry
• human error in data entry,
• deliberate errors
• data decay
• inconsistent data representations
• inconsistent use of codes
11/1/2024 Data Preprocessing (Data Cleaning) 14
Data Cleaning as a Process [Cont..]
How can we proceed with discrepancy detection?
• We may use knowledge regarding properties of the data.
• Such knowledge or “data about data” is referred to as metadata.
• Example:
• what are the domain and data type of each attribute?
• What are the acceptable values for each attribute?
• What is the range of the length of values?
• Do all values fall within the expected range?
• Are there any known dependencies between attributes?
11/1/2024 Data Preprocessing (Data Cleaning) 15
Data Cleaning as a Process [Cont..]
Commercial tools for discrepancy detection.
Data scrubbing tools
• Use simple domain knowledge to detect errors and make corrections in the
data.
• These tools rely on parsing and fuzzy matching techniques when cleaning
data from multiple sources.
Data auditing tools
• Find discrepancies by analyzing the data to discover rules and relationships,
and detecting data that violate such conditions.
11/1/2024 Data Preprocessing (Data Cleaning) 16
Data Cleaning as a Process[ Cont..]
Step-2: Data Transformations
• Data inconsistencies may be corrected manually using external references.
• Example, errors made at data entry may be corrected by performing a paper trace.
• Discrepancies can be corrected by using transformations.
11/1/2024 Data Preprocessing (Data Cleaning) 17
Data Cleaning as a Process [Cont..]
Commercial tools for data transformations
Data migration tools
• Allow simple transformations to be specified, such as to replace the string “gender”
by “sex”.
ETL (extraction/transformation/loading) tools
• allow users to specify transforms through a graphical user interface (GUI).
11/1/2024 Data Preprocessing (Data Cleaning) 18
Summary
• Data cleaning routines attempt to fill in missing values, smooth out noise while
identifying outliers, and correct inconsistencies in the data.
• Data cleaning is usually performed as an iterative two-step process consisting of
discrepancy detection and data transformation.
11/1/2024 Data Preprocessing (Data Cleaning) 19
Thank You
11/1/2024 Data Preprocessing (Data Cleaning) 20