Data Preprocessing
Introduction
• Real-world databases often contain noisy, incomplete, and inconsistent
data, which affects analysis accuracy.
• Data is usually collected from multiple heterogeneous sources, making it
difficult to manage and integrate.
• Low-quality data leads to low-quality data mining results, reducing the
reliability of insights.
• Data preprocessing is essential to improve data quality and enhance the
efficiency and effectiveness of data mining processes.
Data preprocessing is the process of transforming raw data into an
understandable format.
Data preprocessing is an important step in the knowledge discovery
process, because quality decisions must be based on quality data.
Data Quality: Why Preprocess the Data?
• Data is considered high quality if it meets the requirements of its
intended use.
• Several factors define data quality, such as accuracy, completeness,
consistency, timeliness, believability, and interpretability.
• Poor data quality can lead to misleading results and wrong decisions.
• Preprocessing improves data quality, making it more suitable and reliable
for analysis or mining.
The quality can be checked by the following :
1. Accuracy: To check whether the data entered is correct or not.
2. Completeness: To check whether the data is available or not
recorded.
3. Consistency: To check whether the same data is kept in all the places
that do or do not match.
4. Timeliness: The data should be updated correctly.
5. Believability: The data should be trustable.
6. Interpretability: The understandability of the data
Major Tasks in Data Preprocessing
There are four main tasks in data preprocessing:
• Data Cleaning – Handles missing values, noise, and
inconsistencies in data.
• Data Integration – Combines data from multiple
sources into a unified format.
• Data Reduction – Reduces the volume of data while
maintaining its integrity.
• Data Transformation – Converts data into suitable
formats or scales for analysis.
These steps help improve the quality and efficiency of
data mining.
Data Cleaning
Incomplete (Missing) Data:
Handling Missing Data
1. Ignore the tuple: usually done when class label is missing
2. Fill in the missing value manually.
3. Use a measure of central tendency for the attribute (e.g., the mean
or median) to fill in the missing value
4. Use the attribute mean or median for all samples belonging to the
same class as the given tuple.
5. Use a global constant to fill in the missing values.
6. Use the most probable value to fill in the missing value.
Noisy Data
Noisy data means data that contains errors, random variations, or
irrelevant information.
This can make it difficult to analyze or get accurate results.
• Incorrect attribute values may be due to
• faulty data collection instruments
• data entry problems
• data transmission problems
• technology limitation
• inconsistency in naming convention
• Other data problems which require data cleaning
• duplicate records
• incomplete data
• inconsistent data
How to Handle Noisy Data
Binning
• first sort data and partition into (equal-frequency) bins
• then one can smooth by bin means, smooth by bin median, smooth by
bin boundaries, etc.
Regression
• smooth by fitting the data into regression functions
Clustering
• detect and remove outliers
Combined computer and human inspection
• detect suspicious values and check by human (e.g., deal with possible
outliers)
Binning – A Data Smoothing Technique
• Binning smooths data by grouping sorted values into small groups called
bins.
• Each bin replaces its values with a representative value to reduce noise.
1. In smoothing by bin means, all values in a bin are replaced with the bin’s
average.
2. In smoothing by bin medians, values are replaced with the bin’s median.
3. In smoothing by bin boundaries, each value is replaced with the closest
bin boundary (minimum or maximum).
• Bins can be created using equal-frequency (same number of values) or
equal-width (same range of values).
• Binning performs local smoothing by considering neighboring values.
• Larger bin sizes lead to greater smoothing but may reduce data accuracy.
• Binning is also used as a data discretization technique to convert numeric
data into categories.
Regression
• Regression is a predictive modeling technique used to find the
relationship between variables and predict future values.
• In linear regression, a straight line is fit between two variables to predict
one from the other.
• The goal is to find the best-fit line that minimizes prediction errors.
• Multiple linear regression extends this concept to more than two
variables, fitting data to a surface in multi-dimensional space.
• Regression helps in identifying trends and can be used for forecasting
values.
• It is useful for data smoothing, data analysis, and predictive modeling.
Outlier Analysis
Outlier analysis is the process of identifying data points that are very different
from most of the data.
Clustering-based detection:
• Data is grouped into clusters (similar values together).
• Points that do not belong to any cluster or are far away from all clusters
are potential outliers.
• Clustering is useful for data analysis, pattern discovery, and noise
detection.
Data Integration
• Data integration is the process of combining data from multiple sources
into a unified and coherent data store.
• It is essential for preparing high-quality data for mining and analysis.
Issues in Data Integration:
1. Schema Integration and Object Matching.
2. Redundancy.
3. Detection and resolution of data value conflicts
Entity Identification Problem
• Data integration combines data from multiple sources like databases, data cubes, or
flat files.
• Challenge: Matching the same real-world entities across different sources.
• Metadata (name, meaning, data type, value range, null rules) helps correctly identify
and match attributes.
• Metadata also aids data transformation when codes differ
• Attention to data structure is crucial—e.g., discounts applied at order level vs item
level—to avoid incorrect data after integration.
• This problem is linked to schema integration and data cleaning processes.
Redundancy and Correlation Analysis
• Redundancy is another important issue in data integration.
• An attribute (such as annual revenue, for instance) may be redundant if
it can be “derived” from another attribute or set of attributes.
• Inconsistencies in attribute or dimension naming can also cause
redundancies in the resulting data set.
• Some redundancies can be detected by correlation analysis.
For nominal data (categories like colors or types), we use the chi-square
test to check if two attributes are related.
For numeric data (numbers like age or income), we use the correlation
coefficient and covariance to see how the values of one attribute change
compared to another.
Correlation Analysis (Nominal Data)
Χ2 (chi-square) test
Correlation Coefficient for Numeric Data
Covariance of Numeric Data
Covariance measures how two numeric attributes vary together:
Positive covariance → When one increases, the other tends to increase.
Negative covariance → When one increases, the other tends to decrease.
Covariance ≈ 0 → No consistent relationship in how they vary.
Calculate the correlation coefficient (Pearson’s product moment
coefficient). Are these two attributes positively or negatively correlated?
Compute their covariance.
Tuple Duplication
• Duplicate tuples (rows) may occur when the same data entry appears
more than once.
• Duplicates can result from denormalized tables, often used to improve
performance by avoiding joins.
• These duplicates can cause inconsistencies due to data entry errors or
partial updates
Example: In a purchase order database, if the purchaser’s name and
address are stored directly (not as a reference key), the same purchaser
might appear with different addresses.
This leads to conflicting records for the same real-world entity.
Data Value Conflict Detection and Resolution
When integrating data, the same entity’s attribute values may differ across
sources due to:
• Different units or scales (e.g., metric vs. imperial weight),
• Different currencies, taxes, or services (e.g., hotel prices),
• Different grading systems or course structures (e.g., university grading
scales),
• Different levels of detail (e.g., sales for a single store vs. a whole
region).
These conflicts make data integration challenging.
Detecting and resolving such conflicts is essential for clean, consistent data.
Data Transformation and Data Discretization
Data Transformation is a key step in data preprocessing where raw data is
converted into a suitable format or structure for analysis or mining.
• Smoothing: Remove noise from the data (Techniques include binning,
regression, and clustering)
• Attribute Construction: Create new attributes from existing ones
• Aggregation: Summarize data (e.g., daily sales → monthly sales).
• Normalization: Scale data to a smaller range (e.g., 0.0 to 1.0).
• Discretization: Replace numeric values with intervals or conceptual labels
(e.g., age → "youth," "adult," "senior").
• Concept Hierarchy Generation: Generalize data to higher-level concepts
(e.g., street → city → country).
Data Normalization
• Different attributes may have different units or scales (e.g., height in
meters vs. weight in kilograms).
• Without normalization, attributes with larger ranges can dominate or bias
the results in data analysis or mining.
• Normalization helps algorithms (like neural networks or clustering) work
better by treating all data fairly.
Common methods:
Min-Max Normalization: Rescales values to a fixed range (e.g., 0 to 1).
Z-Score Normalization: Adjusts data to have a mean of 0 and standard
deviation of 1.
Decimal Scaling: Moves decimal points to bring values into a smaller range.
Data discretization
Data Discretization is a data preprocessing technique that converts
continuous numerical data into discrete, categorical intervals or bins.
This transformation simplifies the data, making it easier to analyze and
interpret. Instead of handling a wide range of continuous values, data
mining works with a limited set of categories.
Suppose we have an attribute of Age with the given values
Types of Discretization Techniques
Based on Class Information:
• Supervised Discretization: Uses class labels (known categories) to guide
how data is divided.
• Unsupervised Discretization: Does not use class labels; divides data
purely based on data distribution.
Based on Direction:
• Top-Down Discretization (Splitting):
Starts by finding one or a few split points to divide the entire data range.
Then, it repeats this splitting recursively on each smaller interval.
• Bottom-Up Discretization (Merging):
Starts with all possible split points (each value treated separately) and
merges neighboring intervals step-by-step to form bigger intervals. This
merging is applied recursively.
Supervised Discretization
Uses class labels to decide splits. Temperature (° Weather Type
Example: Temperatures and Weather Types: C) (Class)
• 15–19°C → Cold
15 Cold
• 20–27°C → Mild
• 28–32°C → Hot 18 Cold
22 Mild
Unsupervised Discretization
• Does NOT use class labels. 25 Mild
Example: Temperatures only:
30 Hot
• 15–22°C
• 23–30°C 32 Hot
• 31–32°C
Discretization by Binning
• Binning splits continuous data into a fixed number of bins or intervals.
• It’s a top-down method that groups data values based on equal width
(same size intervals) or equal frequency (same number of data points
per bin).
• Each bin’s values can be replaced by the bin’s mean or median to
smooth the data.
• This method helps reduce data size and create concept hierarchies by
grouping values step-by-step.
• Binning is unsupervised, meaning it doesn’t use any class labels.
• It depends on how many bins you choose and can be affected by
extreme values (outliers).
Discretization by Histogram Analysis
• Histogram analysis is an unsupervised discretization technique because it
does not use class labels.
• It divides the values of an attribute into separate, non-overlapping ranges
called buckets or bins.
There are different ways to create these buckets:
• Equal-width histogram: Each bucket has the same size
• Equal-frequency histogram: Buckets are made so that each contains
roughly the same number of data points.
• The method can be applied recursively on these buckets to create multiple
levels of groups, forming a concept hierarchy.
• The process stops when the desired number of levels is reached or when
buckets get too small (based on a minimum width or number of data
points).
• Histograms can also be created by grouping data based on cluster analysis
of data distribution.
Discretization by Cluster, Decision Tree, and Correlation Analysis
Cluster Analysis
Groups numeric attribute values into clusters based on similarity and
closeness.
Can be done in two ways:
• Top-down: Start with large clusters and split them into smaller
subclusters.
• Bottom-up: Start with many small clusters and merge neighboring
ones to form bigger clusters.
Each cluster forms a node in a concept hierarchy, helping organize data at
different levels.
Decision Tree Analysis for Discretization
• Uses class labels (supervised method).
• Splits numeric data into intervals that best separate different classes.
• Uses measures like entropy to find the best split points.
• Recursively splits data until good class separation is achieved, forming a
concept hierarchy.
• Helps improve classification accuracy by choosing meaningful splits.
Correlation Analysis for Discretization
• A bottom-up, supervised method using class labels.
• Starts with each distinct value as its own interval.
• Performs a chi-square (χ²) test on adjacent intervals to check similarity in
class distributions.
• Merges adjacent intervals with similar class distributions (low χ² value).
• Continues merging until a stopping condition is met, forming larger
intervals.
Concept Hierarchy Generation
• Concept hierarchy organizes concepts (i.e., attribute values) hierarchically
and is usually associated with each dimension in a data warehouse
• Concept hierarchies facilitate drilling and rolling in data warehouses to view
data in multiple granularity
• Concept hierarchy formation: Recursively reduce the data by collecting and
replacing low level concepts (such as numeric values for age) by higher level
concepts (such as youth, adult, or senior)
• Concept hierarchies can be explicitly specified by domain experts and/or data
warehouse designers
Concept hierarchy can be automatically formed for both numeric and nominal
data.
Concept Hierarchy Generation for Nominal Data
Four methods for concept hierarchy generation (nominal data):
1. Specification of a partial ordering of attributes explicitly at the schema
level by users or experts.
2. Specification of a portion of a hierarchy by explicit data grouping.
3. Specification of a set of attributes, but not their partial ordering →
automatic generation based on the number of distinct values.
4. Specification of only a partial set of attributes → system fills missing ones
using semantic connections.