Data Preprocessing ([Link]
)
Data Preprocessing is a crucial step in data mining. It involves transforming raw data into a clean, structured,
and suitable format for mining. Proper data preprocessing helps improve the quality of the data, enhances the
performance of algorithms, and ensures more accurate and reliable results.
Why Preprocess the Data?
In the real world, many databases and data warehouses have noisy, missing, and inconsistent data due to
their huge size. Low quality data leads to low quality data mining.
Noisy: Containing errors or outliers. E.g., Salary = “-10”
Noisy data may come from
Human or computer error at data entry.
Errors in data transmission.
Missing: lacking certain attribute values or containing only aggregate data. E.g., Occupation = “”
Missing (Incomplete) may data come from
“Not applicable” data value when collected.
Human/hardware/software problems.
Inconsistent: Data inconsistency meaning is that different versions of the same data appear in different places.
For example, the ZIP code is saved in one table as 1234-567 numeric data format; while in another table it may be
represented in 1234567.
Inconsistent data may come from
Errors in data entry.
Merging data from different sources with varying formats.
Differences in the data collection process.
Advantages of Data Preprocessing
Improves data quality.
Increases mining accuracy.
Reduces execution time.
Eliminates redundant data.
Improves classifier performance.
Reduces storage requirements.
Major Tasks in Data Preprocessing
Data preprocessing is an essential step in the knowledge discovery process(KDD), because quality decisions
must be based on quality data. And Data Preprocessing involves Data Cleaning, Data Integration, Data
Reduction and Data Transformation.
1. Data Cleaning
Def: Data cleaning is a process that "cleans" the data by filling in the missing values, smoothing noisy data,
analyzing, and removing outliers, and removing inconsistencies in the data.
Missing Values
1. Imagine that you need to analyze All Electronics sales and customer data.
2. You note that many tuples have no recorded value for several attributes such as customer income.
3. How can you go about filling in the missing values for this attribute?
4. There are several methods to fill the missing values.
Those are,
a. Ignore the tuple: This is usually done when the class label is missing(classification). This method is
not very effective, unless the tuple contains several attributes with missing values.
b. Fill in the missing value manually: In general, this approach is time consuming and may not be
feasible given a large data set with many missing values.
c. Use a global constant to fill in the missing value: Replace all missing attribute values by the same
constant such as a label like “Unknown” or “- ∞ “.
d. Use the attribute mean or median to fill in the missing value: Replace all missing values in the
attribute by the mean or median of that attribute values.
Noisy Data
Noise is a random error or variance in a measured variable. Data smoothing techniques are used to eliminate noise
and extract the useful patterns. The different techniques used for data smoothing are:
a. Binning: Binning methods smooth a sorted data value by consulting its “neighbourhood,” that is, the
values around it. The sorted values are distributed into several “buckets,” or bins. Because binning
methods consult the neighbourhood of values, they perform local smoothing.
There are three kinds of binning. They are:
o Smoothing by Bin Means: In this method, each value in a bin is replaced by the mean value of
the bin. For example, the mean of the values 4, 8, and 15 in Bin 1 is 9. Therefore, each original
value in this bin is replaced by the value 9.
o Smoothing by Bin Medians: In this method, each value in a bin is replaced by the median value
of the bin. For example, the median of the values 4, 8, and 15 in Bin 1 is 8. Therefore, each
original value in this bin is replaced by the value 8.
o Smoothing by Bin Boundaries: In this method, the minimum and maximum values in each bin
are identified as the bin boundaries. Each bin value is then replaced by the closest boundary
value
o For example, the middle value of the values 4, 8, and 15 in Bin 1is replaced with nearest
boundary i.e., 4.
Example:
Sorted data for price (in dollars): 4, 8, 15, 21, 21, 24, 25, 28, 34
Partition into (equal-frequency) bins:
Bin 1: 4, 8, 15
Bin 2: 21, 21, 24
Bin 3: 25, 28, 34
Smoothing by bin means:
Bin 1: 9, 9, 9
Bin 2: 22, 22, 22
Bin 3: 29, 29, 29
Smoothing by bin medians:
Bin 1: 8, 8, 8
Bin 2: 21, 21, 21
Bin 3: 28, 28, 28
Smoothing by bin boundaries:
Bin 1: 4, 4, 15
Bin 2: 21, 21, 24
Bin 3: 25, 25, 34
Real-Time Example 1: Student Marks Analysis
A university smooths students' marks before generating overall academic performance reports.
Real-Time Example 2: Daily Temperature
Weather departments smooth temperature data to remove sensor fluctuations.
Real-Time Example 3: Online Shopping Orders
Hospitals group patient ages before statistical analysis to reduce minor age variations.
Real-Time Example 5: Monthly Electricity Consumption
Electricity boards smooth consumption data before forecasting future power demand.
b. Regression: Data smoothing can also be done by regression, a technique that used to predict the
numeric values in a given data set. It analyses the relationship between a target variable (dependent) and
its predictor variable (independent).
o Regression is a form of a supervised machine learning technique that tries to predict any
continuous valued attribute.
o Regression done in two ways; Linear regression involves finding the “best” line to fit two
attributes (or variables) so that one attribute can be used to predict the other. Multiple linear
regression is an extension of linear regression, where more than two attributes are involved and
the data are fit to a multidimensional surface.
c. Clustering: It supports in identifying the outliers. The similar values are organized into clusters and
those values which fall outside the cluster are known as outliers.
2. Data Integration
Data integration is the process of combining data from multiple sources into a single, unified view. This
process involves identifying and accessing the different data sources, mapping the data to a common format.
Different data sources may include multiple data cubes, databases, or flat files.
Why is Data Integration Needed?
Organizations store data in different systems such as:
Sales Database
Customer Database
HR Database
Inventory Database
Online Shopping Website
To perform accurate data mining, all these datasets must be integrated into one database.
Real-Time Example 1: Online Shopping (Amazon/Flipkart)
Suppose an e-commerce company stores information in different databases.
Customer Database
Customer ID Name City
101 Ravi Hyderabad
102 Anita Vijayawada
Orders Database
Customer ID Product Amount
101 Laptop 60000
102 Mobile 25000
Payment Database
Customer ID Payment Mode
101 UPI
102 Credit Card
After Data Integration
Customer ID Name City Product Amount Payment
101 Ravi Hyderabad Laptop 60000 UPI
102 Anita Vijayawada Mobile 25000 Credit Card
Benefit
The company can now analyze:
Customer buying behavior
Preferred payment methods
Sales by city
Real-Time Example 3: University Management System
Separate databases:
Student Details
Attendance
Examination
Fee Payment
Integrated Database
Roll No Name Attendance Marks Fee Status
101 Rahul 92% 85 Paid
102 Priya 88% 90 Pending
Benefit
The university can identify students with:
Low attendance
Fee dues
Poor academic performance
Data Integration Process:
Issues in Data Integration
1. Entity Identification Problem: The same person may have different names in different databases.
Example:
Database 1
ID Name
101 R. Kumar
Database 2
ID Name
101 Raj Kumar
Both refer to the same person.
2. Redundancy: Duplicate information exists.
Example:
Sales Database
Product Price
Laptop 50000
Inventory Database
Product Price
Laptop 50000
The same information appears twice.
3. Data Value Conflict: Different databases contain different values for the same attribute.
Example:
Database 1: Salary = ₹50,000
Database 2: Salary = ₹52,000
Which one is correct?
4. Naming Conflict: Different attribute names refer to the same data.
Example:
Database A: Cust_ID
Database B: Customer_Number
Both represent Customer ID.
5. Data Format Conflict: Different formats for the same data.
Example
Database A: 04-08-2026
Database B: 2026/08/04
Need to convert to one standard format.
Advantages of Data Integration
Combines data from multiple sources.
Eliminates duplicate records.
Improves data quality.
Supports better decision-making.
Provides a unified view of data.
Increases accuracy of data mining.
Saves storage space by reducing redundancy.
Disadvantages
Time-consuming for very large databases.
Complex due to different data formats.
Data conflicts need manual resolution.
High implementation cost.
Requires continuous maintenance.
Applications
Banking
Healthcare
E-commerce
Education
Government departments
Insurance
Social media analytics
Business intelligence
3. Data Reduction
Imagine that you have selected data from the AllElectronics data warehouse for analysis. The data set will
likely be huge! Complex data analysis and mining on huge amounts of data can take a long time, making such
analysis impractical or infeasible.
Data reduction techniques can be applied to obtain a reduced representation of the data set that is much
smaller in volume, yet closely maintains the integrity of the original data. That is, mining on the reduced data set
should be more efficient yet produce the same (or almost the same) analytical results.
In simple words, Data reduction is a technique used in data mining to reduce the size of a dataset while still
preserving the most important information. This can be beneficial in situations where the dataset is too large to be
processed efficiently, or where the dataset contains a large amount of irrelevant or redundant information.
There are several different data reduction techniques that can be used in data mining, including:
a. Data Sampling: This technique involves selecting a subset of the data to work with, rather than using the
entire dataset. This can be useful for reducing the size of a dataset while still preserving the overall
trends and patterns in the data.
b. Dimensionality Reduction: This technique involves reducing the number of features in the dataset,
either by removing features that are not relevant or by combining multiple features into a single feature.
c. Data compression: This is the process of altering, encoding, or transforming the structure of data in
order to save space. By reducing duplication and encoding data in binary form, data compression creates
a compact representation of information. And it involves the techniques such as lossy or lossless
compression to reduce the size of a dataset.
4. Data Transformation
Data transformation in data mining refers to the process of converting raw data into a format that is suitable
for analysis and modelling. The goal of data transformation is to prepare the data for data mining so that it can be
used to extract useful insights and knowledge.
Data transformation typically involves several steps, including:
1. Smoothing: It is a process that is used to remove noise from the dataset using techniques include
binning, regression, and clustering.
2. Attribute construction (or feature construction): In this, new attributes are constructed and added
from the given set of attributes to help the mining process.
3. Aggregation: In this, summary or aggregation operations are applied to the data. For example, the daily
sales data may be aggregated to compute monthly and annual total amounts.
4. Data normalization: This process involves converting all data variables into a small range. such as -1.0
to 1.0, or 0.0 to 1.0.
5. Generalization: It converts low-level data attributes to high-level data attributes using concept
hierarchy. For Example, Age initially in Numerical form (22, ) is converted into categorical value
(young, old).