Data preprocessing
Tasks in data preprocessing
Data Cleaning: It is also known as scrubbing. This task involves filling of
missing values, smoothing or removing noisy data and outliers along
with resolving inconsistencies.
Data Integration: This task involves integrating data from multiple
sources such as databases (relational and non-relational), data cubes,
files, etc. The data sources can be homogeneous or heterogeneous. The
data obtained from the sources can be structured, unstructured or
semi-structured in format.
Data Transformation: This involves normalisation and aggregation of
data according to the needs of the data set.
Data Reduction: During this step data is reduced. The number of
records or the number of attributes or dimensions can be reduced.
Reduction is performed by keeping in mind that reduced data should
produce the same results as original data.
Data Discretization: It is considered as a part of data reduction. The
numerical attributes are replaced with nominal ones.
Data Integration
Data transformation
• [Link]:
It is a process that is used to remove noise from the dataset
using some algorithms It allows for highlighting important
features present in the dataset. It helps in predicting the
patterns. When collecting data, it can be manipulated to
eliminate or reduce any variance or any other noise form.
The concept behind data smoothing is that it will be able to
identify simple changes to help predict different trends and
patterns. This serves as a help to analysts or traders who
need to look at a lot of data which can often be difficult to
digest for finding patterns that they wouldn’t see otherwise.
Data transformation
• 2. Aggregation:
Data collection or aggregation is the method of storing and
presenting data in a summary format. The data may be obtained
from multiple data sources to integrate these data sources into a
data analysis description. This is a crucial step since the accuracy of
data analysis insights is highly dependent on the quantity and
quality of the data used. Gathering accurate data of high quality and
a large enough quantity is necessary to produce relevant results.
• The collection of data is useful for everything from decisions
concerning financing or business strategy of the product, pricing,
operations, and marketing strategies.
• For example, Sales, data may be aggregated to compute monthly&
annual total amounts.
Data transformation
• [Link]:
It is a process of transforming continuous data into set of small intervals.
Most Data Mining activities in the real world require continuous
attributes. Yet many of the existing data mining frameworks are unable to
handle these attributes.
• Also, even if a data mining task can manage a continuous attribute, it can
significantly improve its efficiency by replacing a constant quality attribute
with its discrete values.
• For example, (1-10, 11-20) (age:- young, middle age, senior).
• 4. Attribute Construction:
Where new attributes are created & applied to assist the mining process
from the given set of attributes. This simplifies the original data & makes
the mining more efficient.
Data transformation
• 5. Generalization:
It converts low-level data attributes to high-level
data attributes using concept hierarchy. For
Example Age initially in Numerical form (22, 25)
is converted into categorical value (young, old).
• For example, Categorical attributes, such as
house addresses, may be generalized to higher-
level definitions, such as town or country.
Data transformation
• 5. Generalization:
It converts low-level data attributes to high-level
data attributes using concept hierarchy. For
Example Age initially in Numerical form (22, 25)
is converted into categorical value (young, old).
• For example, Categorical attributes, such as
house addresses, may be generalized to higher-
level definitions, such as town or country.
Data transformation
6. Normalization: Data normalization involves
converting
• all data variable into a given range.
Techniques that are used for normalization
are:
10
• min-max
normalization
Normalization
v min v
v' (new _ max v v
max v new_min ) new_minv
min v
• z-score
normalization
vv does not eliminate
v'
v
outliers
• normalization by decimal
scaling
v Where j is the smallest integer such that Max(| v'
v'
10
|)<1
j
range: -986 to 917 => j=3 -986 -> -0.986 917 - 11
> 0.917
Normalization
• min-max
normalization
v min v
v' (new _ max v v
max v new_min ) new_minv
min v
• z-score
normalization
vv does not eliminate
v'
v
outliers
• normalization by decimal
scaling
v Where j is the smallest integer such that Max(| v'
v'
10
|)<1
j
range: -986 to 917 => j=3 -986 -> -0.986 917 -> 12
0.917
Min max
Let’s Understand it with an example: Suppose the minimum and
maximum values for the price of the house be $125,000 and $925,000
respectively. We need to normalize that price range in between (0,1). We
can use min-max normalization to transform any value between them
(say, 300,000). In this case, we use the above formula to find U i with,
v min v
v' (new _ max v v
max v new_min ) new_minv
min v
Vi=300,000
X1= 125,000
X2= 925,000
Y1= 0
Y2= 1
Normalization
• Z-Score Normalization:
– In z-score normalization (or zero-mean normalization) the values of an
attribute (A), are normalized based on the mean of A and its standard
deviation
– A value, v, of attribute A is normalized to v’ by computing
Vi’= (vi-(Mean A))/(standard deviation)
• For example:
Let mean of an attribute P = 60, 000, Standard Deviation = 10, 000,
for the attribute P. Using z-score normalization, a value of 85000 for
P can be transformed to:
• And hence we get the value of v’ to be 2.5
Normalization
• Decimal Scaling:
– It normalizes the values of an attribute by changing the
position of their decimal points
– The number of points by which the decimal point is moved can
be determined by the absolute maximum value of attribute A.
• Normalization by decimal scaling normalizes by moving
the decimal point of values of attribute A. The number
of decimal points moved depends on the maximum
absolute value of A. A value, v , of A is normalized to v by
i 0i
computing
• vi’=vi/10 power j
Data Reduction Techniques
Data cube aggregation: In this technique the data is reduced by applying
OLAP operations like slice, dice or rollup. It uses the smallest level necessary
to solve the problem.
Dimensionality reduction: The data attributes or dimensions are reduced.
Not all attributes are required for data mining. The most suitable subset of
attributes are selected by using techniques like forward selection, backward
elimination, decision tree induction or a combination of forward selection
and backward elimination.
Data compression: In this technique. large volumes of data is compressed
i.e. the number of bits used to store data is reduced. This can be done by
using lossy or lossless compression. In loss compression, the quality of data is
compromised for more compression. In lossless compression, the quality of
data is not compromised for higher compression level.
Numerosity reduction : This technique reduces the volume of data by
choosing smaller forms for data representation. Numerosity reduction can be
done using histograms, clustering or sampling of data. Numerosity reduction
is necessary as processing the entire data set is expensive and time
consuming.