What does Data Preprocessing mean?
Data preprocessing is a data mining technique that involves transforming the
collected raw data into an understandable format. Real-world data is often
incomplete, inconsistent, and/or lacking in certain behaviors or trends, and is
likely to contain many errors. Data preprocessing is a proven method of
resolving such issues. Data preprocessing prepares raw data for further
processing like data mining or storing them into data warehouse.
Data goes through different steps during preprocessing:
• Data Cleaning: Data is cleansed through processes such as filling in
missing values, smoothing the noisy data, or resolving the
inconsistencies in the data.
• Data Integration: Data with different representations are put together
and conflicts within the data are resolved.
• Data Transformation: Data is normalized, aggregated and generalized.
• Data Reduction: This step aims to present a reduced representation of
the data in a data warehouse.
• Data Discretization: Involves the reduction of a number of values of a
continuous attribute by dividing the range of attribute intervals.
Descriptive Data Summarization
For data preprocessing to be successful, it is essential to have an overall
picture of your data. Descriptive data summarization techniques can be used
to identify the typical properties of your data and highlight which data values
should be treated as noise or outliers. Thus, we first introduce the basic
concepts of descriptive data summarization before getting into the concrete
workings of data preprocessing techniques. For many data preprocessing
tasks, users would like to learn about data characteristics regarding both
central tendency and dispersion of the data. Measures of central tendency
include mean, median, mode, and midrange, while measures of data dispersion
include quartiles, interquartile range (IQR), and variance. These descriptive
statistics are of great help in understanding the distribution of the data. Such
measures have been studied extensively in the statistical literature. From the
data mining point of view, we need to examine how they can be computed
efficiently in large databases.
In particular, it is necessary to introduce the notions of distributive measure,
algebraic measure, and holistic measure. Knowing what kind of measure we
are dealing with can help us choose an efficient implementation for Data
warehouse.
• Measuring the Central Tendency
The central tendency of data can be measured by Mean, Median, or Mode. The
most common and most effective numerical measure of the “center” of a set of
data is the (arithmetic) mean. Let x1;x2; : : : ;xN be a set of N values or
observations, such as for some attribute, like salary. The mean of this set of
values is
Sometimes, each value xi in a set may be associated with a weight wi, for i = 1;
: : : ;N. The weights reflect the significance, importance, or occurrence
frequency attached to their respective values. In this case, we can compute
Median is another way measuring central tendency:
Another measure of central tendency is the mode. The mode for a set of data is
the value that occurs most frequently in the set. It is possible for the greatest
frequency to correspond to several different values, which results in more
than one mode. Data sets with one, two, or three modes are respectively called
unimodal, bimodal, and trimodal. In general, a data set with two or more
modes is multimodal. At the other extreme, if each data value occurs only
once, then there is no mode.
Mean-mode = 3*(mean-median).
• Measuring the Dispersion of Data
The degree to which numerical data tend to spread is called the dispersion, or
variance of the data. The most common measures of data dispersion are
range, the five-number summary (based on quartiles), the interquartile range,
and the standard deviation.
The most commonly used percentiles other than the median are quartiles. The
first quartile, denoted by Q1, is the 25th percentile; the third quartile, denoted
by Q3, is the 75th percentile. The quartiles, including the median, give some
indication of the center, spread, and shape of a distribution. The distance
between the first and third quartiles is a simple measure of spread that gives
the range covered by the middle half of the data. This distance is called the
interquartile range (IQR) and is defined as
IQR = Q3-Q1.
A common rule of thumb for identifying suspected outliers is to single out
values falling at least 1.5*IQR above the third quartile or below the first
quartile.
This is known as the five-number summary. The five-number summary of a
distribution consists of the median, the quartiles Q1 and Q3, and the smallest
and largest individual observations, written in the order
[ Minimum; Q1; Median; Q3; Maximum]
The variance of N observations, x1,x2, ….., xN, is
• Graphic Displays of Basic Descriptive Data Summaries
Numerical:
Data Cleaning