Sampling
Sampling can be used as a data reduction technique because it allows a large
data set to be represented by a much smaller random data sample (or subset).
Suppose that a large data set, D, contains N tuples. Let’s look at the most
common ways that we could sample D for data reduction.
Simple random sample without replacement (SRSWOR)
Simple random sample with replacement (SRSWR)
Cluster sample
Stratified sampling
SRSWOR
Simple random sample without replacement (SRSWOR) of size
s: SRSWOR is a method of selection of n units out of the N units one by
one such that at any stage of selection, any one of the remaining units
have the same chance of being selected.
SRSWR
Simple random sample with replacement (SRSWR) of size s:
This is similar to SRSWOR, except that each time a tuple is drawn
from D, it is recorded and then replaced. That is, after a tuple is
drawn, it is placed back in D so that it may be drawn again. or
SRSWR is a method of selection of n units out of the N units one by one
such that at each stage of selection, each unit has an equal chance of
being selected.
Cluster sample
If the tuples in D are grouped into M mutually
disjoint “clusters,” then an SRS of s clusters can
be obtained, where s < M. For example, tuples in a
database are usually retrieved a page at a time, so
that each page can be considered a cluster. A
reduced data representation can be obtained by
applying, say, SRSWOR to the pages, resulting in a
cluster sample of the tuples. Other clustering
criteria conveying rich semantics can also be
explored. For example, in a spatial database, we
may choose to define clusters geographically
based on how closely different areas are located.
Stratified sample
If D is divided into mutually disjoint parts called strata, a stratified sample of
D is generated by obtaining an SRS at each stratum. This helps ensure a
representative sample, especially when the data are skewed. For example, a
stratified sample may be obtained from customer data, where a stratum is
created for each customer age group. In this way, the age group having the
smallest number of customers will be sure to be represented.
Advantage of sampling
An advantage of sampling for data reduction is that the cost of
obtaining a sample is proportional to the size of the sample, s, as
opposed to N, the data set size. Hence, sampling complexity is
potentially sublinear to the size of the data. Other data reduction
techniques can require at least one complete pass through D. For a
fixed sample size, sampling complexity increases only linearly as the
number of data dimensions, n, increases, whereas techniques using
histograms, for example, increase exponentially in n.
Data cleaning process contd..
The data should also be examined regarding unique rules, consecutive
rules, and null rules.
A unique rule says that each value of the given attribute must be
different from all other values for that attribute.
A consecutive rule says that there can be no missing values between the
lowest and highest values for the attribute, and that all values must also
be unique (e.g., as in check numbers).
A null rule specifies the use of blanks, question marks, special characters,
or other strings values should be handled.
Data integration
• Entity Identification Problem
• Redundancy and Correlation Analysis
• Tuple Duplication
• Data Value Conflict Detection and Resolution
Data integration
Entity Identification Problem
Schema integration and object matching can be tricky.
How can equivalent real-world entities from multiple data sources be matched up?
This is referred to as the entity identification problem.
For example, how can the data analyst or the computer be sure that customer_id
in one database and cust_number in another databases refer to the same
attribute? Examples of metadata for each attribute include the name, meaning,
data type, and range of values permitted for the attribute, and null rules for
handling blank, zero, or null values.
Then matching attributes from one database to another during integration, special
attention must be paid to the structure of the data.
Data integration
Redundancy and Correlation Analysis
• An attribute (such as annual revenue, for instance) may be redundant if it
can be “derived” from another attribute or set of attributes.
• Inconsistencies in attribute or dimension naming can also cause
redundancies in the resulting data set.
• Some redundancies can be detected by correlation analysis.
• For nominal data, we use the χ 2 (chi-square) test. For numeric attributes,
we can use the correlation coefficient and covariance, both of which access
how one attribute’s values vary from those of another
Data integration
Covariance Analysis
Data integration
Covariance Analysis
Q. Below table shows the stock prices observed at five time points for AllElectronics and HighTech, a
high-tech company. If the stocks are affected by the same industry trends, will their prices rise or fall
together?
Data integration
Covariance Analysis
Data integration Data integration
χ 2 Correlation Test for Nominal Data
For nominal data, a correlation relationship between two attributes, A and B, can be
discovered by a χ 2 (chi-square) test. Suppose A has c distinct values, namely a 1,a2,...ac .
B has r distinct values, namely b1,b2,...br . The data tuples described by A and B can be
shown as a contingency table, with the c values of A making up the columns and the r
values of B making up the rows. Let (Ai ,Bj) denote the joint event that attribute A takes
on value ai and attribute B takes on value bj , that is, where (A = ai ,B = bj). Each and
every possible (Ai ,Bj) joint event has its own cell (or slot) in the table. The χ 2 value
(also known as the Pearson χ 2 statistic) is computed as
where oij is the observed frequency (i.e., actual count) of the joint event (A i ,Bj) and eij
is the expected frequency of (Ai ,Bj), which can be computed as
Data integration
χ 2 Correlation Test
where n is the number of data tuples, count(A = ai) is the number of tuples having
value ai for A, and count(B = bj) is the number of tuples having value bj for B.
Data integration
Data integration
χ 2 Correlation Test
The χ 2 statistic tests the hypothesis that A and B are independent, that is, there is no correlation between them.
The test is based on a significance level, with (r − 1) × (c − 1) degrees of freedom. We illustrate the use of this
statistic in Example 3.1. If the hypothesis can be rejected, then we say that A and B are statistically correlated.
Data Transformation
In this preprocessing step, the data are transformed or consolidated so that
the resulting mining process may be more efficient, and the patterns found
may be easier to understand. Data discretization, is a form of data
transformation. In data transformation, the data are transformed or
consolidated into forms appropriate for mining. Strategies for data
transformation include the following:
Data Transformation
1. Smoothing, which works to remove noise from the data. Techniques include binning,
regression, and clustering.
2. Attribute construction (or feature construction), where new attributes are
constructed and added from the given set of attributes to help the mining process.
3. Aggregation, where summary or aggregation operations are applied to the data. For
example, the daily sales data may be aggregated so as to compute monthly and annual
total amounts. This step is typically used in constructing a data cube for data analysis
at multiple abstraction levels.
4. Normalization, where the attribute data are scaled so as to fall within a smaller
range, such as −1.0 to 1.0, or 0.0 to 1.0.
Data Transformation
5. Discretization, where the raw values of a numeric attribute (e.g., age)
are replaced by interval labels (e.g., 0–10, 11–20, etc.) or conceptual
labels (e.g., youth, adult, senior). The labels, in turn, can be recursively
organized into higher-level concepts, resulting in a concept hierarchy for
the numeric attribute. Figure 3.12 shows a concept hierarchy for the
attribute price. More than one concept hierarchy can be defined for the
same attribute to accommodate the needs of various users.
6. Concept hierarchy generation for nominal data, where attributes such
as street can be generalized to higher-level concepts, like city or country.
Many hierarchies for nominal attributes are implicit within the database
schema and can be automatically defined at the schema definition level.
Smoothing Data Transformation
Binning…
Refer module 3 part 1.
Attribute Subset selection Data Transformation
Attribute construction Data Transformation
Creation of new attribute…..
Aggregation Data Transformation
From quarterly to yearly……
Data Transformation
Data Transformation
Normalization
To give all attributes an equal weight normalization is useful. Normalization
is recommended for classification and clustering.
Min-max normalization
z-score normalization
Decimal scaling
Examples related to Normalization
Data Transformation
Data Transformation
Advantages or disadvantages of
Normalization
Advantages:
• Improved performance of machine learning algorithms: Normalization can help to
improve the performance of machine learning algorithms by scaling the input
features to a common scale. This can help to reduce the impact of outliers and
improve the accuracy of the model.
• Better handling of outliers: Normalization can help to reduce the impact of outliers
by scaling the data to a common scale, which can make the outliers less influential.
• Improved interpretability of results: Normalization can make it easier to interpret
the results of a machine learning model, as the inputs will be on a common scale.
• Better generalization: Normalization can help to improve the generalization of a
model, by reducing the impact of outliers and by making the model less sensitive to
the scale of the inputs.
Normalization Data Transformation
Disadvantages:
• Loss of information: Normalization can result in a loss of information if the original
scale of the input features is important.
• Impact on outliers: Normalization can make it harder to detect outliers as they will be
scaled along with the rest of the data.
• Impact on interpretability: Normalization can make it harder to interpret the results of
a machine learning model, as the inputs will be on a common scale, which may not align
with the original scale of the data.
• Additional computational costs: Normalization can add additional computational costs
to the data mining process, as it requires additional processing time to scale the data.
In conclusion, data normalization can have both advantages and disadvantages. It can
improve the performance of machine learning algorithms and make it easier to interpret
the results. However, it can also result in a loss of information and make it harder to
detect outliers. It’s important to weigh the pros and cons of data normalization and
carefully assess the risks and benefits before implementing it.
Dimensionality reduction is the process of reducing the
number of random variables or attributes under consideration.
Dimensionality reduction methods include wavelet transforms and
principal components analysis,which transform or project the original
data onto a smaller space. Attribute subset selection is a method of
dimensionality reduction in which irrelevant, weakly relevant, or
redundant attributes or dimensions are detected and removed (Section
3.4.4).
Numerosity reduction techniques replace the
original data volume with alternative, smaller forms of data
representation. These techniques may be parametric or
nonparametric. For parametric methods, a model is used to estimate
the data, so that typically only the data parameters need to be stored,
instead of the actual data. Regression and log-linear models are
examples. Nonparametric methods for storing reduced representations
of the data include histograms, clustering, sampling, and data cube
aggregation.
In data compression, transformations are applied so as to
obtain a reduced or “compressed” representation of the original data.
If the original data can be reconstructed from the compressed data
without any information loss, the data reduction is called lossless. If,
instead, we can reconstruct only an approximation of the original data,
then the data reduction is called lossy. There are several lossless
algorithms for string compression; however, they typically allow only
limited data manipulation. Dimensionality reduction and numerosity
reduction techniques can also be considered forms of data
compression