Unit: 2 Data Preprocessing

1.1. Data Objects and Attribute Types

1.2. Basic Statistical Descriptions of Data
1.3. Data Preprocessing
1.4. Data Cleaning
1.5. Data Integration
1.6. Data Reduction
1.7. Data Transformation and Data Discretization
1.8. Measures of Similarity and Dissimilarity

Data Objects and Attribute Types

A data object represents an entity. Data sets are made up of data objects. In the sales
database, the objects may be customers, store items and sales in a university
database, the objects Maybe students, professors and courses. Data typically described
by attributes. If the data objects are stored in a database, they are data tuples. That is,
the rows of a database correspond to the data object, and the columns correspond to
attributes.

An attribute is a data field representing a characteristic or feature of a data object. For

example. A customer object can include ID, name and address.
Types of attributes:

1. Nominal Attributes

The values of nominal attributes are symbols or names of things. Each value represents
Some kind of category, code or state without any order. The values do not have any
meaningful order. They can be represented with numbers. Example Hair color possible
values are black, brown, gray, white?
2. Binary Attributes
A binary attribute is a nominal attribute with only two categories or state: 0 or 1,
where 0 typically means that the attribute is absent, and 1 means that it is
present. Binary attributes are referred to as Boolean if the two states correspond
to true and false.
Example: yes/ No, True/ False, binary attributes given the attribute smoker
describing a patient object, 1 indicate that the patient smokes while 0 indicates
that the patient does not. There are two types of binary attributes.
i. Symmetric: If binary attribute is symmetric if both of its states are equally
valuable and carry the same weight. example (male/female).
 ii. Asymmetric: A binary attributes asymmetric if the outcomes of the states
not equally important. Such as the positive and negative outcomes of a
medical test for HIV.
3. Numeric Attribute:
A numeric attribute is quantitative, meaning it represents measurable values, either as
integers or real numbers. It’s types

1. Interval-Scaled Attribute
An interval-scaled attribute is measured on a scale where the difference between values is
meaningful, and the units are equal in size.
 The values can be positive, zero, or negative.
 The values have a natural order (they can be ranked).
Example:
Suppose we record temperatures for several days. We can order the days from coldest to
warmest, and the differences (e.g., 5°C warmer) are meaningful.

2. Ratio-Scaled Attributes:
A ratio-scaled attribute is one where the values are numerical and have a true zero
point, meaning that ratios between values are meaningful. For example, height,
weight, and income are ratio-scaled attributes (Height: 0 cm means no height).

Statistical Description of Data: To preprocess data effectively, it's important to first

understand the overall structure and characteristics of the data. Basic statistics help us
spot important patterns and identify any unusual or incorrect values (called noise or
outliers).

Two key types of statistics are used:

1. Measures of Central Tendency – These tell us the average or typical value in
the data. Common examples include:

 Mean (Average)
Example: For data 4, 5, 6, 7, 8
Mean = (4+5+6+7+8) / 5 = 6
 Median (Middle value)
Example: For data 3, 5, 7
Median = 5 (middle value)
For data 3, 5, 7, 9 (even count)
Median = (5+7)/2 = 6
 Mode (Most frequent value)
Example: For data 2, 3, 3, 4, 5
Mode = 3 (it appears most often)
 2. Measures of Dispersion – These shows how spread out the data values are.
Common examples include:

 Range
Example: For data 10, 15, 20
Range = 20 - 10 = 10
Variance
Example: For data 2, 4, 6
Mean = 4
Variance = [(2−4)² + (4−4)² + (6−4)²]/3 = [4 + 0 + 4]/3 = 2.67
Standard Deviation
Standard deviation = √Variance
√2.67 ≈ 1.63

Using these statistics helps us find outliers and better prepare the data for further
analysis or modeling.

Data Preprocessing
Raw data collection from the environment can be dirty. Data is said to be dirty if it is incomplete,
noisy or inconsistent. Quality of data affects quality of data mining results. To improve quality of
data and quality of data mining results, raw data need to be expressed. Data preprocessing is one
of the critical steps of data mining. Data preprocessing methods can be divided into the following
categories.

i. Data cleaning
ii. Data integration
iii. Data transformation
iv. Data reduction
v. Dat discretization

Data cleaning

 Real-world data tends to be incomplete, noisy, and inconsistent.

 Data cleaning attempts to fill in missing values, smooth out noise, and correct out
inconsistencies in data.

Ways for handling missing values:

1. Ignore the tuples:

Sometimes, data has missing values. One simple way to handle this is to ignore or remove the
entire row (tuple) that has missing data. This method is not very effective, unless tuple contain
several attributes with missing values.
 2. Fill the missing values manually: This approach is time-consuming and may not be
feasible given a large data set with missing values.
3. Use global constant:
We can fill in missing values using a fixed word or value, called a global constant.
 Common constants are "unknown", "N/A", or 0.
 This is a simple method but may reduce data accuracy if used too much.
4. Using Mean:
Sometime average value or mean value can be used in filling missing values.

Noisy Data:
Common methods for removing noise from data are as follows

i. Regression: Data regression consists of fitting the data to a function. A linear

regression for instance, finds the line to fit two variables so that on variable
can predict the other.
ii. Clustering: Outliers may be detected by clustering, where similar values are
organized into groups of clusters. The value which falls outside the cluster
may be considered as outlier i.e. noisy here and discarded.
iii. Binning: Binning technique smooths a sorted data value by considering a
neighborhood or value around it. The sorted values are distributed into a
number of buckets or bins. Since binning techniques consult neighborhood
values, they perform local smoothing.

Example: consider the following sorted data for price

4, 8, 15, 21, 21, 24, 25, 28, 34

Now dividing partitioning these data into Equi-depth bins of depth 3 i.e. each bin contains 3
values

i.e. bin1 = 4, 8, 15

bin2 = 21, 21,24

bin3 = 25, 28, 34

Technique 1:
In smoothing by bin means, each value in a bin is replaced by the Mean of values of the bin

i.e. bin1= 9,9,9

bin2=22, 22, 22
 bin3 29, 29, 29

Technique 2:
In smoothing by bin median, each value in a bin is replaced by the median of values of the Bin

i.e. bin1= 8, 8, 8

bin2 = 21, 21, 21

bin3 = 28, 28, 28

Technique 3:
In smoothing by bin boundary, each value in a bin is replaced by the closest boundary value.
Here first we find the minimum and maximum value in a bin as bin boundaries, then each bin
value is replaced by the closest boundary value.

i.e. bin1 = 4, 4, 15

bin2 = 21, 21, 24

bin3= 25, 25, 34

Inconsistent data

 Data inconstancies may be corrected manually using external references. For example,
Error made at data entry may be corrected by performing paper trace.
 Also, some tools can be used to detect the violation of data constraints.

2. Data integration and transformation

 Data integration merges data from multiple data sources.

 These sources may include multiple databases, data cubes or flat files.
 There are number of issues to be considered during data integration such as

i. Schema integration and object matching

When data comes from multiple sources (e.g., two different databases), they often
have different ways of naming and storing the same thing.
Problem: How do we know that customer_id in Database A and customer_number in
Database B refer to the same customer? This is called the entity identification
problem.

ii. Redundancy
  Inconsistency in attribute or dimension naming can also cause redundancy in the resulting
data set.
 There can also be two identical tuples for a given unique data entry case.
Example:
 In one table, you have "CustomerID" as a column name.
 In another table, the same data is stored under "Cust_ID".
 Because of this naming inconsistency, a system might treat them as different attributes,
even though they represent the same information, causing duplicate storage (redundancy).

iii. Data value conflict

 For the same real-world entity, attribute values from different sources may differ.
 Example: the price of different hotels may involve different currency.

Careful integration of the data from multiple sources can help reduce and avoid redundancy and
inconsistency in the resulting data set. This can help improve the accuracy and speed of the
subsequent data mining task.

3. Data Transformation

 In data transformation, we change or combine data into a better format for data
mining (analysis).
 This helps make the data easier to understand and useful for finding patterns.

Data transformation can involve the following:

i. Smoothing: work to remove noise from data. Such technique includes clustering,
binning etc.
ii. Aggregation: when summary operations are applied to the data. For example, the
daily sales can be aggregated to compute monthly total sales.
iii. Generalization: Here low-level data are replaced by the higher-level concepts
through the use of concept hierarchies. For example: attributes like city, can be
generalized to higher level concept like country. Similarly, age mapped to young,
middle or aged.
iv. Normalization: where the attribute data are scaled so as to fall within a small
specified range such as 0.0 to 1.0. Normalization is mostly useful for classification
algorithm involving neural network. If using the neural network backpropagation
algorithm for classification mining, normalizing the inputs values for each attribute
measured in the training tuple will help speed up the learning phase. Types of
normalization is:
 Min-max normalization:
Where y is the value to be normalized, maxnew and minnew are the new maximum and minimum
range and maxold and minold are the old maximum and minimum range.

Example:

Suppose that the minimum and maximum values for the attribute salary are 12000 and 98000
respectively. Normalize the salary 73600 in the range 0.0 to 1.0 using min max normalization.

Solution,

X= ( (73600-12000) (1-0)+0)/ (98000-12000)

= 0.716

4. Data reduction
 If the size of the data set is too large, then the task of data mining can take longer time.
 Data reduction achieves a reduced version of data set that is smaller in volume but
generate the same or almost same analytical result.

Strategies for data reduction include the following:

i. Data cube aggregation: Data cube aggregation in which operation are applied to the
data in construction of a data cube. Data cube store multidimensional aggregated
information. Concept hierarchy exists for each attribute. So, we can reduce the data at
the concept level needed in the analysis.
ii. Data compression: where encoding mechanisms are used to reduce the data set size.
This save the storage size and reduces communication time.
iii. Dimension Reduction: Data set for analysis may contain hundreds of attributes,
many of which may be irrelevant to the mining task. Dimension reduction reduces
data set size by removing such irrelevant attributes or dimensions. One of the
techniques for dimension reduction is decision tree induction.

Data Discretization:
Discretization reviews the number of values for a given continuous attribute by
dividing the range of the attribute into intervals. Intervals labels can then be used to
replace actual data values.
Examples: suppose we have an attribute of age with the given values:
Age: 1,4,5,7,9,11,14,17,18,19,21,31,33,36,42,44,46,70,74,77,78
After Discretization:
Attributes Age Age Age Age
1,4,5,7,9 11,14,17,18,19 31,33,36,42,46 70,74,77,78
After Child Young Mature old
Discretization

Data Discretization Techniques:

1. Discretization by Histogram Analysis:
Discretization by Histogram Analysis is a data preprocessing technique used to
convert continuous (numerical) attributes into discrete (categorical) ones by analyzing
the distribution of data through histograms.
2. Discretization by Decision Tree:
This method uses a top-down approach to split continuous data into intervals. It is a
supervised technique, meaning it uses the class label (e.g., diagnosis) to decide how
to split the data.
Example:
Suppose we have patient data with symptoms like temperature or blood pressure, and
each patient has a diagnosis (e.g., flu or no flu). A decision tree can find the best
points to split the symptom values (like temperature ≤ 100°F) to help predict the
diagnosis. These split points are used to create discrete intervals. Patients with
temperature ≤ 100°F usually have No flu. Patients with temperature > 100°F usually
have Flu.
3. Cluster Analysis:
Cluster analysis is a way to group similar data values together. It can be used for
discretization by dividing values of a variable (like X) into clusters based on
similarity. These clusters can then be treated as discrete groups for further
analysis.
Measuring data similarity and dissimilarity.
similarity:
 Numerical measure of how alike two data objects are.
 The value is higher when objects are more alike.
 often falls in the range [0,1]

Dissimilarity:
Numerical measure of how different two data objects are:
 lower when objects are more alike.
 Minimum dissimilarity is often 0.
 Upper limit depends on the data and the dissimilarity measure used (e.g.,
Euclidean distance, Manhattan distance).

Dissimilarity matrix

 A dissimilarity matrix is used to measure how different objects are from

one another in a dataset.
 Form: nxn matrix.
Representation: stores dissimilarities d(i,j) between objects i and j.
properties:
 d(i, i)=0
 d(i, j)= d( j,i) (i.e element below diagonal=element above diagonal)