Data Pre-processing

2
Descriptive Data
Summarization
Provides an initial understanding of the data and helps identify
inconsistencies or patterns.
• Central Tendency: Mean, median, mode
• Dispersion: Range, variance, standard deviation
• Distribution: Skewness
• Visualization: Histograms, box plots, scatter plots
• Correlation analysis: Measures linear relationships between
attributes (e.g., Pearson’s correlation)
 Purpose: Identify trends, anomalies, and relationships before
deeper processing.
 Data Pre-processing
 Data preprocessing is a data mining technique which is used to transform the raw data in a useful and efficient
format.
 Steps Involved in Data Preprocessing:
1. Data Cleaning:
The data can have many irrelevant and missing parts. To handle this part, data cleaning is done. It involves
handling of missing data, noisy data etc.
(a) Missing Data:
This situation arises when some data is missing in the data. It can be handled in various ways.
Some of them are:
1. Ignore the tuples:
This approach is suitable only when the dataset we have is quite large and multiple values are missing within
a tuple.
2. Fill the Missing values:
There are various ways to do this task.
1. fill the missing values manually
2. Fill it automatically
1. A global constant Unknown a new class
2. by central tendency attribute mean or median
3. Use the most probable value.
 Data Pre-processing
(b). Noisy Data:
Noisy data is a meaningless data that can’t be interpreted by machines. It can be generated due
to faulty data collection, data entry errors etc. It can be handled in following ways :

• Binning: Binning methods smooth a sorted data value by consulting its “neighborhood,”that is,
the values around it. The sorted values are distributed into a number of “buckets,” or bins.
Because binning methods consult the neighborhood of values, they perform local smoothing.

• Regression: Data smoothing can also be done by regression, a technique that conforms data
values to a function.

• Outlier analysis: Outliers may be detected by clustering, for example, where similar values are
organized into groups, or “clusters.” Outlier Detection: Statistical methods (e.g., Z-score)
(c) Inconsistency Correction:
• Standardizing formats (e.g., date formats)
• Resolving duplicates
Measure of Dispersion
Measure of Dispersion
Dispersion refers to the extent to which data values vary or spread out around a central
value (like the mean or median). It helps us understand the variability in the dataset.

Measure Description Formula / Notes

Difference between the
Range maximum and minimum Range = Max − Min
values
Spread of the middle 50% of IQR = Q3 − Q1 (3rd quartile −
Interquartile Range (IQR)
the data 1st quartile)
Average squared deviation
Variance
from the mean
Square root of variance;
Standard Deviation shows average deviation from
the mean

Ratio of standard deviation to

Coefficient of Variation
the mean; used to compare CV = (σ / μ) × 100%
(CV)
variability across datasets
 Z-Score
 The Z-score tells you how many standard deviations a data point is from the mean
of the dataset.
Z = (X − μ) / σ
The formula for Standard Deviation (σ) is:
For a Population:
Where :σ = population standard deviation
N = number of data points
𝑥𝑖 = each individual data point
μ = population mean
For a Sample:
Where:𝑠 = sample standard deviation
n = number of sample data points
𝑥𝑖​ = each individual data point
xˉ = sample mean
Dataset:
Mango yield per tree (in kg):
40, 42, 39, 45, 41, 300, 43, 44
calculate Z-scores for a given dataset and identify if any data points are outliers
 Data Pre-processing
2. Data integration: Combine data from different sources.
•Combines data from multiple sources into a coherent store
•Entity identification problem:
Identify real world entities from multiple data sources,
e.g., Bill Clinton = William Clinton
•Detecting and resolving data value conflicts
For the same real world entity, attribute values from different sources are different
•Possible reasons: different representations, different scales,
e.g., rupee vs. British units

Handling Redundancy in data

Redundant data occur often when integration of multiple databases
•Object identification: The same attribute or object may have different names in different databases
•Derivable data: One attribute may be a “derived” attribute in another table, e.g., annual revenue
•Redundant attributes: may be able to be detected by correlation analysis and covariance analysis
•Careful integration of the data from multiple sources may help reduce/avoid redundancies and inconsistencies and
improve mining speed and quality
 Data Pre-processing
4. Data Transformation: Format data appropriately for analysis.
 Convert "Gender" to numeric (e.g., Male = 0, Female = 1)
 Normalize the dataset to make the data comparable, efficiently processed,
and accurately analyzed
 Why normalization is important:
[Link] Bring All Features to a Common Scale
 In real datasets, attributes can have different ranges.
 Example:
 Age: 18–70
 Income: ₹10,000–₹1,00,00,000
 Algorithms like k-Means, k-NN, neural networks use distance-based calculations, so large-scale attributes
dominate.
[Link] Improve Accuracy of Machine Learning Algorithms
 Algorithms like gradient descent converge faster when data is normalized.
 Prevents bias towards high-range features.
3. To Speed Up Computation
4. To Prepare Data for Dimensionality Reduction (e.g., PCA)(to give equal weightage)
5. To Improve Clustering & Classification Results
Data Pre-processing
Data Transformation
Common Normalization Techniques
 Min Max
Customer Age (Years) Annual Income (₹)
A 20 40,000
B 40 80,000
C 25 35,000
 Normalize Age:

Custome
Min = 20 Age Normalized Age
r
 Max = 40 A 20 (20−20)/(40−20) = 0.0
B 40 (40−20)/(40−20) = 1.0
C 25 (25−20)/(40−20) = 0.25
Customer Income Normalized Income
A 40,000 (40000−35000)/(80000−35000) = 0.11
B 80,000 (80000−35000)/(80000−35000) = 1.0
C 35,000 (35000−35000)/(80000−35000) = 0.0
Min Max
Customer Normalized Age Normalized Income
A 0.0 0.11
B 1.0 1.0
C 0.25 0.0

Why Normalize?
 Now, both features are on the same scale (0 to 1):
 No single feature dominates distance calculations.
 Clustering and classification algorithms work better.
 You get more balanced and meaningful results.
Z-score Normalization

Dataset Value Mean (x̄) Sample SD (s) Z-score

Set 1 20 28.33 10.41 -0.80

40 28.33 10.41 1.12

25 28.33 10.41 -0.32

Set 2 40000 51666.67 24664.98 -0.47

80000 51666.67 24664.98 1.15

35000 51666.67 24664.98 -0.68

Z-score Normalization

 Age: 22 + 35 + 29 + 40 + 31
 Mean μ= (22 + 35 + 29 + 40 + 31)/5 = 31.4
 SD σ ​≈6.72​
Which Normalization to Use and When?

Min-Max Z-score
Criteria
Normalization Normalization
When features have
When features have
known fixed bounds or
When to Use varying distributions or
you want all features
outliers.
between 0 and 1.
[0, 1] or any custom Mean = 0, SD = 1 (no
Output Range
range fixed min/max)
✅ Yes — outliers can ❌ No — it handles
Sensitive to Outliers?
compress other values outliers better
k-Means, k-NN, Neural PCA, Logistic
Used in Algorithms Networks, Deep Regression, SVM, Outlier
Learning Detection
When data is evenly When data has extreme
Best for distributed without values or unknown
extreme outliers range
Dataset

 Cust ID | Age | Monthly Income (₹) | Purchases per Month

------------------------------------------------------------
C001 | 22 | 25000 |5
C002 | 35 | 60000 | 15
C003 | 29 | 45000 | 10
C004 | 40 | 80000 | 12
C005 | 31 | 55000 |9
 Correlation Analysis (Nominal data)

 The Chi-square test of independence tests if there is a significant relationship between two
categorical variables. The test is comparing the observed observations to the expected observations.
The data is usually displayed in a cross-tabulation format with each row representing a category for
one variable and each column representing a category for another variable.

 The H0 (Null Hypothesis): There is no relationship between variable one and variable two.
 The H1 (Alternative Hypothesis): There is a relationship between variable 1 and variable 2.
If the p-value is significant, you can reject the null hypothesis and claim that the findings support the
alternative hypothesis.
(chi-square) test

• The larger the value, the more likely the variables are related
• The cells that contribute the most to the value are those whose actual
count is very different from the expected count
 (chi-square) test

•
(chi-square) test
(chi-square) test
 Correlation Analysis
• In correlation analysis, we estimate a sample correlation coefficient, more specifically the Pearson Product Moment
correlation coefficient.
• The sample correlation coefficient, denoted r,
• ranges between -1 and +1 and quantifies the direction and strength of the linear association between the two variables.
• A “0” means there is no relationship between the variables at all, while -1 or 1 means that there is a perfect negative or
positive correlation.
• The magnitude of the correlation coefficient indicates the strength of the association.
 Pearson’s Correlation
Pearson’s Correlation Coefficient formula is as follows,

Where,
•r = Pearson Coefficient
•n= number of tuples
•∑xy = sum of products
•∑x = sum of the x scores
•∑y= sum of the y scores
•∑x2 = sum of the squared x scores
•∑y2 = sum of the squared y scores

Subject Age X Glucose level Y X*y X*x Y*y

1 43 99
2 21 65
3 25 79
4 42 75
5 57 87
6 59 81
 Pearson’s Correlation
GE X GLUCOS XY X2 Y2
E LEVEL
Y
1 43 99 4257 1849 9801
2 21 65 1365 441 4225
3 25 79 1975 625 6241
4 42 75 3150 1764 5625
5 57 87 4959 3249 7569
6 59 81 4779 3481 6561
Σ 247 486 20485 11409 40022

From our table:

•Σx = 247
•Σy = 486
•Σxy = 20,485
•Σx = 11,409
2

•Σy = 40,022
2

•n= 6
The correlation coefficient =
•6(20,485) – (247 × 486) / [√[[6(11,409) – (247 )] × [6(40,022) – 486 ]]]
2 2

= 0.5298
 Data Reduction
 Data reduction: Obtain a reduced representation of the data set that is much
smaller in volume but yet produces the same (or almost the same) analytical
results
 Why data reduction? — A database/data warehouse may store terabytes of
data. Complex data analysis may take a very long time to run on the
complete data set.

 Dimensionality reduction Strategies include

 Dimensionalality reduction
 Numerosity reduction
 Data compression

Dimensionality reduction
 Avoid the curse of dimensionality
 Help eliminate irrelevant features and reduce noise
 Reduce time and space required in data mining
 Allow easier visualization
 Attribute Subset Selection
 Another way to reduce dimensionality of data
 Redundant attributes
 Duplicate much or all of the information
contained in one or more other attributes
 E.g., purchase price of a product and the
amount of sales tax paid
 Irrelevant attributes
 Contain no information that is useful for the
data mining task at hand
 E.g., students' ID is often irrelevant to the task
of predicting students' GPA

26
 Method of Attribute Selection

[Link] forward selection: The procedure starts with an empty set of attributes
as
the reduced set. The best of the original attributes is determined and added to the
reduced set. At each subsequent iteration or step, the best of the remaining
original
attributes is added to the set.

2. Stepwise backward elimination: The procedure starts with the full set of
attributes.
At each step, it removes the worst attribute remaining in the set.

4. Decision tree induction: Decision tree algorithms (e.g., ID3, C4.5, and CART)
were
originally intended for classification. Decision tree induction constructs a
flowchart-
like structure where each internal (nonleaf) node denotes a test on an attribute,
each
branch corresponds to an outcome of the test, and each external (leaf) node 27
 Concept Hierarchy Generation
for Nominal Data
 Specification of a partial/total ordering of attributes explicitly at the schema
level by users or experts
 street < city < state < country
 Specification of a hierarchy for a set of values by explicit data grouping
 {Urbana, Champaign, Chicago} < Illinois
 Specification of only a partial set of attributes
 E.g., only street < city, not others
 Automatic generation of hierarchies (or attribute levels) by the analysis of
the number of distinct values
 E.g., for a set of attributes: {street, city, state, country}

28
 Automatic Concept Hierarchy Generation
 Some hierarchies can be automatically generated based on
the analysis of the number of distinct values per attribute in
the data set
 The attribute with the most distinct values is placed at

the lowest level of the hierarchy

 Exceptions, e.g., weekday, month, quarter, year

country 15 distinct values

province_or_ state 365 distinct values

city 3567 distinct values

street 674,339 distinct values

29