BIG DATA ANALYTICS

Part-[2]: Data Pre-processing

Dr. Sumit Kumar Tetarave

School of Computer Applications
KIIT Deemed to be University
Bhubaneswar
 Data Science Project

Measures of Dispersion (or Variability) i) Descriptive Analysis

1. Absolute Deviation from Mean ii) Data Pre-processing
2. Variance iii) Predictive Analysis
3. SD
4. Range
5. Quartiles
6. Skewness

Next:
Data Preprocessing
❖ Data Cleaning accounting for: ❖ Some approaches to integrate data: ❖ During data migration, there are certain issues to be considered
❖ Missing values ❖ Data consolidation namely:
❖ Use a global constant to fill in the ❖ Data virtualization I. Schema Integration and Object Matching
missing values ❖ Data propagation II. Redundancy
❖ Use the attribute mean to fill in the
III. Detection and Resolution of data value conflicts.
missing value
❖ Use the mean attribute for all samples
belonging to the same class as the given ❖ Techniques used for data transformation:
tuple ❖ Smoothing, Aggregation,
❖ Using the most likely meaning to fill in ❖ Normalization
the missing value ❖ Min-max normalization
❖ Noisy Data ❖ Z-score normalization
❖ An outlier can be treated as noise ❖ Normalization by decimal scaling
❖ Correcting inconsistent data points, and ❖ Discretization, Generalization,
Smoothing noisy data. ❖ Feature Construction and Concept hierarchy generation
❖ For numeric values, we can use
a scatter plot or box plot to ❖ 4. Reduction of numbers when alternative, smaller data
identify outliers representation replaces or predicts information.
❖ Methods for solving it ❖ There are two ways to minimize numbers
❖ Binning ❖ i. Parametric Methods
❖ Regression ❖ ii. Non-Parametric Methods
❖ Clustering ❖ 5. Discretization
❖ Following are the categories of Discretization:
❖ i. Supervised Discretization
❖ Following are some of the strategies for Data Reduction: ❖ 3. Dimensionality reduction where encoding ❖ ii. Unsupervised Discretization or Splitting
❖ 1. Data cube aggregation, which applies aggregation of data approaches are used to reduce the data collection size to ❖ iii. Top-Down Discretization or Splitting
during data cube construction. a minimum. ❖ iv. Bottom – up Discretization or merging
❖ 2. Set of sub-set attributes, which may be detected and ❖ It can be divided into ✓6. Category hierarchy generation where raw data values for the
removed by obsolete, weakly significant or redundant ❖ i. feature selection and attributes are replaced by ranges or higher logical levels.
measurements. ❖ ii. feature extraction ✓Five methods for concept hierarchy generation
❖ Attribute Subset Selection Methods- ✓Dimensionality Reduction Methods ✓i. Binning
✓1. Stepwise Forward Selection ✓Principal component analysis (PCA) ✓ii. Histogram Analysis
✓ii. Stepwise Backward Elimination ✓High correlation filter ✓iii. Entropy – Based Discretization
✓iii. Combination of Forward Selection and ✓Missing values ratio ✓iv. Interval merging by 2 – Analysis
Backward Elimination ✓Low variance filter ✓v. Cluster Analysis
✓iv. Decision Tree Introduction ✓Random forest ✓vi. Discretization by Intuitive Partitioning
Purpose of Data Transformation
• Data transformation is the process of converting raw data into a format or
structure that would be more suitable for model building and also data discover
in general.
• The process of data transformation can also be referred as
extract/transform/load (ETL).
• The extract phase involves identifying and pulling data from the various
source systems that create data and then moving data to a single repository.
• Next, the raw data is cleaned, if needed.
• It is then transformed into a target format that can be fed into operational
system or into a data warehouse or another repository
• to use in business intelligence and analytics applications.
• The data are transformed in ways that are ideal for analyzing the data.
Steps of Data Transformation
• The data transformation involves steps that are:
1. Smoothing
2. Aggregation
3. Generalization
4. Discretization
5. Normalization
o Min-max normalization
o Z-score normalization
o Normalization by decimal scaling
6. Attribute and feature construction
o New attributes constructed from the given ones.
Steps of Data Transformation…
Steps of Data Transformation…
• Discretization: It is the process of transforming continuous variables
into discrete variables by creating a set of contiguous intervals, also
called bins,
• that span the range of the variable values.
• Discretization is used to change the distribution of skewed variables
and to minimize the influence of outliers, and
• hence improve the performance of some machine learning models.
• Example of 4 bins-
Age: 1,5,9,4,7,11,14,17,13,18,19,31,33,36,42,44,46,70,74,78,77,

✓Bin1- 1,5,9,4,7
✓Bin2- 11,14,17,13,18,19
✓Bin3- 31,33,36,42,44,46
✓Bin4- 70,74,78,77
Binning
• Binning method smooth a sorted data value by consulting its
“neighborhood”, that is, the values around it.
• the data is first sorted and then the sorted values are distributed into
a number of buckets or bins.
• Binning methods consult the neighborhood of values; they perform
local smoothing.
• There are basically two types of binning approaches –
1. Equal width (or distance) binning :
2. Equal depth (or frequency) binning :
 1. Equal width (or distance) binning :
• The simplest binning approach is to partition the range of the variable into k
equal-width intervals.
• The interval width is simply the range [A, B] of the variable divided by k,
w = (B-A) / k
• Thus, 𝑖 𝑡ℎ interval range will be ((A + (i-1)*w), (A + i*w)] where i = 1, 2, 3…..k
• Skewed data cannot be handled well by this method.

• For example, if the values of the variable vary between 0 and 100, we can
create five bins like this: width = (100 - 0) / 5 = 20;
• the 5 bins will be (0 + (1-1)*20, (0 + 1*20)] = (0-20],
(0 + (2-1)*20, (0 + 2*20)] = (20-40],
(0 + (3-1)*20, (0 + 3*20)] = (40-60],
(0 + (4-1)*20, (0 + 4*20)] = (60-80]
(0 + (5-1)*20, (0 + 5*20)] = (80-100]
2. Equal depth (or frequency) binning :

• In equal-frequency binning we divide

the range [A, B] of the variable into
intervals that contain (approximately)
equal number of points;
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐷𝑎𝑡𝑎
𝐷𝑎𝑡𝑎 𝑖𝑛 𝑎 𝐵𝑖𝑛. =
𝐵𝑖𝑛𝑠

• equal frequency may not be possible due

to repeated values.
Smoothing:
There are three approaches to perform smoothing –

1. Smoothing by bin-means : In smoothing by bin means, each value

in a bin is replaced by the mean value of the bin.
2. Smoothing by bin-median : In this method each bin value is
replaced by its bin median value.
3. Smoothing by bin-boundary: In smoothing by bin boundaries, the
minimum and maximum values in a given bin are identified as the
bin boundaries.
• Each bin value is then replaced by the closest boundary value.
 Sorted data for price(in dollar) :
2, 6, 7, 9, 13, 20, 21, 24, 30
Partition using equal frequency approach in 3-bin:
• Bin 1 : 2, 6, 7
• Bin 2 : 9, 13, 20
• Bin 3 : 21, 24, 30

Smoothing by bin mean : Smoothing by bin median : Smoothing by bin boundary :

Bin 1 : 5, 5, 5 Bin 1 : 6, 6, 6 Bin 1 : 2, 7, 7
Bin 2 : 14, 14, 14 Bin 2 : 13, 13, 13 Bin 2 : 9, 9, 20
Bin 3 : 25, 25, 25 Bin 3 : 24, 24, 24 Bin 3 : 21, 21, 30
 Steps of Data Transformation…
(Normalization)

i)
ii)

iii)
Min – Max Normalization
Min – Max Normalization: Example
• Assume the attribute revenue minimum and maximum values are
1000 and 16000 respectively. Diagram income in the [0.0, 1.0] range.
For salaries a value of 14000 is converted into a minimum – maximum
standardization.
Z-Score Normalization
Z-Score Normalization: Example
• Suppose that the average and standard deviation in attribute revenue
number are respectively 22,000 and 7,000. In the case of Z-Score
standardization, the revenue of 42,000 is transferred to Z-Score.

42000 − 22000
= 2.85
7000
Decimal Scaling Normalization
• Standardizes the variable values by changing their decimal position.
• We can measure the number of points that the decimal point is passed by the absolute maximum value of the
X attribute.
• The value of v of the X attribute is computer-standardized as:
𝑣
𝑣′ = 𝑗
10

J is such an integer that maximum.

Example 1:

Here maximum value of CGPA is 3 so we can convert it to a decimal by dividing by 10. Why 10?
• we will count total numbers in our maximum value and then put 1 and after 1 we can put zeros equal to
the length of the maximum value.

Example 2:
Practice Set-2
a) Discretize the values of systolic Blood Pressure of the given dataset into two bins based on
their neighborhood, i.e., Equal width (or distance) and Equal depth (or frequency) binning.
b) Smooth the above Equal depth (or frequency) binning data using bin-mean, bin-median and
bin-boundary.
c) Transform Systolic Blood Pressure values of the given dataset using Min-Max, Z-Score, and
Decimal Scaling Normalization methods