Data Preprocessing

Module 2
What is Data Preprocessing?
• Data preprocessing refers to cleaning, transforming and integrating
data in order to make it ready for data analysis
• The goal of data pre-processing is to improve the quality of the data
to make it more suitable to the specific data mining task
• Some common steps in data pre-processing include:
• Data cleaning
• Data integration
• Data transformation
• Data reduction
• Data discretization
• Data cleaning: Data cleaning routines work to “clean” the data by
filling in missing values, smoothing noisy data, identifying or removing
outliers, and resolving inconsistencies
• Data integration: This step involves combining data from multiple
sources, such as databases, spreadsheets, text files. The goal of the
integration is to create a single, consistent view of the data.
• Data transformation: Converting the data into a format that is more
suitable for the data mining task. This can include normalizing
numerical data, and encoding categorical data.
• Data reduction: selecting a subset of the data that is relevant to the
data mining task. It can be feature selection (selecting a subset of the
variables) or feature extraction (extracting new features ).
• Data discretization: Convert continuous numerical data into
categorical data.
 Data Cleaning
It involves handling of missing data, noisy data etc.
1. Missing Data
This situation arises when some data is missing in the data. It can be handled
in various ways. Some of them are:
Ignore the tuples: This approach is suitable only when the dataset we have is
quite large and multiple values are missing within a tuple
Fill the missing values: There are various ways to do this task. You can
choose to
- fill the missing values manually,
- by attribute mean or
- the most probable value.
- Use a global constant to fill in the missing value
 Data Cleaning
• Noisy data
Noise is a random error or variance in a measured variable. Noisy data is a
meaningless data that can’t be interpreted by machines. It can be generated due to
faulty data collection, data entry errors etc. It can be handled in following data
smoothing techniques:
Binning : 1)Data is sorted,
2)Data are distributed into a number of buckets or bins.
Smoothing by bin means: each value in a bin is replaced by the mean value of the
bin.
Smoothing by bin medians: each bin value is replaced by the bin median.
Smoothing by bin boundaries: the minimum and maximum values in a given bin
are identified as the bin boundaries. Each bin value is then replaced by the closest
boundary value.
Data Cleaning
Noisy Data
• Regression:
The data is made smooth with the help of using regression function.
Regression can be linear or multiple. Linear regression has only one
independent variable, and multiple regression have more than one
independent variables.
Linear regression
Y = a + b.X
• Outlier Analysis
- Clustering
- Similar values are organized into groups, or “clusters.” Intuitively,
values that fall outside of the set of clusters may be considered
outliers.
 Data Transformation
Converting the raw data into a format that is more suitable for the data mining
task. This can include normalizing numerical data, and encoding categorical data.
• Smoothing: removes noise from the data. Such techniques include binning,
regression, and clustering.
• Aggregation, where summary or aggregation operations are applied to the data.
For example, the daily sales data may be aggregated so as to compute monthly
and annual total amounts. This step is typically used in constructing a data cube
for analysis of the data at multiple granularities.
• Generalization of the data, where low-level or “primitive” (raw) data are replaced
by higher-level concepts through the use of concept hierarchies. For example,
categorical attributes, like street, can be generalized to higher-level concepts, like
city or country.
• Data normalization: where the attribute data are scaled so as to fall within a
small specified range, such as −(-1.0, 1.0) or (0.0 to 1.0).
• Attribute selection
• Discretization
• Concept hierarchy generation
Data Transformation

Normalization
• min-max normalization,
• z-score normalization, and
• normalization by decimal scaling
Data Transformation
Normalization
min-max normalization:
• Performs a linear transformation on the original data.
• Suppose that minA and maxA are the minimum and maximum values of an attribute, A.
• Min-max normalization maps a value, v, of A to v’ in the range [new_minA, new_maxA]
by computing

• Min-max normalization preserves the relationships among the original data values.

Suppose that the minimum and maximum values for the attribute income are $12,000 and
$98,000, respectively. Map income to the range [0.0,1.0].
v= $73,600
v’ = [(73,600−12,000) / (98,000−12,000)] x (1.0−0) +0 = 0.716
Data Transformation
• Normalization
• z-score normalization (or zero-mean normalization), the values for an attribute, A,
are normalized based on the mean and standard deviation of A. A value, v, of A is
normalized to v’ by computing

• With z-score normalization, a value of $73,600 for income is transformed to

(73,600−54,000)/ 16,000 = 1.225.
• Normalization by decimal scaling normalizes by moving the decimal point of values
of attribute A. The number of decimal points moved depends on the maximum
absolute value of A. A value, v, of A is normalized to v’ by computing

• where j is the smallest integer such that Max(|v 0 |) < 1.

Data Transformation
• Normalization by Decimal scaling.
Suppose that the recorded values of A range from −986 to 917.
The maximum absolute value of A is 986.
To normalize by decimal scaling, we therefore divide each value by
1,000 (i.e., j = 3)
so that −986 normalizes to −0.986 and 917 normalizes to 0.917.
• Use the two methods below to normalize the following group of data:
200, 300, 400, 600, 1000
• (a) min-max normalization by setting min = 0 and max = 1
• (b) z-score normalization
• Suppose that the data for analysis includes the attribute age. The age
values for the data tuples are (in increasing order) 13, 15, 16, 16, 19,
20, 20, 21, 22, 22, 25, 25, 25, 25, 30, 33, 33, 35, 35, 35, 35, 36, 40, 45,
46, 52, 70.
• Use min-max normalization to transform the value 35 for age onto
the range [0.0,1.0].
• (b) Use z-score normalization to transform the value 35 for age,
where the standard deviation of age is 12.94 years.
• (c) Use normalization by decimal scaling to transform the value 35 for
age.
• Suppose a group of 12 sales price records has been sorted as follows:
5, 10, 11, 13, 15, 35, 50, 55, 72, 92, 204, 215 Partition them into three
bins by each of the following methods:
• (a) equal-frequency (equidepth) partitioning
• (b) equal-width partitioning
• For the attribute age: 13, 15, 16, 16, 19, 20, 20, 21, 22, 22, 25, 25, 25,
25, 30, 33, 33, 35, 35, 35, 35, 36, 40, 45, 46, 52, 70. (a) Use smoothing
by bin means to smooth these data, using a bin depth of 3.
Data Transformation
• Attribute Construction/ Feature Construction: New attributes are
constructed from the given attributes and added in order to help
improve the accuracy and understanding of structure in high-
dimensional data.
For example, we may wish to add the attribute area based on the
attributes, height and width.
• Discretization
Discretization is one form of data transformation technique. It
transforms numeric values to interval labels/ conceptual labels. It is
done based on some concepts.
Ex. Age (10,11,23,25,35,45,50,60,70) can be transformed to (0-5, 6-
12,13-19, 20-39, 40-59, 60-120) or to conceptual labels like baby, child,
teen,youth,adult, middle-aged, senior citizen.
• Supervised Discretization: supervised discretization methods are supplied
with a class label for each data item value.
• Unsupervised Discretization: Unsupervised discretization methods are not
provided with class label information
• Discretization by Binning
• Discretization by Histogram Analysis
• Discretization by Cluster, Decision Tree, and Correlation Analyses
• Discretization from The Top-down
A top-down method starts with a single interval that includes all
data attribute values and then generates a set of intervals by splitting the
initial interval into two or more intervals. .
• Discretization from The Bottom-up
A bottom-up method initially considers each data point as a separate
interval. It then selects one or more adjacent data points and merges them
into a new interval.
 Data Transformation
• Concept hierarchy generation for nominal data: where nominal
attributes can be generalized to higher-level concepts.
• Street < city < country
Concept hierarchy generation for nominal data
1. Specification of a partial ordering of attributes explicitly at the
schema level by users or experts:
2. Specification of a portion of a hierarchy by explicit data grouping:
This is essentially the manual definition of a portion of a concept
hierarchy.
3. Specification of a set of attributes, but not of their partial ordering:
A user may specify a set of attributes forming a concept hierarchy, but
omit to explicitly state their partial ordering. The system can then try to
automatically generate the attribute ordering so as to construct a
meaningful concept hierarchy.
4. Specification of only a partial set of attributes
 Data Reduction
• Data reduction: Obtain a reduced representation of the data set that is much smaller in volume but yet
produces the same (or almost the same) analytical results
• Why data reduction? — A database/data warehouse may store terabytes of data. Complex data analysis
may take a very long time to run on the complete data set.
• As data sets increase in size, exploration, manipulation, and analysis become more complicated and
resource consuming.

Data reduction strategies

◦ Dimensionality reduction, e.g., remove unimportant attributes
◦ Wavelet transforms
◦ Principal Components Analysis (PCA)
◦ Feature subset selection (or Feature selection)
◦ Numerosity reduction (some simply call it: Data Reduction)
◦ Regression and Log-Linear Models
◦ Histograms, clustering, sampling
◦ Data compression
 Dimensionality Reduction
What are Dimensions?
• Dimensions are features that may be dependent or independent.
• The curse of dimensionality in machine learning is defined as follows,
As the number of dimensions or features increases, the amount of data needed to
generalize the machine learning model accurately increases exponentially. The
increase in dimensions makes the data sparse, and it increases the difficulty of
generalizing the model. More training data is needed to generalize that model
better.
Dimensionality reduction
◦ Avoid the curse of dimensionality
◦ Help eliminate irrelevant features and reduce noise
◦ Reduce time and space required in data mining
◦ Allow easier visualization
 Curse of Dimensionality - issues

•Data Sparsity: Data points become increasingly spread out, making it hard to find patterns or
relationships.
•Computational Complexity: The computational burden of algorithms increases exponentially.
•Overfitting: Models become more likely to memorize the training data without generalizing well.
•Distortion of Distance Metrics: Traditional distance metrics become less reliable in measuring proximity.
•Visualization Challenges: Projecting high-dimensional data onto lower dimensions leads to loss of
information.
•Algorithmic Efficiency: Algorithms need to be scalable and efficient to handle the complexity of high-
dimensional spaces.
•Interpretability Issues: Understanding the decision-making process of high-dimensional models becomes
increasingly difficult.
•Data Storage Requirements: Efficient data storage and retrieval strategies are essential for managing large
volumes of high-dimensional data
 Dimensionality Reduction

Dimensionality reduction techniques

(Create new attributes that can capture the important information in
a data set much more efficiently than the original attributes)
◦ Wavelet transforms
◦ Principal Component Analysis
◦ Attribute Subset Selection
 What Is Wavelet Transform?
• The discrete wavelet transform (DWT) is a linear signal processing technique. Applicable
to n-dimensional signals.
• Wavelet Transform is a lossy method for dimensionality reduction, where a data vector X
is transformed into another vector X’, in such a way that both X and X’ still represent the
same length. The result of wavelet transform can be truncated, unlike its original, thus
achieving dimensionality reduction.
• It transforms a vector into a numerically different vector (D to D’) of wavelet
coefficients. The two vectors are of the same length.
• Wavelet transformed data can be truncated.
• From the resultant coefficients, it is enough to store only a small fraction of the strongest
wavelet coefficient (i.e., wavelet coefficients larger than some particular threshold, All
other coefficients are set to 0. ) → A compressed approximation of the data
• Given a set of wavelet coefficients, an approximation of the original data can be obtained
by applying the inverse DWT.
• Lossy compression technique
• Wavelet transforms can be applied to multidimensional data such as data cubes.
• Wavelet transforms have many real world applications, including the compression of
fingerprint images, computer vision, and analysis of time-series data and data cleaning
 Wavelet Transform
The general procedure for applying a discrete wavelet transform uses a hierarchical
pyramid algorithm that halves the data in each iteration, resulting in fast
computational speed. The method is as follows:
• The length, L , of the input data vector must an integer power of 2. This condition
can be met by padding the data vector with zeros as necessary.
• Each transform involves applying two functions. The first applies some data
smoothing, such as sum or weighted average .The second performs a weighted
difference, which acts to bring out the detailed features of the data.
• The two functions are applied to pairs of input data, resulting in two sets of data
of length L/2.
• The two functions are recursively applied to sets of data obtained in the previous
loop, until the resulting data sets obtained are of length 2.
• A selection of values from the data sets obtained in the above iterations are
designated as the wavelet coefficients of the transformed data.
Principal Component Analysis
• Principal component analysis, or PCA, is a dimensionality reduction
method that is often used to reduce the dimensionality of large data
sets, by transforming a large set of variables into a smaller one that still
contains most of the information in the large set.
• reduce the number of variables of a data set, while preserving as much
information as possible.
• Transforms the correlated original variables into a new set of linearly
uncorrelated variables , known as principal components, that capture
most information/variance.
• Principal components are the linear combinations of original variables
and are uncorrelated to each other.
• Principal components are numbered based on the variance retained in
them. Hence, the principal component which has the maximum
variance is known as the first principal component.
• Finds a projection that captures the largest amount of variation in data
• Geometrically speaking, principal components
represent the directions of the data that explain
a maximal amount of variance, that is to say, the lines
that capture most information of the data (Find a
projection that captures the largest amount of variation
in data). The relationship between variance and
information here, is that, the larger the variance
carried by a line, the larger the dispersion of the data
points along it, and the larger the dispersion along a
line, the more information it has.

• If the two components are uncorrelated, their directions

should be orthogonal.
• With n dimensions, n principal components.
 PCA (Principle Component Analysis) is defined as an
orthogonal linear transformation that transforms the data to
a new coordinate system such that the greatest variance
comes to lie on the first coordinate, the second greatest
variance on the second coordinate and so on.
Principal Component
Each Coordinate in Principal Component Analysis is
called Principal Component.

Ci = bi1 (x1) + bi2 (x2) + … + bin(xn)

where, Ci is the ith principal component, bij is the

regression coefficient for observed variable j for the
principal component i and xi are the
variables/dimensions.
• Step 1: Standardization/ Normalization
• The aim of this step is to standardize the range of the continuous initial
variables so that each one of them contributes equally to the analysis. If
there are large differences between the ranges of initial variables, those
variables with larger ranges will dominate over those with small ranges (for
example, a variable that ranges between 0 and 100 will dominate over a
variable that ranges between 0 and 1), which will lead to biased results. So,
transforming the data to comparable scales can prevent this problem.

• Once the standardization is done, all the variables will be transformed to

the same scale.
• Step 2: Covariance Matrix computation
• The variance is a measure of how far a set of numbers is spread out.
The aim of this step is to understand how the variables of the input
data set are varying from the mean with respect to each other, or in
other words, to see if there is any relationship between them. Because
sometimes, variables are highly correlated in such a way that they
contain redundant information. So, in order to identify these
correlations, we compute the covariance matrix.

For example, for a 3-dimensional data set with 3 variables x, y, and z,

the covariance matrix is a 3×3 data matrix of this from:
• What do the covariances that we have as entries of the matrix tell
us about the correlations between the variables?
• It’s actually the sign of the covariance that matters:
✓If positive then: the two variables increase or decrease together
(correlated)
✓If negative then: one increases when the other decreases (Inversely
correlated)
 Variance and Covariance

• The variance is a measure of how far a set of

numbers is spread out.

• Covariance is a measure of how much two

random variables change together.
• Covariance Matrix is a n*n matrix where each element can be defined
as

• A covariance matrix over 2 dimensional dataset is

• Step 3: Compute the eigenvectors and eigenvalues of the covariance
matrix to identify the principal components
The eigenvectors of the Covariance matrix are actually the directions of
the axes where there is the most variance (most information) and that
we call Principal Components. And eigenvalues are simply the
coefficients attached to eigenvectors, which give the amount of
variance carried in each Principal Component.
By ranking your eigenvectors in order of their eigenvalues, highest to
lowest, you get the principal components in order of significance.
• Eigenvector of a square matrix is defined as a non zero-vector in
which when a given matrix is multiplied, it is equal to a scalar
multiple of that vector.
• The vector x is an eigenvector of the matrix A with eigenvalue λ
(lambda) if the following equation holds:
• Step 4. Sort Eigenvectors by Eigenvalues:
– Sort the eigenvectors based on their corresponding eigenvalues in
descending order.
• Step 5 Choose the top k eigenvectors :
– Select the top k eigenvectors (principal components) where k is the
desired dimensionality of the reduced dataset.
• Step 6 Transform the Data:
– Multiply the original standardized data by the selected principal
components to obtain the new, lower-dimensional representation of the
data.
• Works only for numeric data
Data Reduction - Dimensionality Reduction

Attribute Subset Selection

reduces the data set size by removing irrelevant or redundant
attributes (or dimensions).
• The goal of attribute subset selection is to find a minimum set of
attributes such that the resulting probability distribution of the data
classes is as close as possible to the original distribution obtained
using all attributes.
• Mining on a reduced set of attributes has an additional benefit. It
reduces the number of attributes appearing in the discovered
patterns, helping to make the patterns easier to understand.
 Attribute Subset Selection
Basic heuristic methods of attribute subset selection include the following
techniques.
• Stepwise forward selection: The procedure starts with an empty set of attributes
as the reduced set. The best of the original attributes is determined and added to
the reduced set. At each subsequent iterate on or step, the best of the remaining
original attributes is added to the set.
• Stepwise backward elimination: The procedure starts with the full set of
attributes. At each step, it removes the worst attribute remaining in the set.
• Combination of forward selection and backward elimination: The stepwise
forward selection and backward elimination methods can be combined so that, at
each step, the procedure selects the best attribute and removes the worst from
among the remaining attributes.
• Decision tree induction: Decision tree algorithms, such as ID3, C4.5, and CART,
were originally intended for classification. Decision tree induction constructs a
flowchart like structure where each internal (non-leaf) node denotes a test on an
attribute, each branch corresponds to an outcome of the test, and each external
(leaf) node denotes a class prediction. At each node, the algorithm chooses the
“best” attribute to partition the data into individual classes.
Numerosity Reduction
- reducing the data volume by choosing an alternative, ‘smaller’ forms of data representation
- Two primary approaches to numerosity reduction include parametric and non-parametric methods.
- Parametric numerosity reduction is a technique that involves storing only the essential parameters of
data instead of the entire original dataset. E.g., Regression model and Log-linear model
• Regression:
• Linear regression and
• multiple linear regression
• Log-Linear models
• - Non-parametric: Non-parametric numerosity reduction techniques don’t rely on specific models.
Instead, they aim to reduce data more broadly without assuming a particular structure within the
data. While they offer a uniform reduction across various data sizes, parametric techniques might
achieve a lower reduction rate than they do.
• Unlike the parametric method, these methods may not give a very high decrease in data reduction
• histograms,
• clustering, and
• Sampling
• Data cube aggregation
 Numerosity Reduction
• Linear regression
❖creating a model representing the relationship between two
attributes through a linear equation.
❖ y = wx + b,
‘y’ represents the outcome attribute,
‘x’ signifies the predictor attribute,
while ‘w’ (slope of the line) and ‘b’ (intercept of the line) are the
coefficients of the regression.

b = y̅̅̅̅ - wx̅̅
• Multiple linear regression extends the idea of modeling the response
variable by considering linear relationships among two or more
predictor variables.
y=b0+b1x1+b2x2
x y X-x̅ Y-y̅ (X-x̅)2
2 3
4 7
6 5 b = y̅̅ - wx̅
8 10

W = 0.95; b = 1.5
Σ Σ Σ
y = .95 x + 1.5
Data Reduction – Numerosity Reduction
Log-linear Models
• The log-linear model discovers the relationship between two or more
discrete attributes. Assume, we have a set of tuples in n-dimensional space;
the log-linear model helps to derive the probability of each tuple in this n-
dimensional space.
• Log-linear models can be used to estimate the probability of each point in a
multidimensional space for a set of discretized attributes, based on a smaller
subset of dimensional combination. This allows a higher-dimensional data
space to be constructed from lower-dimensional spaces.
• They may be used to estimate the probability value associated with data
cube cells
• Hence, applies to dimensionality reduction also.
 Log-linear Models

• A log-linear model is a type of statistical model that is used

to analyze (non-linear) relationships between categorical
variables.
• The log (natural log, ln) of the response variable is
modelled using a linear combination of predictors.
ln y = b0 + b1x1 + b2x2 + epsilon
• A log-linear model in econometrics is a mathematical
approach that transforms non-linear relationships into
linear forms using logarithmic functions.
Numerosity Reduction
Non-parametric method
Histogram
• Histograms use binning to approximate data distributions - A
histogram for an attribute, A, partitions the data distribution of A into
disjoint subsets, or buckets.
• If each bucket represents only a single attribute-value/frequency pair,
the buckets are called singleton buckets.
There are several partitioning rules, including the following:

• Equal-width: In an equal-width histogram, the width of each bucket range is uniform

• Equal-frequency (or equidepth): In an equal-frequency histogram, the buckets are created

so that, roughly, the frequency of each bucket is constant (that is, each bucket contains
roughly the same number of contiguous data samples).

• V-Optimal: group contiguous sets of frequencies into buckets so as to minimize the

variance of the overall frequency approximation. If we consider all of the possible
histograms for a given number of buckets, the V-Optimal histogram is the one with the least
variance.

• MaxDiff: In a MaxDiff histogram, we consider the difference between each pair of adjacent
values.
- Suppose we need β buckets. Then, a bucket boundary is established between two values
that are adjacent (in sort parameter order) if the difference between these values is one of
the β -1 largest such differences.
- The goal is to avoid grouping attribute values with vastly different source parameter values
into a bucket.
• The following data are a list of prices of
commonly sold items at AllElectronics
(rounded to the nearest dollar). The
numbers have been sorted: 1, 1, 5, 5, 5,
5, 5, 8, 8, 10, 10, 10, 10, 12, 14, 14, 14,
15, 15, 15, 15, 15, 15, 18, 18, 18, 18, 18,
18, 18, 18, 20, 20, 20, 20, 20, 20, 20, 21,
21, 21, 21, 25, 25, 25, 25, 25, 28, 28, 30,
30, 30.
 Data Cube Aggregation
• multidimensional aggregation
• Aggregation at various levels of a data cube
• For example, suppose you have the data of All
Electronics sales per quarter for the year 2018 to
the year 2022. If you want to get the annual sale
per year, you just have to aggregate the sales per
quarter for each year. In this way, aggregation
provides you with the required data, which is much
smaller in size, and thereby we achieve data
reduction even without losing any data.
• The data cube present precomputed and
summarized data
Sampling
• Simple random sample without replacement (SRSWOR) of size s: This is
created by drawing s of the N tuples from D (s < N), where the probability
of drawing any tuple in D is 1/N, that is, all tuples are equally likely to be
sampled.
• Simple random sample with replacement (SRSWR) of size s: This is similar
to SRSWOR, except that each time a tuple is drawn from D, it is recorded
and then replaced. That is, after a tuple is drawn, it is placed back in D so
that it may be drawn again.
• Cluster sample: If the tuples in D are grouped into M mutually disjoint
“clusters,” then an SRS of s clusters can be obtained, where s < M.
• Stratified sample: If D is divided into mutually disjoint parts called strata, a
stratified sample of D is generated by obtaining an SRS at each stratum.
This helps ensure a representative sample, especially when the data are
skewed.
Discretization and concept hierarchy
generation
• Discretization is the process of transforming numerical variables into
categorical counterparts.
E.g. Converting Age into categories/intervals such as 0-19, 20-39, 40-59
• Discretization improves the accuracy by reducing the noise
• Discretization allows easy identification of outliers, invalid and missing
values of numerical variables
Discretization and Concept Hierarchy
Generation for Numerical Data
• Binning
• Histogram Analysis
• Entropy-Based Discretization
• Interval Merging by χ2 Analysis
• Cluster Analysis
• Discretization by Intuitive Partitioning
Entropy-based Discretization
• In supervised discretization, cut points/split points are determined by
referring to the class information.
• Entropy-based discretization is a supervised discretization
• It finds the best split point so that the intervals are as pure as possible (that
is the majority of the values in a bin correspond to have the same class
label.)
• It helps us split our data at points where we will gain the most insights
• Formally, it is characterized by finding the split with maximal information
gain/ minimum entropy.
• We use the split-point to partition the range of A into two intervals,
corresponding to A ≤ split-point and A > split-point.
• To discretize a numerical attribute, A, the method selects the value of A
that has the minimum entropy/maximum information gain as a split-point
and recursively partitions the resulting intervals to arrive at a hierarchical
discretization. Such discretization forms a concept hierarchy for A.
Entropy-based Discretization
• Do entropy-based discretization for the following dataset.

Note: Calculate entropy at each possible split-point. Select the split-

point with the minimum entropy.

Possible Split-points: At age=21; age=22; age=24; age= 25; age= 27;

age= 35; age= 41
Corresponding to each split-point, the dataset is divided into two
intervals (age<=splitPoint and age> splitPoint) and the entropy is to be
calculated.
• 1) For age=21; Age Grade
The splits are: a) age <=21 b) age >21 22 F
24 P
Age Grade
25 F
21 F 27 P
27 P
𝟏 𝟏 27 P
Entropy for (a) = -[ * log 2 𝟏] =0
𝟏 35 P
41 P

𝟐 𝟐 𝟔 𝟔
Entropy for (b) = -[ x log 2 + x log 2 ] = .812
𝟖 𝟖 𝟖 𝟖
𝟏 𝟖
Total entropy for the entire partitioning = * Entropy for (a) + * Entropy for (b)
𝟗 𝟗
𝟏 𝟖
= *0 + *.812 = .721
𝟗 𝟗
2) For age = 22, splits are a) age<=22 b) age>22
Age Grade Age Grade
21 F 24 P
22 F 25 F
27 P
27 P
𝟐 𝟐
Entropy for (a) = -[ * log 2 𝟐] =0 27 P
𝟐
35 P
41 P
𝟏 𝟏 𝟔 𝟔
Entropy for (b) = -[ x log 2𝟕+𝟕 x log 2 𝟕] = .592
𝟕
𝟐 𝟕
Total entropy for the entire partitioning = * Entropy for (a) + * Entropy for (b)
𝟗 𝟗

𝟐 𝟕
= *0 + *.592 = .460
𝟗 𝟗
3) For age = 24, splits are a) age<=24 Age Grade
b) age>24
Age Grade
21 F
22 F 25 F

24 P 27 P
27 P

𝟐 𝟐 𝟏 𝟏 27 P
Entropy for (a) = -[ * log 2𝟑 + ∗ log 2 ] = 0.592 35 P
𝟑 𝟑 𝟑
41 P
𝟏 𝟏 𝟓 𝟓
Entropy for (b) = -[ x log 2 + x log 2 ] = .095
𝟔 𝟔 𝟔 𝟔
𝟑 𝟔
Total entropy for the entire partitioning = * Entropy for (a) + * Entropy for (b)
𝟗 𝟗

𝟐 𝟕
= *.592 + *.095 = .205
𝟗 𝟗
4) For age = 25, splits are a) age<=25 Age Grade b) age>25 Age Grade
21 F 27 P
22 F 27 P
24 P 27 P
25 F 35 P
41 P
𝟏 𝟏 𝟑 𝟑
Entropy for (a) = -[ * log 2 + ∗ log 2 ] = .811
𝟒 𝟒 𝟒 𝟒

𝟓 𝟓
Entropy for (b) = -[ x log 2 ]=0
𝟓 𝟓

𝟒 𝟓
Total entropy for the entire partitioning = * Entropy for (a) + * Entropy for (b)
𝟗 𝟗

𝟒 𝟓
= *.811 + * 0.0 = .360
𝟗 𝟗
5) For age = 27, splits are a) age<=27 Age Grade b) age>27 Age Grade
21 F 35 P
22 F 41 P
24 P
25 F
27 P
27 P
27 P
𝟑 𝟑 𝟒 𝟒
Entropy for (a) = -[ * log 2 + ∗ log 2 ] = .986
𝟕 𝟕 𝟕 𝟕

𝟐 𝟐
Entropy for (b) = -[ x log 2 ]=0
𝟐 𝟐

𝟕 𝟐
Total entropy for the entire partitioning = * Entropy for (a) + * Entropy for (b)
𝟗 𝟗

𝟕 𝟐
= * .986 + * 0.0 = .767
𝟗 𝟗
6) For age = 35, splits are a) age<=35 Age Grade b) age>35 Age Grade
21 F 41 P
22 F
24 P
25 F
27 P
27 P
27 P
35 P
𝟑 𝟑 𝟓 𝟓
Entropy for (a) = -[ * log 2𝟖 + ∗ log 2 ] = .954
𝟖 𝟖 𝟖

𝟏 𝟏
Entropy for (b) = -[ x log 2 ]=0
𝟏 𝟏

𝟖 𝟏
Total entropy for the entire partitioning = * Entropy for (a) + * Entropy for (b)
𝟗 𝟗

𝟖 𝟏
= * .954 + * 0 = .848
𝟗 𝟗
 The lowest entropy is at 24. Hence the data set is divided into
intervals for age<=24 and age>24.
Split-point Entropy
Age Grade Age Grade
At 21 .721
21 F 25 F
At 22 .460 22 F 27 P

At 24 .205 24 P 27 P
27 P
At 25 .360
35 P
At 27 .767 [21-41] 41 P

At 35 .848

[21-24] [25- 41]

Again, partitioning is done recursively in each interval

until some stopping condition (for e.g. we can set a threshold
for number of intervals or for the resultant entropy ) is met.