Chapter-2

Data Pre-Processing
◾ 2.1 Why preprocess the data?

◾ 2.2 MajorTasks in Data Preprocessing

◾ Data Exploration

◾ Data understanding

◾ Data cleaning and reduction

◾ Data Integration andTransformation

◾ Discretization and concept hierarchy generation

Data Pre-Processing
◾ Machine Learning requires collecting great amount of data (available in data
warehouses or databases) to achieve the intended objective. Machine Learning starts by
understanding the business or problem domain in order to gain the business knowledge

◾ When creating a machine-learning project, it is not always a case that we come across
the clean and formatted data. Moreover, while doing any operation with data, it is
mandatory to clean it and put in a formatted way. Therefore, for this, we use data pre-
processing task.

◾ Data pre-processing is a process of preparing the raw data and making it suitable
for a machine-learning model. It is the first and crucial step while creating a machine-
learning model

Why Pre-Process the data

◾ A real-world data generally contains noises, missing values, and maybe in an unusable format
which cannot be directly used for machine learning models.

A. It transforms raw data into meaningful information

B. New technologies like ML (Machine Learning) is highly dependent upon data

 C. It is simply used for achieving better results from applying model

D. The dataset should be formatted in such way that more than one ML and deep
algorithm are executed in one dataset and then best out of them is selected.

E. It increases accuracy and efficiency of an ML model

F. It provides and improves generalizability of ML model

Data Quality Measures:

◾ Before feeding data to machine learning algorithm, we have to make sure the quality of data?

◾ A well-accepted multidimensional data quality measures are the following:

– Accuracy (No errors, no outliers): To check whether the data entered is correct or not.

– Completeness (no missing values): To check whether the data is available or not recorded

– Consistency (no inconsistent values and attributes):To check whether the same data is
kept in all the
places that do or do not match.

– Timeliness (appropriateness): The data should be updated correctly.

– Believability (acceptability): The data should be trustable

– Interpretability (easy to understand): The understandability of the data

◾ Most of the data in the real world are poor quality; that is: Incomplete, Inconsistent,
Noisy, Invalid, Redundant

Data is often of low Quality. Why?

◾ You didn’t collect it yourself!

◾ It probably was created for some other use,and then you came along
wanting to integrate it
 ◾ People make mistakes (typos)

◾ People are busy (“this is good enough”) to systematically organize

carefully using structured formats

Types of problems with data

◾ Some data have problems on their own that needs to be cleaned:

– Outliers:misleading data that do not fit to most of the data/facts
– Missing data: attribute values might be absent which needs to be replaced with
estimates
– Irrelevant data:attributes in the database that might not be of interest to the machine
learning task being developed
– Noisy data: attribute values that might be invalid or incorrect. E.g. typographical errors
– Inconsistent data, duplicate data, etc.

◾ Other data are problematic only when we want to integrate it

– Everyone had their own way of structuring and formatting data,based on what
was convenient for them
How to integrate data organized in different format following different conventions

CASE STUDY: GOVERNMENT AGENCY DATA

◾ What we want

◾ How to prepare enough and complete data we need for machine learning? Coming up with good
 quality data needs to pass through different data pre-processing tasks

MAJOR TASKS IN DATA PREPROCESSING

◾ Data cleansing:to get rid of bad data

– Fill in missing values, smooth noisy data, identify or remove outliers, and resolve
inconsistencies

◾ Data integration
– Integration of data from multiple sources, such as databases, data warehouses, or files

◾ Data reduction
– obtains a reduced representation of the data set that is much smaller in volume, yet produces
almost
the same results.

o Dimensionality reduction
o Numerosity/size reduction
o Data compression
◾ Data transformation
– Normalization
– Discretization and/or Concept hierarchy generation

DATA CLEANING: REDUNDANCY

◾ Duplicate or redundant data is data problems which require data cleaning

• What’s wrong here?

 DATA CLEANING: MISSING DATA

◾ How to clean it:manually or automatically?

◾ Data is not always available, lacking attribute values. E.g., Occupation=“ ”

◾ many tuples have no recorded value for several attributes, such as customer income
in sales data

◾ What’s wrong here? A missing required field

◾ Missing data may be due to

–inconsistent with other recorded data and thus deleted
 –data not entered due to misunderstanding and may not be considered important at
the time of entry
–not register history or changes of the data

◾ How to handle Missing data? Missing data may need to be inferred

– Ignore the missing value:not effective when the percentage of missing values
per attribute varies considerably
– Fill in the missing value manually: tedious + infeasible?
– Fill automatically
Calculate,say,using Expected Maximization (EM)Algorithm the most probable value

DATA CLEANING: NOISY DATA

◾ Noisy:containing noise, errors,or outliers – e.g.,Salary=“-10” (an error)

◾ Typographical errors are errors that corrupt data. Let say ‘green’ is written as ‘rgeen’

◾ Incorrect attribute values may be due to

– faulty data collection instruments (e.g.: OCR)
– data entry problems
– data transmission problems
– technology limitation
– inconsistency in naming convention

DATA CLEANING: HOW TO CATCH NOISY DATA

◾ Manually check all data : tedious + infeasible?

◾ Sort data by frequency

– ‘green’ is more frequent than ‘rgeen’
– Works well for categorical data

◾ Use, say Numerical constraints to Catch Corrupt Data

• Weight can’t be negative
• People can’t have more than 2 parents
• Salary can’t be less than Birr 300
◾ Use statistical techniques to Catch Corrupt Data
– Check for outliers (the case of the 8 meters man)
– Check for correlated outliers using n-gram (“pregnant male”)
 • People can be male
• People can be pregnant
• People can’t be male AND pregnant

DATA INTEGRATION

◾ Data integration combines data from multiple sources (database, data warehouse,
files & sometimes from non-electronic sources) into a coherent store

◾ Because of the use of different sources, data that is fine on its own may become
problematic when we want to integrate it.

◾ Some of the issues are:

– Different formats and structures
– Conflicting and redundant data
– Data at different levels
◾ Not everyone uses the same format. Do you agree?
– Schema integration: e.g.,[Link]-id ≡ [Link]-# when Integrate
metadata from different sources

◾ Dates are especially problematic:

– 12/19/97
– 19/12/97
– 19/12/1997
– 19-12-97
– Dec 19,1997
– 19 December 1997

– 19th Dec. 1997

◾ Are you frequently writing money as: – Birr 200, Br. 200, 200 Birr, …

DATA INTEGRATION: INCONSISTENT

◾ Inconsistent data: containing discrepancies in codes or names,which is
also the problem of lack of standardization / naming conventions. e.g.,
– Age=“29” [Link]=“03/07/1986”

– Some use “1,2,3” for rating;others “A, B,C”

◾ Discrepancy between duplicate records

DATA INTEGRATION: DIFFERENT STRUCTURE

◾ What’s wrong here? No data type constraints

DATA INTEGRATION: CONFLICTING DATA

◾ Detecting and resolving data value conflicts
–For the same real world entity,attribute values from different sources are different

–Possible reasons: different representations, different scales, e.g.,American vs. British units
 • weight measurement: KG or pound
• Height measurement:meter or inch
◾ Information source #1 says that Alex lives in Bahirdar. Information source #2
says that Alex lives in Mekele

◾ What to do?
– Use both (He lives in both places)
– Use the most recently updated piece of information
– Use the “most trusted” information
– Flag row to be investigated further by hand
– Use neither (We’d rather be incomplete than wrong)

HANDLING REDUNDANCY IN DATA INTEGRATION

◾ Redundant data occur often when integration of multiple databases

– Object identification: The same attribute or object may have different

names in different databases

– Derivable data: One attribute may be a “derived” attribute in another table,

e.g., annual revenue,age

◾ Redundant attributes may be able to be detected by correlation analysis and covariance analysis

◾ Careful integration of the data from multiple sources may help

reduce/avoid redundancies and inconsistencies and improve
mining speed and quality

COVARIANCE
Example:

◾ Suppose two stocks A and B have the

following values in
one week: (2, 5), (3,
8), (5, 10), (4, 11), (6,
14).
◾ Question:If the stocks are affected by the same industry trends,will their prices rise or
fall together?
– E (A) = (2 + 3 + 5 + 4 + 6)/ 5 = 20/5 = 4
– E (B) = (5 + 8 + 10 + 11 + 14) /5 = 48/5 = 9.6
– Cov (A,B) = (2×5+3×8+5×10+4×11+6×14)/5 - 4 × 9.6 = 4

◾ Thus,A and B rise together since Cov (A,B) > 0

DATA REDUCTION STRATEGIES

◾ Data reduction: Obtain a reduced representation of the data set that is much smaller in
volume but yet produces the same (or almost the same) analytical results

◾ Why data reduction?A database/data warehouse may store terabytes of [Link] data
analysis may take a very long time to run on the complete data set.

◾ Data reduction strategies

– Dimensionality reduction:Select best attributes or remove unimportant attributes

– Numerosity reduction: Reduce data volume by choosing alternative, smaller forms
of data representation
– Data compression: Is a technology that reduce the size of large files such that smaller
– files take less memory space and fast to transfer over a network or the Internet

DATA REDUCTION: DIMENSIONALITY REDUCTION

Dimensionality reduction

◾ Helps to eliminate Irrelevant attributes and reduce noise: that contain no information
useful for the machine learning task at hand • E.g.,is students' ID relevant to predict
students' GPA?
◾ Helps to avoid redundant attributes :that contain duplicate information in one or more other
attributes E.g.,purchase price of a product & the amount of sales tax paid

◾ Reduce time and space required in machine learning

◾ Allow easier visualization

◾ Method: attribute subset selection

–One of the method to reduce dimensionality of data is by selecting best attributes
–Given M attributes there are 2M possible attribute combinations

HEURISTIC SEARCH IN ATTRIBUTE SELECTION

◾ Commonly used heuristic attribute selection methods:

◾ Best step-wise attribute selection:

• Start with empty set of attributes
• The best single-attribute is picked first
• Then combine best attribute with the remaining to select the best combined
two attributes,then three attributes
• The process continues until the performance of the combined attributes starts to decline
◾ Step-wise attribute elimination:
• Start with all attributes as best
• Eliminate one of the worst performing attribute
• Repeatedly continue the process if the performance of the combined attributes increases
DATA REDUCTION: NUMEROSITY REDUCTION
• Different methods can be used,including Clustering and sampling

• Clustering
• Partition data set into clusters based on similarity,and
store cluster representation
(e.g., centroid and diameter) only
• There are many choices of clustering definitions and
clustering algorithms

• Sampling
• obtaining a small sample s to represent the whole data
set N
• Allow a mining algorithm to run in
 complexity that is potentially sub-
linear to the size of the data
Key principle: Choose a representative subset of the data using suitable sampling technique

TYPES OF SAMPLING
◾ Stratified sampling:
– partition the data set, and draw samples from each partition (proportionally,
i.e., approximately the same percentage of the data)

◾ Simple random sampling

– There is an equal probability of selecting any particular item
– Simple random sampling may have very poor performance in the presence of skew

◾ Sampling without replacement

– Once an object is selected,it is removed from the population

◾ Sampling with replacement

– A selected object is not removed from the population

SAMPLING: CLUSTER OR STRATIFIED SAMPLING

DATA TRANSFORMATION
◾ A function that maps the entire set of values of a given attribute to a new set of
replacement values such that each old value can be identified with one of the
new values

◾ Methods for data transformation

◾ Normalization:Scaled to fall within a smaller, specified range of values

• min-max normalization
• z-score normalization
 ◾ Discretization: Reduce data size by dividing the range of a continuous
attribute into intervals. Interval labels can then be used to replace actual
data values

◾ Discretization can be performed recursively on an attribute using method such as

– Binning: divide values into intervals
– Concept hierarchy climbing: organizes concepts (i.e., attribute values) hierarchically

NORMALIZATION
Min-Max Normalization is a feature scaling technique used to rescale data values to a fixed
range, typically [0,1] or [−1,1]. It helps in making different numerical features comparable by
transforming them to a common scale.

Formula:

Where:

 X is the original value.

 Xmin is the minimum value in the dataset.
 Xmax is the maximum value in the dataset.
 X′ is the normalized value.

Example:

Suppose we have a dataset with values:

[10,20,30,40,50]
Advantages:

 Preserves relationships in the data (does not change the distribution).

 Useful for machine learning algorithms that require normalized input (e.g., neural
networks, k-means clustering).

Disadvantages:

 Sensitive to outliers: If there are extreme values, they will dominate the scaling process.

z-score normalization

Z-Score Normalization (Standardization)

Z-score normalization, also known as standardization, transforms data so that it has a mean of
0 and a standard deviation of 1. This technique is useful when data follows a normal (Gaussian)
Example:

Consider a dataset: [10,20,30,40,50]

 Advantages:

✅ Handles outliers better than Min-Max normalization.

✅ Useful for algorithms that assume normally distributed data, like linear regression and k-
means clustering.
✅ Preserves original distribution of data.

Disadvantages:

❌ If data is not normally distributed, Z-score normalization might not be ideal.

❌ Does not keep data within a fixed range like Min-Max scaling.

SIMPLE DISCRETIZATION: BINNING

Binning into Ranges (Discretization)

Binning is a technique that groups continuous numerical data into intervals or bins (categories).
This is useful for reducing noise, handling outliers, and making patterns in data more
interpretable.

How Binning Works:

1. Define the number of bins (or specify bin ranges).

2. Divide the data into these bins based on the chosen strategy.
3. Assign each value to a bin.
Example of Equal-Width Binning

Dataset:

[5,12,18,24,35,45,50,60,75,85,90,100]
Advantages of Binning:

✅ Reduces noise in continuous data.

✅ Improves interpretability by converting continuous values into categories.
✅ Handles outliers better than raw numerical features.

Disadvantages of Binning:

❌ Loss of precision due to grouping.

❌ Choosing the wrong number of bins can affect data quality.
❌ Equal-width binning can lead to imbalanced bin sizes if data is skewed.
 CONCEPT HIERARCHY GENERATION

◾ Concept hierarchy organizes concepts (i.e., attribute

values) hierarchically and is usually associated with each
dimension in a data warehouse

◾ Concept hierarchy formation: Recursively reduce the

data by collecting and replacing low level concepts (such as
numeric values for age) by higher level concepts (such as
child, youth, adult, or senior)

◾ Concept hierarchies can be explicitly specified by domain

experts and/or data warehouse designers

◾ Concept hierarchy can be automatically formed by the

analysis of the number of distinct values. E.g., for a set of
attributes: {Kebele, city, state, country}

DATA SETS PREPARATION FOR LEARNING

◾ A standard machine learning technique is to divide the dataset into a training set and a test set.

A. Training dataset is used for learning the parameters of the model in order to produce
hypotheses.

◾ A training set is a set of problem instances (described as a set of properties and their
values), together with a classification of the instance.

B. Test dataset, which is never seen during the hypothesis forming stage, is used to get a
final,
unbiased estimate of how well the model works.

◾ Test set evaluates the accuracy of the model/hypothesis in predicting the

categorization of unseen examples.

◾ A set of instances and their classifications used to test the accuracy of a learned
hypothesis
 CLASSIFICATION:TRAIN,VALIDATION,TEST SPLIT

DIVIDE THE DATASET INTO TRAINING &

TEST
◾ There are various ways in which to separate the data into training and test sets

◾ The established ways by which to use the two sets to assess the effectiveness and the
predictive/ descriptive accuracy of a machine learning techniques over unseen
examples.

1. The holdout method

• Repeated holdout method

2. Cross-validation

3. The bootstrap

THE HOLDOUT METHOD

◾ The holdout method reserves a certain amount for testing and uses the remainder for
training
 – Usually:one third for testing,the rest for training
◾ For small or “unbalanced” datasets, samples might not be representative

– Few or none instances of some classes

◾ Stratified sample: advanced version of balancing the data

– Make sure that each class is represented with approximately equal

proportions in both subsets

CROSS-VALIDATION
◾ Cross-validation works as follows:

– First step: data is split into k subsets of equal-sized sets randomly. A partition of
a set is a collection of subsets for which the intersection of any pair of sets is empty.
That is,no element of one subset is an element of another subset in a partition.

– Second step:each subset in turn is used for testing and the remainder for training
◾ This is called k-fold cross-validation

– Often the subsets are stratified before the cross-validation is performed

◾ The error estimates are averaged to yield an overall error estimate
CROSS-VALIDATION EXAMPLE: