School of Informatics

Department of Information Technology

Course: Artificial Intelligence and Machine learning
Course Code: (CoSc 2063)
Degree Program: MSc IT Year 1 Sem I [Regular]

Compiled by: Siraj Sebhatu (PhD)


 E-mail – [Link]@[Link]

3:07:57 PM 1
 Chapter Three

Data Preprocessing for Machine Learning

Outline
• Data Processing
• Data quality problems
• Data preprocessing
• Data cleaning
• Data integration
• Data transformation
• Data reduction
 What is Data Processing
• Data processing is typically used for the initial stage
followed by analysis to handle the overall data.
 The collection and manipulation of items of data to
produce meaningful information.
 The processing is in any manner detectable by an observer.
 The conversion of data into usable and desired form. The conversion
is carried out using a predefined sequence of operations either
 Manual data processing or
 Automatic (Electronic) data processing
Data processing
• Data processing is the conversion of data into usable and
desired form. This conversion or “processing” is carried out
using a predefined sequence of operations either manually
or automatically.
• Most of the data processing is done by using computers and
thus done automatically.
• The raw data is collected, filtered, sorted, processed,
analyzed, stored, and then presented in a readable format.
• Manual Data Processing
Data processing ……
 Data processing may involve various processes;
 Validation: ensuring that supplied data is correct and relevant.
 Sorting: is an arrangement of items in some sequence and/or in different
sets.
 Summarization: reducing detailed data to its main points.
 Aggregation: combining multiple pieces of data.
 Analysis: collection, organization, analysis, interpretation and presentation
of data.
 Reporting: list detail or summary of the data processed.
 Classification: separation of data into various categories.
Data quality problems
• Incomplete Data
• Data duplication
• Inconsistent Formats
• Accessibility
• System upgrades
• Data purging and storage
• Poor organization
Incomplete Data
• Data can be incomplete from many different reasons
 The data is accurate but incomplete

 Data has not been entered in the system correctly,

 Certain files may have been corrupted,
 Some data has several missing variables.
Data duplication
• Duplicate data is one of the most common and annoying issues that
you may encounter when preparing your data for machine learning.
• It can affect the quality, performance, and reliability of your models,
as well as waste your time and resources.
• Multiple copies the same take a roll on the
computation and storage

Duplicated data Deduplicated

from source data
Inconsistent Formats
• Inconsistent data formatting refers to the situation where the data you
want to use for machine learning has different formats, structures, or
standards.
• For example, you may have data from different sources that use
different date formats, currencies, units, or encodings.
Data cleaning
• A key component of data preprocessing, the process of identifying
removing or correcting irrelevant, incomplete, or inaccurate data ,
correct the errors, and handle missing values.
• This process is essential because the quality of the data used in
machine learning significantly impacts the performance of the models
 Advantages of data processing in Machine
Learning
• Improved model performance: Data processing helps improve the
performance of the ML model by cleaning and transforming the data
into a format that is suitable for modeling.
• Better representation of the data: Data processing allows the data to
be transformed into a format that better represents the underlying
relationships and patterns in the data, making it easier for the ML
model to learn from the data.
• Increased accuracy: Data processing helps ensure that the data is
accurate, consistent, and free of errors, which can help improve the
accuracy of the ML model.
 Disadvantages of Data Processing in Machine
Learning
• Time-consuming: Data processing can be a time-consuming task,
especially for large and complex datasets.

• Error-prone: Data processing can be error-prone, as it involves

transforming and cleaning the data, which can result in the loss of
important information or the introduction of new errors.

• Limited understanding of the data: Data processing can lead to a

limited understanding of the data, as the transformed data may not be
representative of the underlying relationships and patterns in the data.
Data Preprocessing

• Data preprocessing involves cleaning and transforming the data to

make it suitable for analysis.
• The goal of data preprocessing is to make the data accurate, consistent,
and suitable for analysis. It helps to improve the quality and efficiency
of the machine learning process.
• The process of generating raw data for machine learning models. This
is the first step in creating a machine-learning model.
• This is the most complex and time-consuming aspect of ML. Data
preprocessing is required in machine learning algorithms to reduce its
complexities.
Preprocessing operation
 Each operation aims to help ML build better predictive models. Some of the
operations for structured data:
 Data cleansing
 Removing or correcting records with corrupted or invalid values from raw data, as well as
removing records that are missing a large number of columns.
 Instances selection and partitioning
 Selecting data points from the input dataset to create training, evaluation (validation), and
test sets using random sampling, minority classes oversampling, and stratified partitioning.
 Feature tuning
 Improving the quality of a feature for ML, which includes scaling and normalizing
numeric values, imputing missing values, clipping outliers, and adjusting values with
skewed distributions.
Cont..
 Representation transformation
 Converting a numeric feature to a categorical feature and vice verse.
 Feature extraction
 Reducing the number of features by creating lower-dimension and more powerful data
representations using PCA, embedding extraction, and hashing.
 Feature selection
 Selecting a subset of the input features for training the model, and ignoring the irrelevant
or redundant ones, using filter or wrapper methods which involve simply dropping features
if the features are missing a large number of values.
 Feature construction
 Creating new features either by using typical techniques, such as polynomial
expansion or feature crossing. The process of creating new features or variables.
Data Cleaning

 Is the process of preparing data for analysis by removing or modifying

data that is incorrect, incomplete, irrelevant, duplicated, or improperly
formatted.
 Data cleaning clean the data by:
 Filter unwanted outliers and smoothing noisy data
 Remove duplicate and irrelevant observations
 Fix structural errors such as types or inconsistent capitalization
 Filling in missing values
Missing Values Handling
• Missing values are data points that are absent for a specific variable in
a dataset.
• They can be represented in various ways, such as blank cells, null
values, or special symbols like “NA” or “unknown.”
• These missing data points pose a significant challenge in data analysis
and can lead to inaccurate or biased results.
 Missing values: Incomplete
• Some of the most common ones are as
follows:
• Incomplete filling of forms by the users
• If the database is migrated from some
other, some data may have
been lost
• Errors due to programs or other
technical reasons
Missing Values Handling or Treatment
• Deleting rows with null values: One of the easiest ways to handle
null values is to simply delete any rows containing them.
• However, if the number of rows containing null values is large, this
method can result in appreciable data loss.
• Mean or median impute: Another common method to deal with the
null values is to replace these with the mean or median of the
corresponding feature.
• However, this method works well if the data of the feature is normally
distributed.
 Missing Values Handling or Treatment
• Filling values with a constant value: Sometimes null values can be
replaced by 0 (or some other agreed-upon value). Such values tell the
model that the corresponding value in the original data was missing.
• Using KNN Impute: One of the popular methods to replace the
missing values is to use algorithms like K- Nearest Neighbors
(KNN) to find the missing values.
• The KNN algorithm fills the
unknown data with the average of those of the K nearest neighbors.
As a matter of fact, you can use other Machine Learning algorithms
as well to accomplish the task.
Missing Values Handling or Treatment
• Using Linear interpolation: To fill in missing values, you can use
linear interpolation if the data has a time-based structure. This
method works if the missing values lie on a linear line as determined
by the algorithms.
Missing Values Handling
• The choice of the algorithm depends on the dataset and the task that
you aim to perform with the given data.
• Also, the nature of the missing data is an important factor in deciding
which of the above techniques should be used.
• Figure 2.2
 The following codes Missing Values Treatment
The missing value is then filled in using the following:

• ffill (forward fill), which replaces the missing value with that in the next
row.
• bfill (backward fill), which replaces the missing value with that in the
previous row
• fillna([Link]()) replaces the missing values with the mean of the column
• fillna([Link]()) replaces the missing values with the median of the
column
• KNNImputer() replaces the missing values using the KNN algorithm
 Data Integration
• Data integration combines data from multiple sources (database, data
warehouse, files & sometimes from non-electronic sources) into a
coherent store
• Because of the use of different sources, data that that is fine on its own
may become problematic when we want to integrate it.
• Some of the issues are:
• Different formats and structures
• Conflicting and redundant data
• Data at different levels

27
Data Integration Methods
• Extract, Transform, Load (ETL): This method involves:
Extracting data from multiple sources, Transforming it to a common
format and Loading it into a target system
• Extract, Load, Transform (ELT): In this method, the data is loaded
into the target system, and then the transformation is done.
• Data consolidation: This method involves physically integrating data
from multiple sources into a single repository. Though it is a time
consuming process, it allows for efficient analysis of the data.
• Data synchronization: In this method, data in multiple sources are
synchronized with each other, so that changes made in one source are
reflected in the others.
Data integration: Format
• Not everyone uses the same format. Do you agree?
– Schema integration: e.g., [Link]-id  [Link]-#
• Integrate metadata from different sources
• Dates are especially problematic:
– 12/19/97
– 19/12/97
– 19/12/1997
– 19-12-97
– Dec 19, 1997
– 19 December 1997
– 19th Dec. 1997
• Are you frequently writing money as:
– Birr 200, Br. 200, 200 Birr, …
 Data Integration: Inconsistent
• Inconsistent data: containing discrepancies in codes or names, which
is also the problem of lack of standardization / naming conventions.
e.g.,
• Age=“26” vs. Birthday=“03/07/1986”
• Some use “1,2,3” for rating; others “A, B, C”
• Discrepancy between duplicate records

ID Name City State

1 Ministry of Transportation Addis Ababa Addis Ababa region
Addis Ababa
2 Ministry of Finance Addis Ababa administration
Addis Ababa regional
3 Office of Foreign Affairs Addis Ababa administration
 Data Integration: Different structure
What’s wrong here? No data type constraints
ID Name City State
Ministry of
1234 Transportation Addis Ababa AA

ID Name City State

GCR34 Ministry of Finance Addis Ababa AA

Name ID City State

Office of Foreign Affairs
GCR34 Addis Ababa AA
32
Data Transformations
• Data transformation: Many times, the given data needs to be
converted into a suitable form for analysis and modeling. This can be
done by converting variables into different formats.
Data Transformations / Normalization
 ML data may not be in the right format or may require
transformations to make it more useful. Data Transformation
activities and techniques include:
 Categorical encoding
▪ Label encoding converts categorical variables to numerical
representation, something that is machine-readable.
 Dealing with skewed data
▪ Regression algorithms with linear regression or ANN, a better
improvements registered with more symmetric distribution, you
can use roots (square-root, cube root), logarithms (base e, or base
10), reciprocals (positive or negative), or Box-Cox
transformation.
Data Transformations / Normalization

 Scaling
▪ Scaling is a method of transforming data into a particular range.
This is important when using regression algorithms and
algorithms using Euclidean distances (e.g. KNN, or K-Means) as
they are sensitive to the variation in magnitude and range across
features.
▪ The goal of scaling is to change the values of each numerical feature
in the data set to a common scale. Such as min-max scaling or z-
score standardization.
Data Transformations / Normalization

• Min-max normalization

• Z-score (zero-mean) normalization

Data Transformations / Normalization
Cont..

Five number
summary
Data Reduction
 Most machine learning techniques may not be effective for high-
dimensional data. Adatabase or date warehouse may store TB of
data.
 This may take very long to perform data analysis on such huge
amounts of data.
 Data reduction techniques can be applied to
• obtain a reduced representation of the actual data in volume but still
contain critical information.
Feature Engineering

 This refers to the dataset with the tuned features expected by the
model.
 Performing certain ML specific operations on the
• columns in the prepared dataset, and creating new features for your
model during training and prediction under Preprocessing operations.
Feature Engineering
• It may be necessary to create new features by combining or
transforming existing variables. This step can improve the performance
of machine learning models.
• This refers to the dataset with the tuned features expected by the
model.
• Performing certain ML specific operations on the columns in the
prepared dataset, and creating new features for your model during
training and prediction under Preprocessing operations.
• Scaling numerical columns to a value between 0 and 1, clipping values,
and one-hot-encoding categorical features.
Thank You!!!
Any Questions

End of Chapter Three