Data Pre Processing
What is Data Preprocessing?
Data preprocessing is an essential step in the data analysis and machine learning pipeline. It involves
cleaning, transforming, and organizing raw data to make it suitable for further analysis or
modeling. The main goals of data preprocessing are:
• Improving data quality
• Ensuring consistency
• Making the data more manageable
• Preparing the data for specific analytical or modeling techniques
Why is it important to preprocess data?
It's important to preprocess data as a preparation step for data analysis. Here are four distinct
reasons preprocessing data can help you achieve better results:
• It increases accuracy. By removing missing or inconsistent data values made from human or
computer error, the accuracy of your dataset improves.
• It increases consistency. Data duplicates can happen and removing them during preprocessing
helps ensure more consistent data values get analyzed, producing reliable results that aren't
misrepresented.
• It makes data more complete. Preprocessing data allows you to add in missing data where it's
needed.
• It prepares data for easier algorithm reading. Preprocessing typically makes your data easier to
read, use and interpret, especially when using machine learning software that's automated.
Key aspects of data preprocessing include:
• Data cleaning: Handling missing values, removing duplicates, and correcting errors.
• Data integration: Combining data from multiple sources.
• Data transformation: Scaling, normalization, encoding categorical variables, and feature
engineering.
• Data reduction: Selecting relevant features and reducing dimensionality.
Data Cleaning
This step involves handling missing data, removing duplicates, correcting errors, and dealing with
outliers.
• Handling missing data:
o Ignore/Drop the rows: This approach is suitable only when the dataset we have is
quite large and multiple values are missing within a tuple.
o Standard values like “Not Available” or “NA” can be used to replace the missing
values.
o Missing values can also be filled manually, but it is not recommended when that
dataset is big.
o The attribute’s mean value can be used to replace the missing value when the data is
normally distributed
wherein in the case of non-normal distribution median value of the attribute can be
used.
o While using regression or decision tree algorithms, the missing value can be replaced
by the most probable value.
• Removing duplicates: Duplicate records can skew your analysis and should be removed. the
process involves identifying and deleting identical rows or records within a dataset to ensure
data integrity and accuracy.
• Correcting errors: Errors refers to inconsistencies or irregularities in data. This situation can
occur due to the way data was captured or if data came from different sources. This might
involve fixing typos, misspellings, formatting issues or correcting impossible [Link] handle
this situation identifying the irregularities, then apply techniques like data standardization
• Dealing with outliers: An outlier is an observation that lies an abnormal distance from other
values in a random sample from a population. Outliers can be legitimate extreme values or
errors. You need to investigate them and decide whether to keep, modify, or remove them. It
can be handled in following ways :
o Binning Method: This method works on sorted data in order to smooth it. The whole
data is divided into segments of equal size and then various methods are performed
to complete the task. Each segmented is handled separately. One can replace all data
in a segment by its mean or boundary values can be used to complete the task.
o Regression:Here data can be made smooth by fitting it to a regression [Link]
regression used may be linear (having one independent variable) or multiple (having
multiple independent variables).
o Clustering: This approach groups the similar data in a cluster. The outliers may be
undetected or it will fall outside the clusters.
Data Integration
Data integration is the process of combining data from multiple sources into a cohesive and
consistent view. This process involves identifying and accessing the different data sources, mapping
the data to a common format, and reconciling any inconsistencies or discrepancies between the
sources. The goal of data integration is to make it easier to access and analyze data that is spread
across multiple systems or platforms, in order to gain a more complete and accurate understanding
of the data.
There are some problems to be considered during data integration.
• Schema integration: Integrates metadata(a set of data that describes other data) from
different sources.
• Entity identification problem: Identifying entities from multiple databases. For example, the
system or the user should know the student id of one database and studentname of another
database belonging to the same entity.
• Detecting and resolving data value concepts: The data taken from different databases while
merging may differ. The attribute values from one database may differ from another
database. For example, the date format may differ, like “MM/DD/YYYY” or “DD/MM/YYYY”.
Steps in Schema Integration
The process of schema integration involves several steps:
1. Identify the source schemas: The first step in schema integration is to identify the schemas
of the databases or data sources that need to be integrated.
2. Analyze the source schemas: Once the source schemas have been identified, they should be
analyzed to identify common attributes and data structures that can be used to integrate the
data.
3. Define the target schema: The target schema is the schema that will be used to represent
the integrated data. The target schema should be designed to support the requirements of
the application or task for which the data will be used.
4. Map the source schemas to the target schema: The next step in schema integration is to
map the attributes and data structures from the source schemas to the target schema. This
involves identifying the common attributes and creating mappings between the source and
target schema.
5. Merge the schemas: Once the source schemas have been mapped to the target schema, the
schemas can be merged to create a single schema that represents the integrated data.
6. Resolve conflicts: Inevitably, conflicts will arise during the schema integration process, such
as data type conflicts, naming conflicts, or conflicts in data models. These conflicts must be
resolved to ensure the integrity of the integrated data.
7. Test the integrated schema: The final step in schema integration is to test the integrated
schema to ensure that it meets the requirements of the application or task for which the
data will be used.
Conflicts in Schema Integration
Naming conflict
Naming conflicts are of two types synonyms and homonyms. A synonym occurs when two schemas
use different names to describe the same concept, for example, an entity type CUSTOMER in one
schema may describe an entity type CLIENT in another schema. A homonym occurs when two
schemas use the same name to describe different concepts. For example, an entity type Classes may
represent TRAIN classes in one schema and AEROPLANE classes in another schema.
Type conflicts
A similar concept may be represented in two schemas by different modeling constructs. For example,
DEPARTMENT may be an entity type in one schema and an attribute in another.
Domain conflicts
A single attribute may have different domains in different schemas. For example, we may declare Ssn
as an integer in one schema and a character string in another. A conflict of the unit of measure could
occur if one schema represented weight in pounds and the other used kgs.
Conflicts among constraints
Two schemas may impose different constraints, for example, the KEY of an entity type may be
different in each schema.
Data Transformation
This step involves modifying the data to make it more suitable for analysis or modeling.
Common transformations include:
Normalization: It is the method of scaling the data so that it can be represented in a smaller range
like -1.0 to 1.0 or or 0.0 to 1.0.
When different features have vastly different ranges , Data Analysis / Machine learning models may
give more importance to features with larger values, which can lead to biased [Link]
is essential in machine learning and data analysis because it ensures fair feature weighting, faster
convergence in algorithms, and improved model accuracy. The choice of method depends on the
dataset and the machine learning model being used.
standardization: A process of converting data from various sources into a consistent, uniform format,
typically by calculating the mean and standard deviation of each feature and then scaling the data so
that the new mean is 0 and the standard deviation is 1
Difference between normalization and standardization is that normalization Rescales values to a
specific range, often between 0 and 1 and Data standardization Rescales values so they have a mean
of 0 and a standard deviation of 1
Encoding categorical variables: the process of converting categorical or textual data into numerical
format, so that it can be used as input for algorithms to process. The reason for encoding is that most
machine learning algorithms work with numbers and not with text or categorical variables.
Feature engineering: Feature Engineering is the process of transforming raw data into meaningful
features that improve the performance of machine learning models. It involves creating, selecting,
modifying, or combining features to enhance predictive power.
Handling skewed data: Skewed data is when the distribution of a dataset is asymmetric, meaning
most values are concentrated on one side. To improving model accuracy, we apply transformations
like log or square root to make the distribution more normal.
Discretization: Data discretization refers to a method of converting a huge number of data values
into smaller ones so that the evaluation and management of data become easy. In other words, data
discretization is a method of converting attributes values of continuous data into a finite set of
intervals with minimum data loss. Discretization reduces the data size. For example, rather than
specifying the class time, we can set an interval like (3 pm-5 pm, or 6 pm-8 pm).
Aggregation: Aggregation is the process of finding, collecting, and presenting the data in a
summarized format to perform statistical. E.g Finding the average age of customer buying a
particular product which can help in finding out the targeted age group for that particular product.
Data Reduction
Data reduction is a crucial step in the data mining process that involves reducing the size of the
dataset while preserving the important information. This is done to improve the efficiency of data
analysis and to avoid overfitting of the model. Some common steps involved in data reduction are:
Feature Selection: This involves selecting a subset of relevant features from the dataset. Feature
selection is often performed to remove irrelevant or redundant features from the dataset.
Feature Extraction: This involves transforming the data into a lower-dimensional space while
preserving the important information. Feature extraction is often used when the original features are
high-dimensional and complex.
Sampling: This involves selecting a subset of data points from the dataset. Sampling is often used to
reduce the size of the dataset while preserving the important information. It can be done using
techniques such as random sampling, stratified sampling, and systematic sampling.
Clustering: This involves grouping similar data points together into clusters. Clustering is often used
to reduce the size of the dataset by replacing similar data points with a representative centroid. It
can be done using techniques such as k-means, hierarchical clustering, and density-based clustering.
Compression: This involves compressing the dataset while preserving the important information.
Compression is often used to reduce the size of the dataset for storage and transmission purposes. It
can be done using techniques such as wavelet compression, JPEG compression, and gif compression.
Concept Hierarchies
Concept hierarchy refers to the organization of data into a tree-like structure, where each level of the
hierarchy represents a concept that is more general than the level below it. This hierarchical
organization of data allows for more efficient and effective data analysis, as well as the ability to drill
down to more specific levels of detail when needed. The concept of hierarchy is used to organize and
classify data in a way that makes it more understandable and easier to analyze.
Types of Concept Hierarchies
1. Schema Hierarchy: Schema Hierarchy is a type of concept hierarchy that is used to organize the
schema of a database in a logical and meaningful way, grouping similar objects together. A
schema hierarchy can be used to organize different types of data, such as tables, attributes, and
relationships, in a logical and meaningful way. This can be useful in data warehousing, where
data from multiple sources needs to be integrated into a single database.
Example : Levels of Schema Hierarchy
External Schema (View Level)
What: The highest level that defines how individual users or applications see the data.
Purpose: Customizes views for different user needs, ensuring security and simplicity.
Conceptual Schema (Logical Level)
What: The middle level that provides a community user view of the entire database.
Purpose: Describes the structure of the whole database independently of physical
considerations.
Internal Schema (Physical Level)
What: The lowest level that describes the physical storage of the database.
Purpose: Deals with file structures, indexing, partitioning, and performance.
2. Set-Grouping Hierarchy: Set-Grouping Hierarchy is a type of concept hierarchy that is based on
set theory, where each set in the hierarchy is defined in terms of its membership in other sets.
Set-grouping hierarchy can be used for data cleaning, data pre-processing and data integration.
This type of hierarchy can be used to identify and remove outliers, noise, or inconsistencies from
the data and to integrate data from multiple sources.
Example : For a course dataset with fields like course id, course name, department, difficulty
level.
Entities are categorized by shared properties, such as department, category, or function.
We define two hierarchical structures:
Department-Based Hierarchy – Courses grouped under departments.
Difficulty-Based Hierarchy – Courses grouped by difficulty level.
3. Operation-Derived Hierarchy: An Operation-Derived Hierarchy is a type of concept hierarchy
that is used to organize data by applying a series of operations or transformations to the data.
The operations are applied in a top-down fashion, with each level of the hierarchy representing a
more general or abstract view of the data than the level below it. This type of hierarchy is
typically used in data mining tasks such as clustering and dimensionality reduction. The
operations applied can be mathematical or statistical operations such as aggregation,
normalization
Example: For a sales dataset with data like sale id, sales person, region, revenue levels are
determined based on computed values like SUM, COUNT, AVG to categorize data dynamically.
Top-Level: Company (sum of all sales).
Mid-Level: Regions (sum of all sales within a region).
Bottom-Level: Individual salespeople.
4. Rule-based Hierarchy: Rule-based Hierarchy is a type of concept hierarchy that is used to
organize data by applying a set of rules or conditions to the data. This type of hierarchy is useful
in data mining tasks such as classification, decision-making, and data exploration. It allows to the
assignment of a class label or decision to each data point based on its characteristics and
identifies patterns and relationships between different attributes of the data.
Examples
Organizational Structures (Employee Reporting)
Product Categories (e.g., Electronics → Mobile → Smartphone)
Financial Reporting (Accounts Classification)
Role-Based Access Control (User Permissions in a System)