Unit III: Data Preparation for Analysis

Data Cleaning

Data preparation is the process of making raw data ready for after processing and analysis. The key
methods are to collect, clean, and label raw data in a format suitable for machine learning (ML)
algorithms, followed by data exploration and visualization. The process of cleaning and combining raw
data before using it for machine learning and business analysis is known as data preparation, or
sometimes "pre-processing." But it may not be the most attractive of duties, careful data preparation is
essential to the success of data analytics. Clear and important ideas from raw data require careful
validation, cleaning, and an addition. Any business analysis or model created will only be as strong and
validating as the very first information preparation.

Data preparation is important because:

• It improves data quality

• Enhance model performance
• Saves time and resources
• Facilitates feature engineering
Data preparation process

Step 1: Describe purpose and requirements.

Step 2: Data Collection - Collecting information from a variety of sources, including files, databases,
websites, and social media, to conduct a thorough analysis, providing the usage of reliable and high-
quality data.

Step 3: Data combining and integrating – Data integration requires combining data from multiple
sources or dimensions in order to create a full, logical dataset. Data integration solutions provide a wide
range of operations, including combination, relationship, connection, difference, and join, as well as a
variety of data schemas and types of architecture.

Step 4: Data profiling – Systematic method for assessing and analysing a dataset, making sure its quality,
structure, content, and improving accuracy within an organizational context.
Step 5: Data exploring – Familiar with data, identifying patterns, trends, outliers, and errors in order to
better understand it and evaluate the possibilities for analysis.
Step 6: Data transformations and enrichment - The process of improving a dataset by adding new
features or columns, enhancing its accuracy and reliability, and verifying it against third-party sources.

Step 7: Data validation – For ensuring data accuracy, completeness, and constancy, as it checks data
against predefined rules and criteria that align with your requirements, standards, and regulations.

Handling Missing Data

Missing values are a common issue in data analysis. This occurs when a particular variable lacks data
points, resulting in incomplete information and potentially harming the accuracy and dependability of
your models. It is essential to address missing values efficiently to ensure strong and impartial results in
models.
Missing values are data points that are absent for a specific variable in a dataset. They can be represented
as blank cells, null values, or special symbols. These missing data points pose a significant challenge in
data analysis and can lead to inaccurate or biased results.

Missing values can pose a significant challenge in data analysis, as they can:

• Reduce the sample size: This can decrease the accuracy and reliability of your analysis.

• Introduce bias: If the missing data is not handled properly, it can bias the results of your
analysis.

• Make it difficult to perform certain analyses: Some statistical techniques require complete
data for all variables, making them inapplicable when missing values are present

Data can be missing for many reasons like technical issues, human errors, privacy concerns, data
processing issues, or the nature of the variable itself. Understanding the cause of missing data helps
choose appropriate handling strategies and ensure the quality of your analysis.

It’s important to understand the reasons behind missing data:

• Identifying the type of missing data

• Evaluating the impact of missing data

• Choosing appropriate handling strategies

Handling Noisy Data

Handling noisy data is a critical aspect of preparing high-quality datasets for data analysis. Noisy data
can lead to inaccurate models and poor performance. below are some steps and techniques to manage
noisy data effectively.

• Noise identification: Use data visualization tools like histograms, scatter plots, and box plots to
detect outliers or anomalies in the dataset.
• Data Cleaning: The cleaning process involves correcting errors, removing duplicates, and
dealing with missing values. Data cleaning is a delicate balance, to retain as much useful
information as possible without compromising the integrity of dataset
o Correcting Errors: Identify and correct errors in your data. This can involve fixing
typos, ensuring consistent formatting, and validating data against known standards or
rules.
o Removing Duplicates: Removing duplicate records can help reduce noise and
redundancy in your dataset.
o Dealing with Missing Values: Techniques such as imputation can fill in missing data,
while others may require removal if they’re deemed too noisy or irrelevant.
o Imputation: Fill in missing values using strategies such as mean, median, mode, or more
sophisticated methods like K-Nearest Neighbours (KNN) imputation.
o Removal: Remove rows or columns with a significant amount of missing data if they
cannot be reliably imputed.
o Smoothing Techniques: For continuous data, smoothing techniques such as moving
averages, exponential smoothing, or applying filters can help reduce noise. These
techniques can help smooth out short-term fluctuations and highlight longer-term trends
or cycles.
 o Transformations: Transformations such as logarithmic or square root transformations
can stabilize variance and make the data more closely meet the assumptions of parametric
statistical tests.

Removing Outliers

An Outlier is a data item/object that deviates significantly from the rest of the (so-called normal) objects.
Identifying outliers is important in statistics and data analysis because they can have a significant impact
on the results of statistical analyses. The analysis for outlier detection is referred to as outlier mining.

Outliers can skew the mean (average) and affect measures of central tendency, as well as influence the
results of tests of statistical significance.

Outliers can be caused by a variety of factors, and they often result from genuine variability in the data
or from errors in data collection, measurement, or recording. Some common causes of outliers are:
• Measurement errors: Errors in data collection or measurement processes can lead to outliers.

• Sampling errors: In some cases, outliers can arise due to issues with the sampling process.

• Natural variability: Inherent variability in certain phenomena can also lead to outliers. Some
systems may exhibit extreme values due to the nature of the process being studied.

• Data entry errors: Human errors during data entry can introduce outliers.

• Experimental errors: In experimental settings, anomalies may occur due to uncontrolled

factors, equipment malfunctions, or unexpected events.

• Sampling from multiple populations: Data is inadvertently combined from multiple

populations with different characteristics.

• Intentional outliers: Outliers are introduced intentionally to test the robustness of statistical
methods.
Outliers can be detected using visualization, implementing mathematical formulas on the dataset, or
using the statistical approach.

Visualizing and Removing Outliers Using Box Plot

It captures the summary of the data effectively and efficiently with only a simple box and
whiskers. Boxplot summarizes sample data using 25th, 50th, and 75th percentiles. One can just get
insights (quartiles, median, and outliers) into the dataset by just looking at its boxplot.
In the above graph, can clearly see that values above 10 are acting as outliers. Remove it and then

Visualizing and Removing Outliers Using Scatterplot

It is used when you have paired numerical data and when your dependent variable has multiple values
for each reading independent variable, or when trying to determine the relationship between the two
variables. In the process of utilizing the scatter plot, one can also use it for outlier detection.

Looking at the graph can summarize that most of the data points are in the bottom left corner of the
graph but there are few points that are exactly opposite that is the top right corner of the graph. Those
points in the top right corner can be regarded as Outliers.

Using approximation can say all those data points that are x>20 and y>600 are outliers. The following
code can fetch the exact position of all those points that satisfy these conditions.
IQR (Inter Quartile Range)

IQR (Inter Quartile Range) Inter Quartile Range approach to finding the outliers is the most commonly
used and most trusted approach used in the research field.

IQR = Quartile3 – Quartile1

Data Integration

Data integration is the process of combining data from different sources into a single, unified view. Ie.,
the process of moving data between databases – internal, external, or both. This can involve cleaning
and transforming the data, as well as resolving any inconsistencies or conflicts that may exist between
the different sources. The goal of data integration is to make the data more useful and meaningful for
the purposes of analysis and decision making.

Data Integration is a data preprocessing technique that combines data from multiple heterogeneous data
sources into a coherent data store and provides a unified view of the data. These sources may include
multiple data cubes, databases, or flat files.
The data integration approaches are formally defined as triple <G, S, M>

where, G stand for the global schema, S stands for the heterogeneous source of schema, M stands for
mapping between the queries of source and global schema.
Data integration can be challenging due to the variety of data formats, structures, and semantics used by
different data sources. Different data sources may use different data types, naming conventions, and
schemas, making it difficult to combine the data into a single view. Data integration typically involves a
combination of manual and automated processes, including data profiling, data mapping, data
transformation, and data reconciliation.

Data integration is used in a wide range of applications, such as business intelligence, data warehousing,
master data management, and analytics. Data integration can be critical to the success of these
applications, as it enables organizations to access and analyse data that is spread across different systems,
departments, and lines of business, in order to make better decisions, improve operational efficiency,
and gain a competitive advantage.

There are mainly 2 major approaches for data integration – one is the “tight coupling approach” and
another is the “loose coupling approach”.

Tight Coupling:

This approach involves creating a centralized repository or data warehouse to store the integrated data.
The data is extracted from various sources, transformed and loaded into a data warehouse. Data is
integrated in a tightly coupled manner, meaning that the data is integrated at a high level, such as at the
level of the entire dataset or schema. This approach is also known as data warehousing, and it enables
data consistency and integrity, but it can be inflexible and difficult to change or update.

• Here, a data warehouse is treated as an information retrieval component.

• In this coupling, data is combined from different sources into a single physical location through
the process of ETL – Extraction, Transformation, and Loading.
Loose Coupling:

This approach involves integrating data at the lowest level, such as at the level of individual data
elements or records. Data is integrated in a loosely coupled manner, meaning that the data is integrated
at a low level, and it allows data to be integrated without having to create a central repository or data
warehouse. This approach is also known as data federation, and it enables data flexibility and easy
updates, but it can be difficult to maintain consistency and integrity across multiple data sources.
• Here, an interface is provided that takes the query from the user, transforms it in a way the source
database can understand, and then sends the query directly to the source databases to obtain the
result.

• And the data only remains in the actual source databases.

Issues in Data Integration:

There are several issues that can arise when integrating data from multiple sources, including:

1. Data Quality: Inconsistencies and errors in the data can make it difficult to combine and analyse.

2. Data Semantics: Different sources may use different terms or definitions for the same data,
making it difficult to combine and understand the data.

3. Data Heterogeneity: Different sources may use different data formats, structures, or schemas,
making it difficult to combine and analyse the data.

4. Data Privacy and Security: Protecting sensitive information and maintaining security can be
difficult when integrating data from multiple sources.

5. Scalability: Integrating large amounts of data from multiple sources can be computationally
expensive and time-consuming.
6. Data Governance: Managing and maintaining the integration of data from multiple sources can
be difficult, especially when it comes to ensuring data accuracy, consistency, and timeliness.
7. Performance: Integrating data from multiple sources can also affect the performance of the
system.

8. Integration with existing systems: Integrating new data sources with existing systems can be a
complex task, requiring significant effort and resources.

9. Complexity: The complexity of integrating data from multiple sources can be high, requiring
specialized skills and knowledge.

There are three issues to consider during data integration: Schema Integration, Redundancy Detection,
and resolution of data value conflicts. These are explained in brief below.

1. Schema Integration:
• Integrate metadata from different sources.

• The real-world entities from multiple sources are referred to as the entity identification
problem.
 2. Redundancy Detection:

• An attribute may be redundant if it can be derived or obtained from another attribute or set
of attributes.

• Inconsistencies in attributes can also cause redundancies in the resulting data set.

• Some redundancies can be detected by correlation analysis.

3. Resolution of data value conflicts:

• This is the third critical issue in data integration.

• Attribute values from different sources may differ for the same real-world entity.

• An attribute in one system may be recorded at a lower level of abstraction than the “same”
attribute in another.

Data Transformation: Standardization, Normalization

The process of data transformation, involves converting, cleansing, and structuring data into a
usable format which is used to analysed to support decision-making processes. It includes modifying the
format, organization, or values of data to prepare it for consumption by an application or for analysis.
This crucial process is undertaken by organizations seeking to leverage their data to provide timely
business insights, ensuring that the information is accessible, consistent, safe, and eventually
acknowledged by the targeted business users.

The transformations can be divided into two categories: Simple transformations and Complex
Data Transformations.

• Simple Data Transformations include straightforward procedures including data cleansing,

standardization, aggregation, and filtering. These transformations are often carried out utilizing
simple data manipulation methods and are frequently used to prepare data for analysis or
reporting.

• Complex Data Transformations include more advanced processes such data integration,
migration, replication, and enrichment. These transformations often need complex data
manipulation methods like as data modelling, mapping, and validation, and are commonly used
to prepare data for advanced analytics, machine learning, or data warehousing applications.

Importance of Data Transformation

Data transformation is important because it improves data quality, compatibility, and utility.
The procedure is critical for companies and organizations that depend on data to make informed
decisions because it assures the data's accuracy, reliability, and accessibility across many systems and
applications.
• Improved Data Quality: Data transformation eliminates mistakes, inserts in missing
information, and standardizes formats, resulting in higher-quality, more dependable, and accurate
data.

• Enhanced Compatibility: By converting data into a suitable format, companies may avoid
possible compatibility difficulties when integrating data from many sources or systems.
 • Simplified Data Management: Data transformation is the process of evaluating and modifying
data to maximize storage and discoverability, making it simpler to manage and maintain.

• Broader Application: Transformed data is more useable and applicable in a larger variety of
scenarios, allowing enterprises to get the most out of their data.

• Faster Queries: By standardizing data and appropriately storing it in a warehouse, query

performance and BI tools may be enhanced, resulting in less friction during analysis.

Key Data Transformation Operations for Effective Analysis

Various methods of data transformation exist, depending on the complexity and nature of the process:

1. Normalization: Modifying data scales, such as scaling values from 0 to 1, to enable

comparisons.

2. Standardization: Transforming data to have a unit variance and zero mean, which is frequently
required before using machine learning methods.

3. Encoding: Transforming categorical data into numerical representations using label or one-hot
encoding.

4. Discretization: Converting continuous data into discrete bins, which in some circumstances can
facilitate analysis and enhance model performance.

5. Attribute Generation: Creating new variables from existing data, such as deriving an 'age'
variable from a date of birth.

6. Revising: Ensuring that the data supports its intended usage by deleting duplicates, standardizing
the data collection, and purifying it.

7. Manipulation: Creating new values from existing ones or changing the state of data
through computing.
8. Separating: Splitting down data values into component for filtering on certain values.

9. Combining/Integrating: Bringing together data from several tables and sources to provide a
comprehensive picture of an organization.

10. Binning or Discretization: Continuous data can be grouped into discrete categories, which is
helpful for managing noisy data.

11. Smoothing: Methods like moving averages can be applied to reduce noise in time series or create
smoothed data.

12. Logarithmic or Exponential Transformation: Data distribution can be altered through

logarithmic or exponential functions for specialized analyses.

13. Text Preprocessing: Text data can be prepared for NLP (Natural language processing) tasks by
tokenizing, stemming, or lemmatizing.
Data Transformation Techniques and Tools

There are several ways to alter data, including:

1. Programmatic Transformation: automating the transformation operations via the use of scripts
or computer languages such as Python, R, or SQL.
 2. ETL Tools: Tools for extracting, transforming, and loading data (ETL) are made to address
complicated data transformation requirements in large-scale settings. After transforming the data
to meet operational requirements, they extract it from several sources and load it into a
destination like a database or data warehouse.
3. Normalization/Standardization: Scikit-learn in Python provides functions for normalization
and standardization such as MinMaxScaler and StandardScaler.
4. Encoding Categorical variables: Pandas library in python provides get_dummies function
employed for one-hot encoding. For label encoding LabelEncoder is provided by Scikit-learn.
5. Imputation: Missing values in the dataset are filled using statistical methods like fillna method
in Pandas Library. Additionally, missing data can be imputed using mean, median, or mode
using scikit-learn's SimpleImputer.

6. Feature Engineering: To improve model performance, new features are developed by

combining old ones. Pandas, a Python library, is often used to execute feature engineering
tasks. Functions such as apply, map, and transform are used to generate new features.

7. Aggregation and grouping: Pandas groupby function is used to group data and execute
aggregation operations such as sum, mean, and count.

8. Text preprocessing: Textual data is preprocessed by tokenizing, stemming, and eliminating stop
words using NLTK and SpaCy Python libraries.

9. Dimensional Reduction: The technique involves reducing the number of characteristics while
retaining vital information. Scikit-learn in Python provides techniques such as PCA for Principal
Component Analysis and TruncatedSVD for Dimensionality Reduction.
Advantages and Limitations of Data Transformation

Advantages of Data Transformation

• Enhanced Data Quality: Data transformation aids in the organisation and cleaning of data,
improving its quality.

• Compatibility: It guarantees data consistency between many platforms and systems, which is
necessary for integrated business environments.

• Improved Analysis: Analytical results that are more accurate and perceptive are frequently the
outcome of transformed data.

Limitations of Data Transformation

• Complexity: When working with big or varied datasets, the procedure might be laborious and
complicated.

• Cost: The resources and tools needed for efficient data transformation might be expensive.
• Risk of Data Loss: Inadequate transformations may cause important data to be lost or distorted.

Applications of Data Transformation

Applications for data transformation are found in a number of industries:

 • Business intelligence (BI) is the process of transforming data for use in real-time reporting and
decision-making using BI technologies.

• Healthcare: Ensuring interoperability across various healthcare systems by standardization of

medical records.

• Financial Services: Compiling and de-identifying financial information for reporting and
compliance needs.

• Retail: Improving customer experience through data transformation into an analytics-ready

format and customer behaviour analysis.

• Customer Relationship Management (CRM): By converting customer data, firms may obtain
insights into consumer behaviour, tailor marketing strategies, and increase customer satisfaction.

Standardization

In Machine Learning we train our data to predict or classify things in such a manner that isn’t hardcoded
in the machine. So, for the first, we have the Dataset or the input data to be pre-processed and
manipulated for our desired outcomes. Any ML Model to be built follows the following procedure:

• Collect Data

• Perform Data Munging/Cleaning (Feature Scaling)

• Pre-Process Data

• Apply Visualizations

Feature Scaling is a method to standardize the features present in the data in a fixed range. It has to
perform during the data pre-processing. It has two main ways: Standardization and Normalization.

Standardization

The steps to be followed are:

Data collection: Our data can be in various formats i.e., numbers (integers) & words (strings), for now,
we’ll consider only the numbers in our Dataset.

Assume our dataset has random numeric values in the range of 1 to 95,000 (in random order). Just for
our understanding consider a small Dataset of barely 10 values with numbers in the given range and
randomized order.

1) 99

2) 789

3) 1
4) 541
5) 5

6) 6589

7) 94142

8) 7
9) 50826

10) 35464

If we just look at these values, their range is so high, that while training the model with 10,000 such
values will take lot of time. That’s where the problem arises.

Understanding standardization

We have a solution to solve the problem arisen i.e. Standardization. It helps us solve this by:
• Down Scaling the Values to a scale common to all, usually in the range -1 to +1.

• And keeping the Range between the values intact.

So, how do we do that? we’ll there’s a mathematical formula for the same i.e., Z-Score = (Current_value
– Mean) / Standard Deviation.

Standardization Formula

Using this formula, we are replacing all the input values by the Z-Score for each and every value. Hence,
we get values ranging from -1 to +1, keeping the range intact.

Standardization performs the following:

• Converts the Mean (μ) to 0

• Converts to S.D. (σ) to 1

It’s pretty obvious for Mean = 0 and S.D = 1 as all the values will have such less difference and each
value will nearly be equal 0, hence Mean = 0 and S.D. = 1.

NOTE: (Just for Better Understanding)

For Mean
When we Subtract a value Smaller than the Mean we get (-ve) Output
When we Subtract a value Larger than the Mean we get (+ve) Output
Hence, when we get (-ve) & (+ve) Values for Subtraction of Value with Mean, while Summation of all
these values,
We get the Final Mean as 0.

And when we get the Mean as 0, it means that most or nearly all values are equal to highly close to 0
and have very low variance among them.

Therefore, the S.D also becomes 1 (as good as no difference).

Normalization

Data normalization is a technique used in data mining to transform the values of a dataset into a common
scale. This is important because many machine learning algorithms are sensitive to the scale of the input
features and can produce better results when the data is normalized.
There are several different normalization techniques that can be used in data mining, including:

1. Min-Max normalization: This technique scales the values of a feature to a range between 0 and
1. This is done by subtracting the minimum value of the feature from each value, and then
dividing by the range of the feature.

2. Z-score normalization: This technique scales the values of a feature to have a mean of 0 and a
standard deviation of 1. This is done by subtracting the mean of the feature from each value, and
then dividing by the standard deviation.

3. Decimal Scaling: This technique scales the values of a feature by dividing the values of a feature
by a power of 10.

4. Logarithmic transformation: This technique applies a logarithmic transformation to the values

of a feature. This can be useful for data with a wide range of values, as it can help to reduce the
impact of outliers.

5. Root transformation: This technique applies a square root transformation to the values of a
feature. This can be useful for data with a wide range of values, as it can help to reduce the impact
of outliers.

6. It’s important to note that normalization should be applied only to the input features, not the
target variable, and that different normalization technique may work better for different types of
data and models.

In conclusion, normalization is an important step in data mining, as it can help to improve the
performance of machine learning algorithms by scaling the input features to a common scale. This can
help to reduce the impact of outliers and improve the accuracy of the model.
Normalization is used to scale the data of an attribute so that it falls in a smaller range, such as -1.0 to
1.0 or 0.0 to 1.0. It is generally useful for classification algorithms.
Need of Normalization –

Normalization is generally required when we are dealing with attributes on a different scale, otherwise,
it may lead to a dilution in effectiveness of an important equally important attribute (on lower scale)
because of other attribute having values on larger scale. In simple words, when multiple attributes are
there but attributes have values on different scales, this may lead to poor data models while performing
data mining operations. So, they are normalized to bring all the attributes on the same scale.
Methods of Data Normalization –

• Decimal Scaling

• Min-Max Normalization

• z-Score Normalization (zero-mean Normalization)

Decimal Scaling Method for Normalization –

It normalizes by moving the decimal point of values of the data. To normalize the data by this technique,
we divide each value of the data by the maximum absolute value of data. The data value, vi, of data is

normalized to vi‘ by using the formula below – where j is the smallest integer
such that max(|vi‘|)<1. Example –

Let the input data is: -10, 201, 301, -401, 501, 601, 701 To normalize the above data, Step 1: Maximum
absolute value in given data(m): 701 Step 2: Divide the given data by 1000 (i.e j=3) Result: The
normalized data is: -0.01, 0.201, 0.301, -0.401, 0.501, 0.601, 0.701

Min-Max Normalization –
In this technique of data normalization, linear transformation is performed on the original data. Minimum
and maximum value from data is fetched and each value is replaced according to the following formula.

Where A is the attribute data, Min(A), Max(A) are the minimum and maximum absolute value of A
respectively. v’ is the new value of each entry in data. v is the old value of each entry in data.
new_max(A), new_min(A) is the max and min value of the range(i.e boundary value of range required)
respectively.

Z-score normalization –

In this technique, values are normalized based on mean and standard deviation of the data A. The formula
used is:v’, v is the new and old of each entry in data respectively. σA, A is the standard deviation and
mean of A respectively.

Advantages:

1. Improved performance of machine learning algorithms: Normalization can help to improve the
performance of machine learning algorithms by scaling the input features to a common scale.
This can help to reduce the impact of outliers and improve the accuracy of the model.

2. Better handling of outliers: Normalization can help to reduce the impact of outliers by scaling
the data to a common scale, which can make the outliers less influential.
 3. Improved interpretability of results: Normalization can make it easier to interpret the results of a
machine learning model, as the inputs will be on a common scale.

4. Better generalization: Normalization can help to improve the generalization of a model, by

reducing the impact of outliers and by making the model less sensitive to the scale of the inputs.

Disadvantages:

1. Loss of information: Normalization can result in a loss of information if the original scale of the
input features is important.

2. Impact on outliers: Normalization can make it harder to detect outliers as they will be scaled
along with the rest of the data.

3. Impact on interpretability: Normalization can make it harder to interpret the results of a machine
learning model, as the inputs will be on a common scale, which may not align with the original
scale of the data.

4. Additional computational costs: Normalization can add additional computational costs to the data
mining process, as it requires additional processing time to scale the data.

5. In conclusion, data normalization can have both advantages and disadvantages. It can improve
the performance of machine learning algorithms and make it easier to interpret the results.
However, it can also result in a loss of information and make it harder to detect outliers. It’s
important to weigh the pros and cons of data normalization and carefully assess the risks and
benefits before implementing it.

Data Reduction: Dimensionality Reduction – Principal Component Analysis

The method of data reduction may achieve a condensed description of the original data which is much
smaller in quantity but keeps the quality of the original data. Data reduction is a technique used in data
mining to reduce the size of a dataset while still preserving the most important information. This can be
beneficial in situations where the dataset is too large to be processed efficiently, or where the dataset
contains a large amount of irrelevant or redundant information.
There are several different data reduction techniques that can be used in data mining, including:

1. Data Sampling: This technique involves selecting a subset of the data to work with, rather than
using the entire dataset. This can be useful for reducing the size of a dataset while still preserving
the overall trends and patterns in the data.
2. Dimensionality Reduction: This technique involves reducing the number of features in the
dataset, either by removing features that are not relevant or by combining multiple features into
a single feature.

3. Data Compression: This technique involves using techniques such as lossy or lossless
compression to reduce the size of a dataset.

4. Data Discretization: This technique involves converting continuous data into discrete data by
partitioning the range of possible values into intervals or bins.

5. Feature Selection: This technique involves selecting a subset of features from the dataset that
are most relevant to the task at hand.
It’s important to note that data reduction can have a trade-off between the accuracy and the size of the
data. The more data is reduced, the less accurate the model will be and the less generalizable it will be.

In conclusion, data reduction is an important step in data mining, as it can help to improve the efficiency
and performance of machine learning algorithms by reducing the size of the dataset. However, it is
important to be aware of the trade-off between the size and accuracy of the data, and carefully assess the
risks and benefits before implementing it.

Methods of data reduction: These are explained as following below.

1. Data Cube Aggregation: This technique is used to aggregate data in a simpler form. For example,
imagine the information you gathered for your analysis for the years 2012 to 2014, that data includes the
revenue of your company every three months. They involve you in the annual sales, rather than the
quarterly average, so we can summarize the data in such a way that the resulting data summarizes the
total sales per year instead of per quarter. It summarizes the data.

2. Dimension reduction: Whenever we come across any data which is weakly important, then we use
the attribute required for our analysis. It reduces data size as it eliminates outdated or redundant features.

• Step-wise Forward Selection: The selection begins with an empty set of attributes later on we
decide the best of the original attributes on the set based on their relevance to other attributes.
We know it as a p-value in statistics.
Suppose there are the following attributes in the data set in which few attributes are redundant.

Initial attribute Set: {X1, X2, X3, X4, X5, X6}

Initial reduced attribute set: { }

Step-1: {X1}

Step-2: {X1, X2}

Step-3: {X1, X2, X5}

Final reduced attribute set: {X1, X2, X5}

• Step-wise Backward Selection: This selection starts with a set of complete attributes in the
original data and at each point, it eliminates the worst remaining attribute in the set.

Suppose there are the following attributes in the data set in which few attributes are redundant.

Initial attribute Set: {X1, X2, X3, X4, X5, X6}

Initial reduced attribute set: {X1, X2, X3, X4, X5, X6}

Step-1: {X1, X2, X3, X4, X5}

Step-2: {X1, X2, X3, X5}

Step-3: {X1, X2, X5}
Final reduced attribute set: {X1, X2, X5}

• Combination of forwarding and Backward Selection: It allows us to remove the worst and
select the best attributes, saving time and making the process faster.

3. Data Compression: The data compression technique reduces the size of the files using different
encoding mechanisms (Huffman Encoding & run-length Encoding). We can divide it into two types
based on their compression techniques.

• Lossless Compression: Encoding techniques (Run Length Encoding) allow a simple and
minimal data size reduction. Lossless data compression uses algorithms to restore the precise
original data from the compressed data.

• Lossy Compression: Methods such as the Discrete Wavelet transform technique, PCA (principal
component analysis) are examples of this compression. For e.g., the JPEG image format is a
lossy compression, but we can find the meaning equivalent to the original image. In lossy-data
compression, the decompressed data may differ from the original data but are useful enough to
retrieve information from them.

4. Numerosity Reduction: In this reduction technique, the actual data is replaced with mathematical
models or smaller representations of the data instead of actual data, it is important to only store the model
parameter. Or non-parametric methods such as clustering, histogram, and sampling.

5. Discretization & Concept Hierarchy Operation: Techniques of data discretization are used to
divide the attributes of the continuous nature into data with intervals. We replace many constant values
of the attributes by labels of small intervals. This means that mining results are shown in a concise, and
easily understandable way.

• Top-down discretization: If you first consider one or a couple of points (so-called breakpoints
or split points) to divide the whole set of attributes and repeat this method up to the end, then the
process is known as top-down discretization also known as splitting.

• Bottom-up discretization: If you first consider all the constant values as split points, some are
discarded through a combination of the neighbourhood values in the interval, that process is
called bottom-up discretization.

Concept Hierarchies: It reduces the data size by collecting and then replacing the low-level concepts
(such as 43 for age) with high-level concepts (categorical variables such as middle age or Senior).

For numeric data following techniques can be followed:

• Binning: Binning is the process of changing numerical variables into categorical counterparts.
The number of categorical counterparts depends on the number of bins specified by the user.

• Histogram analysis: Like the process of binning, the histogram is used to partition the value for
the attribute X, into disjoint ranges called brackets. There are several partitioning rules:

1. Equal Frequency partitioning: Partitioning the values based on their number of

occurrences in the data set.

2. Equal Width Partitioning: Partitioning the values in a fixed gap based on the number
of bins i.e. a set of values ranging from 0-20.
 3. Clustering: Grouping similar data together.

One of the most widely used dimensionality reduction techniques is Principal Component Analysis
(PCA).

Principal Component Analysis

PCA is a statistical technique introduced by mathematician Karl Pearson in 1901. It works by

transforming high-dimensional data into a lower-dimensional space while maximizing the variance (or
spread) of the data in the new space. This helps preserve the most important patterns and relationships
in the data.

It is a dimensionality reduction technique and helps us to reduce the number of features in a dataset
while keeping the most important information. It changes complex datasets by transforming correlated
features into a smaller set of uncorrelated components.

PCA uses linear algebra to transform data into new features called principal components. It finds these
by calculating eigenvectors (directions) and eigenvalues (importance) from the covariance matrix. PCA
selects the top components with the highest eigenvalues and projects the data onto them simplify the
dataset.

It prioritizes the directions where the data varies the most because more variation = more useful
information.

Step 1: Standardize the Data

Different features may have different units and scales like salary vs. age. To compare them fairly PCA
first standardizes the data by making each feature have:

• A mean of 0

• A standard deviation of 1

where:

• μ is the mean of independent features μ={μ1,μ2,⋯,μm}

• σ is the standard deviation of independent features σ={σ1,σ2,⋯,σm}

Step 2: Find Relationships

Next PCA calculates the covariance matrix to see how features relate to each other whether they increase
or decrease together. The covariance between two features x1 and x2 is:

Where:

The value of covariance can be positive, negative or zeros.

Step 3: Find the Principal Components

• PCA identifies new axes where the data spreads out the most:

o 1st Principal Component (PC1): The direction of maximum variance (most spread).

o 2nd Principal Component (PC2): The next best direction, perpendicular to PC1, and so
on.

• These directions are calculated using Eigenvalues and Eigenvectors where: eigenvectors (math
tools that find these axes), and their importance is ranked by eigenvalues (how much variance
each capture).

For a square matrix A, an eigenvector X (a non-zero vector) and its corresponding eigenvalue λ (a
scalar) satisfy:
AX=λX

This means:

• When A acts on X, it only stretches or shrinks X by the scalar λ.

• The direction of X remains unchanged (hence, eigenvectors define “stable directions” of A).

Eigenvalues help rank these directions by importance.

Step 4: Pick the Top Directions & Transform Data

After calculating the eigenvalues and eigenvectors PCA ranks them by the amount of information
they capture. We then:

1. Select the top k components that capture most of the variance like 95%.
2. Transform the original dataset by projecting it onto these top components.

This means we reduce the number of features (dimensions) while keeping the important patterns in
the data.
 In the above image the original dataset has two features "Radius" and "Area" represented by the
black axes. PCA identifies two new directions: PC₁ and PC₂ which are the principal components.

• These new axes are rotated versions of the original ones. PC₁ captures the maximum variance in
the data meaning it holds the most information while PC₂ captures the remaining variance and is
perpendicular to PC₁.

• The spread of data is much wider along PC₁ than along PC₂. This is why PC₁ is chosen for
dimensionality reduction. By projecting the data points (blue crosses) onto PC₁ we effectively
transform the 2D data into 1D and retain most of the important structure and patterns.