DATA PREPROCESSING:

Raw data collected from various sources is often unorganized, inconsistent, or incomplete. Data
preprocessing in machine learning ensures models are trained on a clean, consistent, and accurate
dataset. Data preprocessing is the process of cleaning and organizing the raw data to ensure accuracy
and consistency. In this blog, you’ll explore data preprocessing in data mining, why it’s important,
and the key steps involved in the process.

Raw data in the data science and data mining world is usually messy and disorganized. This raw data
often contains missing values, errors, duplicates, and other issues. One cannot train a model with
such data or make informed decisions based on it. Data preprocessing is the process of cleaning and
transforming raw data into a useful format. It is the process of activities such as removing the
unknown values, normalization, transforming, and sorting the data for a better understanding.

Uses of Data Preprocessing:

1. Machine Learning: Preprocessing the data is usually required before the data is trained by machine
learning algorithms. Data preprocessing is the last step to clean the data, which helps to make sure
that the data is in a consistent form.

2. Data Analysis: Data preprocessing provides some clean data to give a user better insight and,
therefore, better decision-making. If preprocessing is not done, the user actions may produce reports
or dashboards with results that have errors.

3. Business Intelligence (BI): Business Intelligence (BI) tools commonly rely on preprocessed data,
with BI tools (Power BI, Tableau, etc.) producing reports and visualizations on this processed data.
The user would expect that using a dashboard, report, or visualization at a later point in time is
reliable and correct.
4. Medical and Healthcare: In medical/healthcare applications, preprocessing ultimately makes
health records/medical history/lab results/measures clearer before statistical analysis, therefore
aiming to improve diagnosis, research, and output ultimately relating to better patient
care/improved disease treatment.

5. Finance and Banking: In finance, data preprocessing is useful in areas such as fraud detection, risk
management, and understanding one customer from another or a group of customers. It allows for
cleaning transaction data and standardizing entries.

Importance of Data Preprocessing in Data Mining:

1. Helps handle missing data.

2. Corrects Errors in Raw Data.

3. Ensures Consistent Data Formatting.

4. Improves Model Accuracy.

5. Reduces Bias and Noise

Types of Data in Data Preprocessing:

The data in data processing is mainly classified into:

1. Numerical Data

Numerical data refers to data that can be measured and expressed in numbers. It involves quantities
and is often used for mathematical calculations and analysis.

Example: Age in years, salary in rupees, temperature in degrees, and height in centimeters.

Numerical data is further divided into two types:

a) Discrete Data: Discrete data refers to whole numbers. These are values you can count one by one.
They are not fractions or decimals.

Example: Number of students in a classroom.

b) Continuous Data: Continuous data can consist of any value in a given range. It can consist of
fractions and decimal numbers. Continuous data is more likely to be measured than counted.

Example: Height of the students in a classroom.

2. Categorical Data

Categorical data refers to information that is grouped into categories or labels. These values
represent types or characteristics and are usually non-numeric. When needed, they can be converted
into numbers using encoding techniques for analysis.

Example: Gender (Male, Female), product type (Electronics, Furniture), department (HR, Marketing)

Categorical data is further divided into two types:

a) Nominal Data

Nominal data includes categories that have no specific order or ranking. The values are just labels or
names used to identify items.

Example: Blood type (A, B, AB, O), colors (red, blue, green), city names
b) Ordinal Data

Ordinal data consists of categories that have a defined order or ranking, but the differences between
the values are not measurable.

Example: Customer satisfaction levels (Poor, Fair, Good, Excellent), clothing sizes (S, M, L, XL)

3. Text Data

Text data consists of information in the form of words, sentences, or phrases. It is unstructured and
often used to capture opinions, descriptions, or messages. This type of data is common in areas like
reviews, feedback forms, and comments.

Examples: Customer reviews or feedback comments.

4. Date and Time

Date and time data includes information that represents specific dates, times, or a combination of
both. It helps track events, record activities, or schedule tasks in systems. This data type is important
in reporting, logging, and scheduling.

Examples: Order date, login time, and birthdate.

5. Boolean Data

Boolean data is a type of binary data that can take only two values: true or false. It is commonly used
to represent yes/no choices, system states, or logical decisions. This type of data plays a key role in
control flows and condition checks.

Example: IsActive: True, EmailVerified: False

Advantages and Disadvantages of Data Preprocessing

Data preprocessing is responsible for improving the quality of the data and the performance of the
model, but it can also lead to data loss if done incorrectly. Let’s explore the advantages and
disadvantages of applying data preprocessing techniques to the data.

Advantages of Data Preprocessing

Let’s explore the advantages of data preprocessing:

1. Enhances Data Quality: It assists in enhancing the quality of data because it eliminates errors and
corrects missing data.

2. Improved Model Performance: It assists in enhancing the model performance since it puts the
data in a well-organized and clean manner.

3. Quick Processing: It also assists in reducing the data size to make the processing faster.

4. Helps Data Easy to Read: Organized data is easy to understand and interpret, as well as present to
other people.

5. Makes Data Usable for Tools: To be effective in most applications, tools such as Excel, Power BI,
and machine learning libraries need well-prepared data.

Disadvantages of Data Preprocessing

Let’s explore the disadvantages of data preprocessing:

1. Time-Consuming: If you have large datasets, preprocessing can take a long time.

2. Required Knowledge: You should know a lot about data types, techniques, and tools if you want to
clean and process the data correctly.

3. Losing Data: During cleaning, you can lose useful data if you aren’t careful.

4. Possible Introduced Bias: If you aren’t careful in the preprocessing, you may alter the data enough
to alter the results.

5. Increase Complexity: Managing multiple steps of data preprocessing makes it more complex, as it
includes complex steps such as transforming, encoding, and scaling.

Steps Involved in Data Preprocessing

Follow these organized steps in data preprocessing to clean, transform, and prepare raw data for
analysis or modeling.

Step 1: Data Profiling

The initial step is collecting data from many different data sources. Data could come from many
sources, including databases, online surveys, sensors, or files. All data has a different structure and
could come from different systems.

 Sources: CSV files, SQL databases, APIs, cloud storage, web scraping.

 Goal: Align data in one location.

Step 2: Data Cleaning

After the data is captured, the next step is to clean the data. This is a very important step and
consists of.

 Removing Missing Values: Fill missing values using the mean, median, or drop the row if
necessary.
  Removing Duplicates: Remove duplicate entries to prevent bias.

 Correcting Errors: Fix misspellings or incorrectly entered numbers.

 Removing Outliers: Delete or replace values that are far out of the expected range.

Example: If a person’s age was entered as 500 years, this is definitely an error.

Step 3: Data Integration

Sometimes, data will come from different sources, like Excel sheets, SQL databases, and cloud apps.
This is known as data integration, which merges data from many datasets into one dataset.

 Remove Inconsistencies: Make sure that all values are spelled or grouped the same way.

 Record Merging: When you have data in different tables or files, merge based on shared keys
(ie, user ID).

Step 4: Data Transformation

This phase transforms the data into the appropriate form for analysis or machine learning.

 Normalization: Scale all data to a fixed range (for example, 0 – 1) in order to ensure that large
values do not have a disproportionate effect on small values.

 Encoding: Convert categorical data (for example, “Yes”, “No”, “Male”, “Female”) to numbers.

 Binning: Group numerical data into a category (e.g., Age: 0-18 = “teen”, 19-59 = “adult”, 60+
= “senior”)

Step 5: Data Reduction

This step is the process of removing or reducing parts of the data without losing relevant
information. This is useful when the data is too large to be processed quickly.

 Remove Unnecessary Columns: Delete the columns that are not needed.

 Dimensionality Reduction: Use approaches like principal component analysis (PCA) to reduce
the number of features.

Step 6: Data Discretization

In some instances, continuous values (e.g., age, salary, etc.) can be grouped into a fixed category or a
number of categories. Through the grouping of similar or related values, the analysis can become
clearer.

Step 7: Final Dataset Preparation

Once all the steps are completed and the final dataset is ready, which is now clean and ready to be
used in:

Basic data Object and its attribute:

Have you ever wondered how Netflix recommends shows or how banks detect fraud? The magic lies
in data mining – a powerful tool to analyse patterns in large datasets.

At its core, a data object is simply an entity. Think of it as a unit of data – a customer, a product, or
even a transaction. Every dataset you analyse is made up of multiple data objects.
For instance:

 A data object in e-commerce could be a product like “Red T-shirt.”

 In banking, it could be a customer account.

 In healthcare, it might represent a patient record.

But what makes these data objects meaningful? That’s where attributes come in!

What are Data Attributes?

 Data attributes refer to the specific characteristics or properties that describe individual data
objects within a dataset.

 These attributes provide meaningful information about the objects and are used to analyze,
classify, or manipulate the data.

 Understanding and analyzing data attributes is fundamental in various fields such

as statistics , machine learning , and data analysis, as they form the basis for deriving insights
and making informed decisions from the data.

 Within predictive models, attributes serve as the predictors influencing an outcome. In

descriptive models, attributes constitute the pieces of information under examination for
inherent patterns or correlations.

We can say that a set of attributes used to describe a given object are known as attribute vector or
feature vector.
Examples of data attributes include numerical values (e.g., age, height), categorical labels (e.g., color,
type), textual descriptions (e.g., name, description), or any other measurable or qualitative aspect of
the data objects.

Types of attributes:
This is the initial phase of data preprocessing involves categorizing attributes into different types,
which serves as a foundation for subsequent data processing steps. Attributes can be broadly
classified into two main types:

1. Qualitative (Nominal (N), Ordinal (O), Binary(B)).

2. Quantitative (Numeric, Discrete, Continuous)
Qualitative Attributes:

1. Nominal Attributes :

Nominal attributes, as related to names, refer to categorical data where the values represent
different categories or labels without any inherent order or ranking. These attributes are often used
to represent names or labels associated with objects, entities, or concepts.

Example :

2. Binary Attributes: Binary attributes are a type of qualitative attribute where the data can take on
only two distinct values or states. These attributes are often used to represent yes/no,
presence/absence, or true/false conditions within a dataset. They are particularly useful for
representing categorical data where there are only two possible outcomes. For instance, in a medical
study, a binary attribute could represent whether a patient is affected or unaffected by a particular
condition.

 Symmetric: In a symmetric attribute, both values or states are considered equally important
or interchangeable. For example, in the attribute "Gender" with values "Male" and "Female,"
neither value holds precedence over the other, and they are considered equally significant
for analysis purposes.
  Asymmetric: An asymmetric attribute indicates that the two values or states are not equally
important or interchangeable. For instance, in the attribute "Result" with values "Pass" and
"Fail," the states are not of equal importance; passing may hold greater significance than
failing in certain contexts, such as academic grading or certification exams

3. Ordinal Attributes : Ordinal attributes are a type of qualitative attribute where the values possess
a meaningful order or ranking, but the magnitude between values is not precisely quantified. In
other words, while the order of values indicates their relative importance or precedence, the
numerical difference between them is not standardized or known.

Example:

Quantitative Attributes:

1. Numeric: A numeric attribute is quantitative because, it is a measurable quantity, represented in

integer or real values. Numerical attributes are of 2 types: interval , and ratio-scaled.

 An interval-scaled attribute has values, whose differences are interpretable, but the
numerical attributes do not have the correct reference point, or we can call zero points. Data
can be added and subtracted at an interval scale but can not be multiplied or divided.
Consider an example of temperature in degrees Centigrade. If a day's temperature of one
day is twice of the other day we cannot say that one day is twice as hot as another day.

 A ratio-scaled attribute is a numeric attribute with a fix zero-point. If a measurement is ratio-

scaled, we can say of a value as being a multiple (or ratio) of another value. The values are
 ordered, and we can also compute the difference between values, and the mean, median,
mode, Quantile-range, and Five number summary can be given.

2. Discrete : Discrete data refer to information that can take on specific, separate values rather than a
continuous range. These values are often distinct and separate from one another, and they can be
either numerical or categorical in nature.

Example:

3. Continuous : Continuous data, unlike discrete data, can take on an infinit

Basic Statistical Descriptions of Data

Basic statistical descriptions can be used to identify properties of the data and highlight which data
values should be treated as noise or outliers.

For data preprocessing tasks, we want to learn about data characteristics regarding central
tendency of the data.

 Measures of central tendency include Mean, Median, and Mode.

Mean
The most common and effective numeric measure of the “center” of a set of data is the (arithmetic)
mean. Let x1, x2, ……, xN be a set of N values or observations, such as for some numeric attribute X,
like salary.

The mean of this set of values is

∑N
i=1 𝑥i 𝑥1 + 𝑥2 + . . . +𝑥N
x̄ = =
𝑁 𝑁
Example: Mean. Suppose we have the following values for salary (in thousands of dollars), shown in
increasing order: 30, 36, 47, 50, 52, 52, 56, 60, 63, 70, 70, 110. Using above Eq.,

Statistics

we have
30 + 36 + 47 + 50 + 52 + 52 + 56 + 60 + 63 + 70 + 70 + 110 696
x̄ = = = 58
12 12
Thus, the mean salary is $58,000.
Sometimes, each value xi in a set may be associated with a weight wi for i = 1,, ,N. The weights reflect
the significance, importance, or occurrence frequency attached to their respective values. In this
case, we can compute

∑N
i=1 𝑤i 𝑥i 𝑤1 𝑥1 + 𝑤2 𝑥2 + . . . +𝑤N 𝑥
x̄ = N =
∑i=1 𝑤i 𝑤1 + 𝑤2 + . . . +𝑤N

This is called the weighted arithmetic mean or the weighted average.

Median

Another measure of the center of data is the median. Suppose that a given data set of N distinct
values is sorted in numerical order.

 If N is odd, the median is the middle value of the ordered set;

 If N is even, the median is the average of the middle two values.

Example: Median. Suppose we have the following values for salary (in thousands of dollars), shown
in increasing order: 30, 36, 47, 50, 52, 52, 56, 60, 63, 70, 70, 110.

There is an even number of observations (i.e., 12); therefore, the median is not unique. It can be any
value within the two middlemost values of 52 and 56 (that is, within the sixth and seventh values in
the list). By convention, we assign the average of the two middlemost values as the median;

that is
52 + 56 108
= = 54
2 2
Thus, the median is $54,000.

In probability and statistics, the median generally applies to numeric data; however, we may extend
the concept to ordinal data.

Suppose that a given data set of N values for an attribute X is sorted in increasing order.

 If N is odd, then the median is the middle value of the ordered set.

 If N is even, then the median may not be not unique.

In this case, the median is the two middlemost values and any value in between.

Mode

Another measure of central tendency is the mode. The mode for a set of data is the value that
occurs most frequently in the set.

It is possible for the greatest frequency to correspond to several different values, which results in
more than one mode.

 Data sets with one, two, or three modes: called unimodal, bimodal, and trimodal.

 At the other extreme, if each data value occurs only once, then there is no mode.
Example: Mode. Suppose we have the following values for salary (in thousands of dollars), shown in
increasing order: 30, 36, 47, 50, 52, 52, 56, 60, 63, 70, 70, 110.

The above data has bimodal mode.i. e The two modes are 52 and 70.

Midrange

The midrange can also be used to assess the central tendency of a numeric data set. It is the average
of the largest and smallest values in the set.

Example: Midrange. Suppose we have the following values for salary (in thousands of dollars), shown
in increasing order: 30, 36, 47, 50, 52, 52, 56, 60, 63, 70, 70, 110.

The midrange of the data is

30,000 + 110,000
= $70,000
2
Thus, the median is $70,000

Central Tendency Measures for different attributes:

Central Tendency Measures for Numerical Attributes: Mean, Median, Mode

Central Tendency Measures for Categorical Attributes:

 Central Tendency Measures for Nominal Attributes: Mode

 Central Tendency Measures for Ordinal Attributes: Mode, Median

Example:

What are central tendency measures (mean, median, mode) for the following attributes?

Solution:
attr1 = {2,4,4,6,8,24}
mean = (2+4+4+6+8+24)/6 = 8 average of all values
median = (4+6)/2 = 5 avg. of two middle values
mode = 4 most frequent item

attr2 = {2,4,7,10,12}
mean = (2+4+7+10+12)/5 = 7 average of all values
median = 7 middle value
mode = any of them (no mode) all of them has same freq.

attr3 = {xs, s, s, s, m, m, l}
mean is meaningless for categorical attributes.
median = s middle value
mode = s most frequent item
Introduction

Measuring similarity and dissimilarity in data mining is an important task that helps identify patterns
and relationships in large datasets. To quantify the degree of similarity or dissimilarity between two
data points or objects, mathematical functions called similarity and dissimilarity measures are used.
Similarity measures produce a score that indicates the degree of similarity between two data points,
while dissimilarity measures produce a score that indicates the degree of dissimilarity between two
data points. These measures are crucial for many data mining tasks, such as identifying duplicate
records, clustering, classification, and anomaly detection.

Let’s understand measures of similarity and dissimilarity in data mining and explore various methods
to use these measures.

Basics of Similarity and Dissimilarity Measures

Similarity Measure

 A similarity measure is a mathematical function that quantifies the degree of similarity

between two objects or data points. It is a numerical score measuring how alike two data
points are.

 It takes two data points as input and produces a similarity score as output, typically ranging
from 0 (completely dissimilar) to 1 (identical or perfectly similar).

 A similarity measure can be based on various mathematical techniques such as Cosine

similarity, Jaccard similarity, and Pearson correlation coefficient.

 Similarity measures are generally used to identify duplicate records, equivalent instances, or
identifying clusters.

Dissimilarity Measure

 A dissimilarity measure is a mathematical function that quantifies the degree of dissimilarity

between two objects or data points. It is a numerical score measuring how different two data
points are.

 It takes two data points as input and produces a dissimilarity score as output, ranging from 0
(identical or perfectly similar) to 1 (completely dissimilar). A few dissimilarity measures also
have infinity as their upper limit.

 A dissimilarity measure can be obtained by using different techniques such as Euclidean

distance, Manhattan distance, and Hamming distance.

 Dissimilarity measures are often used in identifying outliers, anomalies, or clusters.