CHAPTER 3 STS3404

DATA COLLECTION DATA SCIENCE WITH APPLICATIONS

(Sains Data dengan Aplikasi )
AND DATA
BLENDING PART 2
nurhaizum_ar@[Link]
Data preprocessing

Data preprocessing is an important step in the data mining process. It

refers to the cleaning, transforming, and integrating of data in order to
make it ready for analysis.

The goal of data preprocessing is to improve the quality of the data and
to make it more suitable for the specific data mining task.

Data mining is the process of finding anomalies, patterns

and correlations within large data sets to predict outcomes.
 Some common steps in data preprocessing include:

• Data cleaning: this step involves identifying and removing missing, inconsistent, or irrelevant data. This
can include removing duplicate records, filling in missing values, and handling outliers.
• Data integration: this step involves combining data from multiple sources, such as databases,
spreadsheets, and text files. The goal of integration is to create a single, consistent view of the data.
• Data transformation: this step involves converting the data into a format that is more suitable for the data
mining task. This can include normalizing numerical data, creating dummy variables, and encoding
categorical data.
• Data reduction: this step is used to select a subset of the data that is relevant to the data mining task. This
can include feature selection (selecting a subset of the variables) or feature extraction (extracting new
variables from the data).
• Data discretization: this step is used to convert continuous numerical data into categorical data, which can
be used for decision tree and other categorical data mining techniques.
 What is dirty data?

• Dirty data, or unclean data, is data that is in some way faulty: it might contain
duplicates, or be outdated, insecure, incomplete, inaccurate, or inconsistent.

• Examples of dirty data include misspelled addresses, missing field values, outdated
phone numbers, and duplicate customer records.

• When ignored, dirty data can cause serious issues for your business. It can
jeopardize the customer experience, lead to the misrepresentation of business
results, and negatively impact strategic decisions.

• To avoid the risks of poor data quality, regular data cleansing is essential. We’ll
discuss how to clean data later
 • Human error
Where does • Department miscommunication
• Poor data strategy
Dirty Data • Customer disinterest or doubt
come from? • Wrong form formats
• Lack of standardization
 Common Types of Dirty Data

1. Duplicate data
• Duplicate data refers to records that partially or fully share the same information. They
come about when the same information is entered multiple times, sometimes in different
formats. Because customer information is scattered across different records.
• Duplicate customer data leads to poor customer service, Incorrect tracking and reporting,
Double (or triple) marketing targeting.

2. Insecure data
• Insecure data is data that is not encrypted, or access controlled. It’s accessible by anyone in
your company and—in worst case scenarios—even by third parties. Insecure data constitutes
not just a privacy risk, but also a legal threat as companies risk being non-compliant with laws
such as Personal Data Protection Act 2010 (PDPA).
3. Incomplete data (N/A) –missing data
• Missing data are unobserved values that would be meaningful for analysis if observed; in
other words, a missing value hides a meaningful value.

• An example of dirty data that’s incomplete would be if your newsletter sign-up form has a
field for the person’s first name, but the field isn’t a required field. people are then able to
sign up without leaving their name, which would render your personalized email campaigns
less effective.

• SURVEY: Respondents forgot to answer the question(s), respondents refuse to answer,

respondents fail to finished the survey.

• INDUSTRY: Sensor failed, someone turned off the recording equipment, power cut, method to
collect data change, internet connection was lost, internet down, hard drive corrupt, etc
4. Inaccurate data
• Inaccurate data is data that contains mistakes. An example of inaccurate data would be a
customer entering their last name on one of your forms but making a typo. In this case, you
have the customer’s last name but it’s inaccurate. It’s a dirty record.

• Another example would be if a sales representative logs an incorrect phone number for a
customer in Salesforce. In this case, it’s crucial to improve Salesforce data to continue the
conversation with this customer.
5. Outdated data
• Outdated data is inaccurate not because it was entered incorrectly, but because it used to
be accurate and now it isn’t anymore. A typical example of dirty data that’s outdated is if
you still lists a customer’s old address after they’ve moved.
• Other examples of outdated data are email addresses that are no longer in use, Titles of people
who’ve switched jobs

6. Incorrect data
• Incorrect data is data that falls outside of previously specified parameters. As such, it is
easier to prevent. An example would be if a customer enters their birthdate using a dropdown
menu. Your system will likely only allow them to select one out of 12 months, one out of 31
days, and perhaps they also won’t be able to select a birth year that would make them older
than 130 years.

7. Inconsistent data
• Inconsistent data is also known as data redundancy. It occurs when companies store the
same information in different places without syncing that information. A prime example
would be a company storing customer information both in its CRM and in its email marketing
tool.
DATA CLEANING
 Data Cleaning

• Dirty data comes in various forms and shapes. To get your data up to scratch
and ready for analysis you’ll first need to clean it.

• Data cleaning, while not always the most favored part of the data journey,
needs to be an integral part of your data preprocessing practice. Dismissing it
at the beginning of the process will create potential headaches for yourself
further down the line.

• Data cleaning, also referred to as data cleansing and data scrubbing. Data
cleaning is the process of fixing or removing incorrect, corrupted, incorrectly
formatted, duplicate, or incomplete data within a dataset
 How to clean your data?
[Link]

The techniques used for data cleaning may vary according to the types of data that company stores.
These basic steps help to map out a framework in data cleaning.

Step 1: Remove duplicate or irrelevant observations

Remove unwanted observations from your dataset, including duplicate observations or irrelevant
observations.

Step 2: Fix structural errors

Structural errors are when you measure or transfer data and notice strange naming conventions, typos, or
incorrect capitalization. These inconsistencies can cause mislabeled categories or classes. For example,
you may find “N/A” and “Not Applicable” both appear, but they should be analyzed as the same category.
• Step 3: Filter unwanted outliers
Outliers can adversely affect model results. If you have a legitimate reason to remove an
outlier, like improper data-entry, doing so will help the performance of the data you are
working with. However, just because an outlier exists, doesn’t mean it is incorrect. This step
is needed to determine the validity of that number.

• Step 4: Handle missing data

Missing values again can affect the performance of your model. Here dropping missing values
or imputation methods like mean, median, and mode should be used. If the scope of the
project allows, then advanced imputation methods involving statistical and machine learning
models can be used. Another option, you can drop observations that have missing values,
but doing this will drop or lose information, so be mindful of this before you remove it.

• Step 5: Validate and QA

Does the data make sense? Does the data follow the appropriate rules for its field? Does it
prove or disprove your working theory, or bring any insight to light? Can you find trends in the
data to help you form your next theory? If not, is that because of a data quality issue?
Dealing with Missing Data
Broadly there are TWO common solutions:

1. Deleting rows with missing values

• The simplest solution - is to not use the records with missing values
when training your model.
• Some issues to be aware of before you starting deleting masses of
rows from your dataset.
2. Filling in the Values (Imputation)
• Fill the missing values with a value. But what value to use? This
depends on a range of factors, including the type of data you are
trying to fill.
Contributed by Daniel D. Gutierrez, Managing Editor
and Resident Data Scientist of insideBIGDATA. In
addition to being a tech journalist, Daniel is also a
practicing data scientist, author, educator and sits on
a number of advisory boards for various start-up
companies.
 (1) DELETION

Problem
What is the difference between listwise and pairwise deletion of cases?
____________________________________________________________________________
In listwise deletion a case is dropped from an analysis because it has a missing value in at least
one of the specified variables. The analysis is only run on cases which have a complete set of
data.

Pairwise deletion occurs when the statistical procedure uses cases that contain some missing
data. The procedure cannot include a particular variable when it has a missing value, but it can
still use the case when analyzing other variables with non-missing values.

Deleting Columns containing missing values. If a column contains a lot of missing values, say
more than 80%, and the feature is not significant, you might want to delete that feature. However,
again, it is not a good methodology to delete data.
listwise deletion Pairwise deletion
Advantage Advantage

Simple, easily compare across analyses Uses all possible information

Disadvantage Disadvantage

Could be biased Separate analyses cannot be compared as the

Reduces statistical power data/sample will be different
(2) IMPUTATION (SIMPLE IMPUTATION)

Popular Averaging Techniques

• Mean, median and mode are the most popular averaging techniques, which are
used to infer missing values. Approaches ranging from global average for the variable
to averages based on groups are usually considered.
(2) IMPUTATION (SIMPLE IMPUTATION) - cont

Predictive Techniques
• Imputation of missing values from predictive techniques assumes that the nature of
such missing observations are not observed completely at random and the variables
chosen to impute such missing observations have some relationship with it, else it
could yield imprecise estimates.
• A predictive model could be used to impute the missing values for Device, OS,
Revenues. There are various statistical methods like regression techniques, machine
learning methods like SVM and/or data mining methods to impute such missing
values.
IMPUTATION (MULTIPLE IMPUTATION)
Missing data puzzle pieces

• Missing data is like missing a puzzle

piece. If you drop it, that’s like
pretending the puzzle slot isn’t there.

• If you impute it, that’s like trying to

squeeze in a piece from somewhere else
in the puzzle.
DATA TRANSFORMATION
 What is the • Data cleaning is the process that
removes data that does not belong in
difference your dataset.

between data • Data transformation is the process of

converting data from one format or
cleaning and data structure into another.
• Transformation processes : transform
transformation? and map data from one "raw" data form
into another format for warehousing and
analyzing.
• Data transformation changes the format,
structure, or values of the data and
converts them into clean, usable data.
 Data Transformation Techniques
• Normalization:
It is done in order to scale the data values in a specified range (-1.0 to 1.0 or 0.0 to 1.0)

• Attribute Selection:
In this strategy, new attributes are constructed from the given set of attributes to help the
mining process.

• Discretization:
This is a process of converting continuous data into a set of data intervals. Continuous attribute
values are substituted by small interval labels. This method is also called a data reduction
mechanism as it transforms a large dataset into a set of categorical data. Discretization also
uses decision tree-based algorithms to produce short, compact, and accurate results when using
discrete values.
Concept Hierarchy Generation:

Here attributes are converted from lower level to higher level in hierarchy. For Example-The
attribute “city” can be converted to “country”.
For example, age data can be in the form of (20, 30) in a dataset. It is transformed into a
higher conceptual level into a categorical value (young, old).

Data Aggregation

Data collection or aggregation is the method of storing and presenting data in a summary
format. The data may be obtained from multiple data sources to integrate these data sources
into a data analysis description. This is a crucial step since the accuracy of data analysis
insights is highly dependent on the quantity and quality of the data used.