INST 3 – DATA SCIENCE ANALYTICS CHAPTER IV

INSTRUCTOR: ENGR. JONATHAN REY S. DE JESUS

Republic of the Philippines

Isabela State University
Echague, Isabela

College of Computing Studies, Information and Communication Technology

CHAPTER IV:
Data Collec on and Prepara on

Importance of Data Collec on and Prepara on

"Garbage in, Garbage out"

Data is the raw material of analy cs. Imagine trying to build a house with weak bricks—it won’t
stand. The same applies to analy cs: if your data is messy or unreliable, your insights or results will be
misleading. Good data collec on ensures that the dataset represents the problem you want to solve or
study. Proper prepara on (cleaning, organizing, forma ng) ensures that the data is usable for tools and
algorithms.

Good Data collec on technique ensures reliability, relevance, and representa veness. In
prac ce, 70–80% of a data analyst’s job is preparing data—not running fancy models or computa on.

Basic Data Quality Assessment

High-quality data result in trustworthy insights, while Low-quality data leads to expensive
mistakes. Before using any dataset, you must test its quality. There are 5 key Dimensions that we look for
in Good quality data:

1. Accuracy – Data must reflect reality. affinity with original intent, reliability as compared to
an authorita ve source, correla on of data elements.
Example: A student’s recorded age is “200” this is inaccurate. Pa ent’s Last name
is spelled incorrectly.
2. Completeness – Missing values reduce reliability. This is the availability of required data
a ributes.
Example: Half the student grades missing therefor it is useless dataset. Zip Code
is missing from an Address.
3. Consistency – Data should be uniform. Data should be compliant with required pa erns
and uniformity rules.
Example: “Male/Female” vs. “M/F” vs. “man/woman” the data collected is
inconsistent.
4. Timeliness – Data should be up to date. This focuses on whether the data is available
when needed and how current it is to accurately inform decision making.
Example: Using 2010 traffic data to design 2025 roads.
5. Validity – Values must be within acceptable ranges. This involves checking whether your
data adheres to a specific format or falls within a certain range.
Example: Nega ve exam scores this should be invalid.
Remember that you can’t trust your results un l you’ve tested the quality of your data.
INST 3 – DATA SCIENCE ANALYTICS CHAPTER IV
INSTRUCTOR: ENGR. JONATHAN REY S. DE JESUS

Ethical Considera ons in Data Collec on

Data is not just numbers, more o en than not, it represents real people. Mishandling it can violate
privacy, introduce bias, and cause harm. As future data scien sts and analysts, you carry responsibility.
Some ethical principles that we should prac ce are:

1. Informed Consent – Individuals must know why data is collected and how it will be used.
Example: Students filling out a stress survey should be told if their answers will be
anonymized.
2. Privacy & Confiden ality – Protect sensi ve personal informa on.
Example: Student grades should not be publicly shared with names.
3. Fairness & Bias Preven on – Ensure data is representa ve.
Example: A survey about student financial struggles should not only target wealthy
schools.
4. Transparency – Be honest about data collec on methods and limita ons.

Philippines Laws and Regual ons

1. Data Privacy Act of 2012 (RA 10173)
 Protects personal data collected by government and private organiza ons.
 Requires organiza ons to inform individuals about data usage and get consent.
 Mandates data security measures to prevent breaches.
2. Cybercrime Preven on Act of 2012 (RA 10175)
 Penalizes illegal access, misuse, or sharing of digital informa on.
 Protects individuals from iden ty the , hacking, and online fraud.
3. Freedom of Informa on (FOI) Execu ve Order No. 2, s. 2016
 Allows ci zens access to government data for transparency.
 Balances openness with data privacy.

Data Cleaning and Preprocessing in Excel

Data cleaning is a me taking & complex task. For achieving excellence in the data industry,
organiza onal needs to have professionals who understand the in(s) & out(s) of the data
cleaning/cleansing process.

Dirty Data Vs. Clean Data

Dirty Data is data that is incomplete, incorrect, or irrelevant to the problem you're trying to solve.
This may include Spellings, Duplicates, formats, and missing data or null data.

Common Examples of Dirty Data:

1. Spelling and Other text errors - Examples of this are misspellings, spelling varia ons, mixed
up le ers, inconsistent punctua on, and typos in general.
2. Inconsistent labels and formats
3. Null/Empty/Missing data
- Null is an indica on that a value does not exist in a dataset.
a. Duplicates
INST 3 – DATA SCIENCE ANALYTICS CHAPTER IV
INSTRUCTOR: ENGR. JONATHAN REY S. DE JESUS

Clean Data is data that is complete, correct, and relevant. Having Clean Data allows you to
understand and analyze the informa on and iden fy important pa erns, connect related
informa on, and draw useful conclusions.

Data Cleaning Process/Steps

The techniques and process for data cleaning will be different depending on the specific
dataset you’re working with. But here is a general guide on what to do and when cleaning
data:

1. Parsing (Deconstruc ng/Analyzing)

Parsing locates and iden fies individual data elements in the source files and then retains or
removes these data elements in the target files. This also includes removing unwanted data
or data that is irrelevant to the goal we want to achieve.

** Before removing unwanted data, it’s always good prac ce to make a copy of the data set.
That way if you remove something that ends up being needed in the future, we can always
have access to it and put it back in the dataset. **

2. Matching (Deduplica on)

Searching and matching records within and across the parsed, corrected, and standardized
data based on predefined business rules to eliminate duplica ons.
3. Handling Missing Data
In this step we need to decide whether to fill, delete, or leave as is data that are missing.
4. Correc ng
Corrects parsed individual data components using sophis cated data algorithms and
secondary data sources. In this step we are cleaning up text or data like removing extra
spaces and blanks, and fixing typos.
5. Standardizing
Standardizing applies conversion rou nes to transform data into its preferred (and
consistent) format using both standard and custom business rules. Examples include
adding/removing forma ng or se ng the number forma ng of certain columns.

6. Merging and/or Spli ng

Analyzing and iden fying rela onships between matched records and merging them into
one representa on or spli ng them into more useful fields for analysis.

Handling Missing Data and Outliers

Real-world data is rarely perfect—some values are missing, others are extreme outliers. How you
treat them can change your analysis results. The concept of missing data is implied in the name: it’s data
that is not captured for a variable for the observa on in ques on. Missing data can skew all kinds of tasks
for data scien sts, a er all, any analysis is only as good as the data used. Fortunately, there are proven
techniques to deal with missing data.
INST 3 – DATA SCIENCE ANALYTICS CHAPTER IV
INSTRUCTOR: ENGR. JONATHAN REY S. DE JESUS

Strategies for Handling Missing Values in Data Analysis

Depending on the nature of the data and the missingness, several strategies can help maintain the
integrity of our analysis. There is no single “best” way to deal with missing data—it depends on the
context. Let's see some of the most effec ve methods to handle missing values.

1. Leave as-is

Removing rows with missing values is a simple and straigh orward method to handle
missing data, used when we want to keep our analysis clean and minimize complexity.

Advantages:

 Preserves authen city of the dataset.

 Missingness itself may carry meaning (e.g., skipped survey ques ons might show
sensi vity to certain topics).

Disadvantages:

 Some tools and algorithms cannot handle missing values directly.

 May reduce accuracy in sta s cal analysis.

1. Removing Rows with Missing Values (Delete)

Removing rows with missing values is a simple and straigh orward method to handle
missing data, used when we want to keep our analysis clean and minimize complexity.

Advantages:

 Simple and efficient: It’s easy to implement and quickly removes data points with
missing values.

 Cleans data: It removes poten ally problema c data points, ensuring that only
complete rows remain in the dataset.

Disadvantages:

 Reduces sample size: When rows are removed, the overall dataset shrinks which can
affect the power and accuracy of our analysis.

 Poten al bias: If missing data is not random (e.g if certain groups are more likely to
have missing values) removing rows could introduce bias.

2. Imputa on Methods

Imputa on involves replacing missing values with es mated values. This approach is
beneficial when we want to preserve the dataset’s sample size and avoid losing data points.
However, it's important to note that the accuracy of the imputed values may not always be
reliable.

2.1 Mean, Median and Mode Imputa on:

INST 3 – DATA SCIENCE ANALYTICS CHAPTER IV
INSTRUCTOR: ENGR. JONATHAN REY S. DE JESUS

This method involves replacing missing values with the mean, median or mode of
the relevant variable. It's a simple approach but it doesn't account for the rela onships
between variables.

Advantages:

 Simple and efficient: Easy to implement and quick.

 Works well with numerical data: It is useful for numerical variables with a
normal distribu on.

Disadvantages:

 Inaccuracy: It assumes the missing value is similar to the central tendency

(mean/median/mode) which may not always be the case.

Handling Outliers

Outliers are values that deviate significantly from other values in a data set. They indicate an
irregularity in the data pa ern. Outlier values can be very large or very small compared to the other values
in the data set.

Outliers can dras cally affect the data results, especially if you are trying to calculate the middle
value or a data set where all the data values have different distances. Outliers can also provide important
insights into the underlying phenomenon you are studying, but they can also result from errors in
measuring data entry, or natural varia on in the data.

Possible Treatment for outliers:

1. Correc on – Fix if it’s a clear error

Example: age recorded as 200, we can correct this to 20).

2. Imputa on – Usually, the outlier value is replaced with the median, mean, or mode value.

3. Capping/Trimming – Replace outliers with upper/lower boundary values.

4. Removal – Delete outlier rows if they are errors and not relevant.

5. Reten on – Keep them if they are valid rare cases

Example: a prodigy student with 0 a endance but 95% grade.