Data Science with Python II
Module 1: Data Cleaning
1
Course Plan
Module Titles
Module 1 – Data Cleaning
Module 2 – Exploratory Analysis & Working with Time Series Data
Module 3 – Introduction to Predictive Modelling & Model Building
Module 4 – Drawing Inferences
Module 5 – Data Management, Privacy & Data Security
2
Introductions
• Name
• Industry
• What do you hope to get out of this course?
3
Topics for this Module
• 1.1 Data Issues
• 1.2 Variable Identification and Data Structuring
• 1.3 Univariate & Bivariate Analysis
• 1.4 Missing Data and Imputation
4
Learning Outcomes for this Module
• Identify issues in a given data set
• Clean and prepare data for analysis
• Impute missing data
5
Module 1 – Section 1
Data Issues
6
Group Discussion
What are the most common data issues you have
come across in the past? What challenges do they
pose? How do you usually deal with them?
Time: 5 minutes
7
Common Data Issues
§ Inconsistently format (416-123-4567 vs. 4161234567)
§ Incorrectly captured data (Toronot vs Toronto)
§ Concatenated data (TorontoON, A1A1A1ON)
§ Irrelevant data (Capturing age, but not birthdate)
§ False data (fake birthdate, fake name)
§ Incomplete data (John S. vs John Smith)
§ Missing data
§ Wrong data type
8
Why do data issues occur?
§ Collection mechanisms (e.g. data entry form) may be
inconsistent or open-ended
§ Input errors
§ Non-mandatory fields
§ System glitches
9
Dealing With Data Issues
§ Re-collection and verification
§ Data clean-up with Python
§ System updates
§ Missing data imputation
10
Module 1 – Section 2
Variable Identification and Data
Structuring
11
Variable Identification
Variable identification refers to understanding what is meant by
each variable in your data set. For example:
• Dollar amounts (in millions? Thousands?)
• Population (of a country? City? Sample?)
• Abbreviations
• Acronyms
Action: build a data dictionary for your data set to ensure all
stakeholders, and team members, are aligned.
12
Data Structuring
Data should be organized such that it can be interpreted by
statistical software.
Guidelines:
1. Each variable should have its own column
2. Each observation should have its own row
3. Each value should have its own cell
4. Multiple tables should be joined by a column
13
Module 1 – Section 3
Univariate and Bivariate Analysis
14
Univariate Analysis
Univariate Analysis refers to understanding the distribution of
each variable. Consider the following:
1. Does the range make sense?
2. Does the distribution make sense?
3. Should there be negative values? Zeros? N/As?
4. Is the data coded correctly?
15
Univariate Analysis in Python
Summary statistics can be used to review individual variables:
• Mean
• Median
• Mode
• Range
• Standard Deviation
• Variance
In addition, visualizing data helps. For distributions, consider
histograms, and for time series – line charts.
16
Normal Distribution
A normal distribution follows the central limit theory which
states that some independent factors influence a particular
characteristic. When these independent factors contribute to
the characteristic individually, their normalized sum tends to
result in a Gaussian distribution (in other words, a Normal
Distribution).
17
Interpreting a Distribution
68% of data points 95% within 2 st dev
should fall within 1 st dev 99% within 3 st dev
Mean
1 standard deviation
within the mean
2 standard deviations
within the mean 18
Group Discussion
Consider the following scenarios. What could this
indicate?
• A survey conducted about home purchase preferences
indicates the average respondent age is 25
• Two sets of samples are collected from a manufacturing
plant about processing times. One has a standard deviation
of 2.3 minutes, while the other is 4.2 minutes.
• A comparison of two store locations shows that the range of
customer annual incomes is $5,000 in one, and $32,000 in
another.
Time: 10 minutes
19
Bivariate Analysis
Bivariate Analysis refers to understanding variables in pairs.
This can be done through the use of a scatter plot to identify
correlations.
20
Module 1 – Section 4
Missing Data and Imputation
21
What is missing data and why does it happen?
• Missing data refers to data points not available for a certain
observation. In Python, they're typically denoted with NaN.
• This occurs when collection efforts are incomplete (e.g. a
survey participants fills out 48/50 questions)
• However, an observation should not be entirely deleted
because of some missing data
22
Missing Data Patterns
MCAR: Missing
Completely at Random
MAR: Missing at
Random
MNAR: Missing Not at
Random
23
MCAR: Missing Completely At Random
• No pattern to define missing data
• Could occur because of technical errors / challenges, errors
in data collection, or survey participants omitting information
• Example: if an analysis is done using a person's postal code
at work and home to determine commute distance, and
someone did not know a work postal code, but no other
relevant information has been collected (e.g. length of time
at employer), this would be MCAR
• MCAR rarely results in biased outputs; therefore it can be
corrected by omitting incomplete records from the sample
24
MAR: Missing at Random
• Data missing at random refers to missing data where a
pattern exists (i.e. the probability of an observation missing
can be explained by the data available)
• It is impossible to prove that data is MAR, however one can
demonstrate the likelihood of data missing based on
individual responses (e.g. if individuals are less likely to
remember, or a condition would trigger them not to respond)
• Example: if we ask individuals in a survey to tell us the
genre of the last movie they went to see and how long ago
they went to the movies, there is a probability that
respondents who have not been at the movies for over 6
months will not remember the genre of the last movie seen.
• If data is MAR, it cannot be ignored.
25
MNAR: Missing Not at Random
• Data missing not at random is missing data with a pattern to
the likelihood that a data element is missing that depends
on the missing element.
• Example: if asking a person how many crimes they
committed but got away with in the last year, they may
choose not to respond due to the nature of the question.
• Data MNAR poses serious problems with the analysis, as
the relevant data is missing.
26
Techniques for Missing Data
Drop
Observation
Mean/Median
Substitution
Modelling
Techniques
27
Drop Observation
• Used for MCAR data
• Drop observations which have missing data
• Count the number of NaNs and determine if removing the
observations would cause a significant impact to the sample
size
• Rule of thumb: less than 10% of observations can be
removed
28
Mean/Median Substitution
• Used for MCAR data
• Substitute missing values with the mean of the field
• Be mindful of data stratification (e.g. age groups, postal
code, etc.); use the groupby() function to segregate different
observation groups
29
Modeling Techniques
• Techniques depend on developing a model for the missing
observation based on the remaining data
• For example, suppose that X2 was omitted on the 5th of
every 200 survey respondents; you could use the remaining
x-variables to build a model to calculate X2
X2=β0+β1X1+β3X3+...+βKXK+Error
30
Group Discussion
1. You conduct a survey, and collect the following
demographic data: name, gender, age, address, and
income. You have some missing age data – which
method could you use to fill in the missing values?
2. You conduct a survey to capture purchase
preferences, and some people did not complete the
survey. What could you do?
3. You collect operational performance data at a
distribution plant. You are missing data about a large
chunk of the shift (2/8 hours). How should you handle
this?
Time: 10 minutes
31
Follow us on social
Join the conversation with us online:
[Link]/uoftscs
@uoftscs
[Link]/company/university-of-toronto-school-of-continuing-studies
@uoftscs
32
Any questions?
33
Thank You
Thank you for choosing the University of Toronto
School of Continuing Studies
34