Data Science with Python II
Module 3: Introduction to Predictive
Modelling & Model Building
1
Course Plan
Module Titles
Module 1 – Data Cleaning
Module 2 – Exploratory Analysis & Working with Time Series Data
Module 3 – Introduction to Predictive Modelling & Model Building
Module 4 – Drawing Inferences
Module 5 – Data Management, Privacy & Data Security
2
Topics for this Module
• 3.1 Predictive Modeling Methods
• 3.2 Introduction to Linear Regression
• 3.3 Model Specifications
• 3.4 Feature Engineering
• 3.5 Dummy Variables
3
Learning Outcomes for this Module
• Understand types of predictive models
• Develop an approach for building a model
• Build a Linear Regression Model
• Engineer features to strengthen your analysis
• Create and use dummy variables for Linear
Regression Modelling
4
Module 3 – Section 1
Predictive Modelling Methods
5
Types of ML Algorithms
Supervised Learning
Unsupervised Learning
Reinforcement
Learning
6
Common Algorithms
• Linear Regression
– Used to estimate real values based on continuous variables (e.g. sales,
house prices)
• Logistic Regression
– Used to predict a binary outcome, usually for discrete data (e.g. winning
a game vs. not, successful sale vs. not)
• Decision Tree
– Used in classification problems (e.g. photos, bio samples)
• SVM (Support Vector Machine)
– Used in classification problems where a point belongs to one group or
another (e.g. height vs. hair length)
• Naive Bayes
– Classification technique which assumes independence between
predictors (e.g. probability of color, weight and shape predictors of fruit
would be considered independently)
• kNN (k-nearest neighbour)
– Classification mechanism which classifies data points based on majority
votes of k neighbours
7
Common Algorithms
• K-Means
– Classify a data set through clustering (e.g. types of clients)
• Random Forest
– An ensemble of decision trees for classification problems
• Dimensionality Reduction Algorithms
– Identifies the variables more significant to the analysis
• Gradient Boosting algorithms
– GBM
– XGBoost
– LightGBM
– CatBoost
8
Module 2 – Section 4
Introduction to Regression
9
Introduction to Regression
A linear regression model combines a series of parameters
along with explanatory variables into a linear equation of the
form:
Y=β0+β1X1+β2X2+...+βKXK+Error
The error term belongs in every regression model. Error term
refers to the residual value (i.e. what is not explained by the
equation).
10
The Econometric View of Regression
This view is known as causal modelling, and sees the model
development as an interaction between theory as expressed
through mathematics and data through the statistics of
regression.
• The mathematical model shows that the independent
variables explain the behavior of the dependent variables
(i.e. linear relationship)
• Data is applied to the model to find the best fit
The objective of this approach is to find the model that best fits
the data with as much of the variation as possible explained by
the relationship (i.e. small error term).
11
The Black Box View of Regression
The black box method is driven by algorithms that are highly
automated. Techniques (e.g. ridge regression, lasso
regression) do not leverage theory of causation but focus on
managing tradeoffs between the number of explanatory
variables used and amount of variation explained by the
model.
12
Econometric vs. Black Box
The econometric approach requires more hands-on
involvement, but allows for business knowledge to be
considered when developing a theory. If the theory is correct,
and independent variables are trustworthy in predicting
dependent variable behavior, this model can be used for
predictive and prescriptive analytics.
Some techniques in the black box approach are better for
prediction (i.e. plugging in x-values to determine what could
happen). If data to support a certain theory is unavailable, but
a large amount of potentially relevant data is available, the
black box method may be a better choice, as it can consider
numerous models very quickly.
13
Python Packages
• There are two packages which can be used for linear
regression in python:
Statsmodels
Sklearn (called scikit-learn)
14
Interpreting Regression Results
Regression line –
estimated values
lie here
Observed values
Residuals
15
Interpreting Regression Results
• R-squared
• P-value
• F-statistic
• Coefficients
• Standard Error
• P-value
16
Interpreting Regression Results
• The (R-squared) value: this articulates what percentage of
the variability in Y is explained by the model. This value will
always be between 0 and 1; if R-squared = 1, then the
model perfectly explains variability in Y, while if R-squared =
0, there is no linear relationship.
• The standard errors (std err): This is the standard error of
the coefficients. This measures the precision of the estimate
of the coefficient.
17
Interpreting Regression Results
• p-values, column P > |t|: To compute probability, it is
assumed that the coefficient is 0 (there is no dependence).
In other words, we assumed the coefficient is zero and
under this assumption the probability to find the fitted value
is P>|t|. The small value of P means that assumption about
the coefficient being zero is not reasonable and should be
discarded. Attributes with the p-value (column p>|t|) greater
than 0.05 are not significant for this model. Attributes with
values between 0 and 0.05 are statistically significant
predictors of the response. In the given example, P>|t| for
the Tobacco is 0.007, it has value much smaller than 0.05
and we accept the fit of the regression line.
18
Interpreting Regression Results
• 95% confidence interval: The two last columns ([0.025
0.975]) give the confidence interval, that is, with 95%
confidence the coefficient is between these limits.
• F-statistic: assessment of the overall effectiveness of the
model (i.e. does the combination of independent variables
have a statistically significant impact on the output, or would
the result be the same without them there?)
19
Module 3 – Section 2
Model Specifications
20
Building the Best Model
• Where possible, use theory to inform your model
development.
• Maintain multiple data sets for developing, estimating, and
testing.
• Where theory is weak, or nonspecific, use data to inform the
model specification.
• When testing models, data permitting, over-specify them,
which is to say, put in more variables than you need and
test down rather than testing up.
• Resolve issues of collinearity, particularly as a result of
multiple potential specifications with multi-variate tests.
21
Module 3 – Section 3
Feature Engineering
22
What is feature engineering?
• Features are any variables which are important in your model
• Feature engineering is the process through which you create new
variables to help make your model more accurate, useful or
simpler
• Feature engineering should be driven by theory:
– Brainstorm which features you may need
– Create or acquire them
– Test them in your model
– Revisit and determine if others are needed
• 5 types of features:
– Indicators
– Interactions
– Representations
– External Data Joins
– Scaling Data
23
Indicators
• Indicator variables represent special groups, classes or
periods of time
• Represented as dummy or binary variables (i.e. 1 is when
observation belongs to a group, 0 when it does not)
• Examples: marking groups belong to Canada vs. US, or
special times of year like Boxing Day or not.
• Use ‘where’ statement to create in Python
24
Interactions
• Interactions are combinations of variables in the dataset
• For example: calculating profit by subtracting cost from
revenues
• In Python, create a new column to calculate a new variable
25
Representations
• Feature representations make variables more useful
• Examples include:
– Combining sparse groups (e.g. small categories grouped as “other”)
– Making variables more detailed (e.g. breaking down height by feet
and inches)
– Creating categorical mapping (e.g. grouping time of day by morning,
afternoon or evening)
• These features are used for data that is difficult to model
• In Python, you can use “where” statements to filter for
certain values only, or you can use location indicators, or
groupby()
26
External Data Joins
• This is the most flexible form of feature engineering
• Joining multiple data frames or tables together to add more
data
• Examples: adding simple data like Statistics Canada or
Environics reports, or using APIs or other models to add
more information to existing dataset (e.g. using Google
Maps to translate coordinates into addresses, postal codes
and cities)
27
Scaling Data
• Data transformations can be used to improve a model
• Common transformations include: natural log or squaring a
variable
• Used if certain patterns are identified in the data
28
Module 3 – Section 4
Dummy Variables
29
What are dummy variables and why use them?
• A dummy variable or indicator variable is an artificial
variable created to represent an attribute with two or more
distinct categories / levels
• Regression analysis treats all independent (X) variables as
numerical (e.g. interval or ratio scale)
– This means typically numerical values have a meaning (e.g. 10 is
double the amount of 5)
• You may want to include an attribute or nominal scale
variable, like Product Brand or Type of Defect
– Let’s say you have 3 types of products labeled ‘1’, ‘2’ and ‘3’ <- in
this case, 3 minus 1 doesn’t mean anything significant. This is a
dummy variable.
30
Dummy Variable Example
• In the following housing price example, sale price is already
a numerical value
• We want to create dummy variables to represent whether a
house is older than 1990 or not, and located on the East or
South East side of the city.
31
Keep In Mind…
• The number of dummy variables necessary to represent a
single attribute variable is equal to the number of levels
(categories) in that variable minus one.
• For a given attribute variable, none of the dummy variables
constructed can be redundant. That is, one dummy variable
can not be a constant multiple or a simple linear relation of
another.
• The interaction of two attribute variables (e.g. Gender and
Marital Status) is represented by a third dummy variable
which is simply the product of the two individual dummy
variables.
32
Dummy Variables in Python
• We can use pandas to create dummy variables in Python
• Function: get_dummies()
• Syntax: pandas.get_dummies(data, prefix=None,
prefix_sep=’_’, dummy_na=False, columns=None,
sparse=False, drop_first=False, dtype=None)[source]
33
Follow us on social
Join the conversation with us online:
[Link]/uoftscs
@uoftscs
[Link]/company/university-of-toronto-school-of-continuing-studies
@uoftscs
34
Any questions?
35
Thank You
Thank you for choosing the University of Toronto
School of Continuing Studies
36