Assignment

Python Data Prep and Machine Learning

Instructor Syed Ali Ameer

Syed Ali Ameer

Syed Ali Ameer
Syed Ali Ameer
 INTRO TO MACHINE LEARNING (ML)

MACHINE LEARNING [ muh-sheen-lur-ning ]

noun

1. The capacity of a computer to process and evaluate data beyond

programmed algorithms, through contextualized inference*

Using statistical models to find patterns and make predictions

*[Link]
Syed Ali Ameer
 C O M M O N ML QUESTIONS

Which customers are most What will sales look like for What patterns do we see in
likely to churn next month? the next 12 months? terms of product cross-selling?

How can we use online When we adjusted tactics Which product is customer X
customer reviews to monitor last month, did we drive any most likely to purchase next?
changes in sentiment? incremental revenue?

Syed Ali Ameer

 WHEN IS ML THE RIGHT FIT?

• Machine Learning is a natural

extension of data profiling and basic
visual analysis

Machine Learning • ML is required when the underlying

Complexity data or analysis is too complex for
of analysis basic data profiling (i.e. visualizing
Multivariate relationships between 3+ variables)

• Machine Learning is ideal for finding

Univariate optimal solutions that would be
impossible or impractical to derive
through human trial-and-error
Complexity of data

Syed Ali Ameer

 THE MACHINE LEARNING PROCESS

Building models • ML models are only as good as

(the “fun” stuff) the data they are built on
(“garbage in, garbage out”)

• Data prep & QA happens behind

the scenes, but typically accounts
Data Prep, QA & Profiling
for the majority (80%+)of the
(the boring but really, really, machine learning workflow
REALLY important stuff)
• While it’s tempting to start with
the “fun” stuff, data prep and QA
skills are absolutely critical!

Syed Ali Ameer

 MACHINE LEARNING PROCESS

PREPARING YOUR DATA UNDERSTANDING YOUR DATA MODELING YOUR DATA

QUALITY ASSURANCE

UNIVARIATE PROFILING

MULTIVARIATE PROFILING

MACHINE LEARNING

Quality Assurance (QA) is about preparing & cleaning data prior to analysis. We’ll cover common QA topics
including variable types, empty/missing values, range & count calculations, censored data, etc.

Syed Ali Ameer

 MACHINE LEARNING PROCESS

PREPARING YOUR DATA UNDERSTANDING YOUR DATA MODELING YOUR DATA

QUALITY ASSURANCE

UNIVARIATE PROFILING

MULTIVARIATE PROFILING

MACHINE LEARNING

Univariate profiling is about exploring individual variables to build an understanding of your data. We’ll cover
common topics like normal distributions, frequency tables, histograms, etc.

Syed Ali Ameer

 MACHINE LEARNING PROCESS

PREPARING YOUR DATA UNDERSTANDING YOUR DATA MODELING YOUR DATA

QUALITY ASSURANCE

UNIVARIATE PROFILING

MULTIVARIATE PROFILING

MACHINE LEARNING

Multivariate profiling is about understanding relationships between multiple variables. We’ll cover common
tools for exploring categorical & numerical data, including kernel densities, violin & box plots, scatterplots, etc.

Syed Ali Ameer

 MACHINE LEARNING PROCESS

PREPARING YOUR DATA UNDERSTANDING YOUR DATA MODELING YOUR DATA

QUALITY ASSURANCE

UNIVARIATE PROFILING

MULTIVARIATE PROFILING

MACHINE LEARNING

Machine learning is a natural extension of multivariate profiling, and uses statistical models and methods to
answer questions which are too complex to solve using simple visual analysis or trial-and-error

Syed Ali Ameer

 MACHINE LEARNING LANDSCAPE

MACHINE LEARNING

Supervised Learning Unsupervised Learning Advanced Topics

Clustering/Segmentation
Classification Regression Reinforcement Learning
K-Means (Q-learning, deep RL, multi-armed-bandit, etc.)

K-Nearest Neighbors Least Squares Outlier Detection

Natural Language Processing
Naïve Bayes Linear Regression Markov Chains (Latent Semantic Analysis, Latent Dirichlet Analysis,
relationship extraction, semantic parsing, contextual
Logistic Regression Forecasting word embeddings, translation, etc.)
Matrix factorization, principal components, factor
analysis, UMAP, T-SNE, topological data analysis,
Sentiment Analysis Non-Linear Regression
advanced clustering, etc.
Computer Vision
Monte Carlo (Convolutional neural networks, style translation, etc.)
Random forest, support
vector machines, gradient LASSO/RIDGE, state-
boosting, neural space, advanced Deep Learning
nets/deep learning, etc. generalized linear (Feed Forward, Convolutional, RNN/LSTM, Attention,
methods, VAR, DFA, etc. Deep RL, Autoencoder, GAN. etc.)

Syed Ali Ameer

Correlation
The sample correlation coefficient quantifies the strength of the relationship between two variables.

Syed Ali Ameer

Correlation
The sample correlation coefficient quantifies the strength of the relationship between two variables.

We describe correlations with a unit-free measure called the correlation coefficient.

It ranges from -1 to +1 and is denoted by r

r close to +1: r close to 0: r close to -1:

positive correlation weak or no relationship negative correlation

Syed Ali Ameer

 REGRESSION

There are two common use cases for linear regression: prediction and root-cause analysis

PREDICTION ROOT-CAUSE ANALYSIS

• Used to predict or forecast a numerical • Used to determine the causal impact of

dependent variable individual model inputs

• Goal is to make accurate predictions, even • Goal is to prove causality by comparing

if causality cannot necessarily be proven the sensitivity of each IV on the DV

Syed Ali Ameer

 REGRESSION 101

The goal of regression is to predict a numeric dependent variable using independent variables

𝒚 Dependent variable (DV) – for regression, this must be numerical (not categorical)!
• This is the variable you’re trying to predict
• The dependent variable is commonly referred to as “Y”, “predicted”, “output”, or “target” variable
• Regression is about understanding how the numerical DV is impacted by, or dependent on, other variables in the model

𝐱 Independent variables (IVs)

• These are the variables which help you predict the dependent variable
• Independent variables are commonly referred to as “X’s”, “predictors”, “features”, “explanatory variables”, or “covariates”
• Regression is about understanding how the IVs impact, or predict, the DV

Syed Ali Ameer

 REGRESSION 101

EXAMPLE: Using marketing and sales data (sample below) to predict revenue for a given month

Social Competitive Marketing Promotion

Month ID
Posts Activity Spend Count
Revenue Revenue is our dependent variable, since
it’s what we want to predict
1 30 High $130,000 12 $1,300,050

2 15 Low $600,000 5 $11,233,310 Since Revenue is numerical, we’ll use

3 11 Medium $15,000 10 $1,112,050 regression (vs. classification) to predict it
4 22 Medium $705,000 11 $1,582,077

5 41 High $3,000 3 $1,889,053

6 5 High $3,500 3 $1,200,089

7 23 Low $280,000 2 $1,300,080

8 12 Low $120,000 11 $700,150

9 8 Low $1,000,000 51 $41,011,150

10 19 Medium $43,000 10 $1,000,210

Social Posts, Competitive Activity, Marketing Spend and Promotion Count are all
independent variables, since they can help us explain, or predict, monthly Revenue

Syed Ali Ameer

 REGRESSION 101

EXAMPLE: Using marketing and sales data (sample below) to predict revenue for a given month

Social Competitive Marketing Promotion

Month ID Revenue
Posts Activity Spend Count

1 30 High $130,000 12 $1,300,050

2 15 Low $600,000 5 $11,233,310

3 11 Medium $15,000 10 $1,112,050 We’ll use records with observed values for
4 22 Medium $705,000 11 $1,582,077 both independent and dependent variables
5 41 High $3,000 3 $1,889,053 to “train” our regression model...
6 5 High $3,500 3 $1,200,089

7 23 Low $280,000 2 $1,300,080

8 12 Low $120,000 11 $700,150

9 8 Low $1,000,000 51 $41,011,150

10 19 Medium $43,000 10 $1,000,210

11 7 High $320,112 8 ??? ...then apply that model to new, unobserved values
containing IVs but no DV
This is what our regression model will predict!
Syed Ali Ameer
Ordinary Least Squares Method (OLS)

Syed Ali Ameer

Ordinary Least Squares Method (OLS)

Syed Ali Ameer

Ordinary Least Squares Method (OLS)

Syed Ali Ameer

Ordinary Least Squares Method (OLS)

Syed Ali Ameer

Ordinary Least Squares Method (OLS)

Syed Ali Ameer

 LINEAR RELATIONSHIPS

A linear relationship means that when you increase or decrease your

Linear Relationships X variable, your Y variable increases or decreases at a steady rate
• If you plot a linear relationship over many X/Y values, it will look like a straight line

Least Squared Error

• Mathematically, linear relationships can be described using the following equation:

Y-intercept X value
(x ,y )
Univariate Linear
Regression y = 𝘢 + βx β
Y value Slope (rise/run) 𝘢
Multiple Linear
Regression • NOTE: Not all relationships are linear (more on that later!)

Non-Linear Common examples:

Regression
• Taxi Mileage (X) and Total Fare (Y)
• Units Sold (X) and Total Revenue (Y)

Syed Ali Ameer

 LINEAR RELATIONSHIPS

Positive Linear Negative Linear

Linear Relationships

Least Squared Error

Univariate Linear
Regression
Non-Linear (logarithmic) No Relationship

Multiple Linear
Regression

Non-Linear
Regression

Sometimes there’s no relationship at all!

Syed Ali Ameer
 LINEAR RELATIONSHIPS

Linear Relationships 50

40
Consider a line that fits
Least Squared Error every single point in the plot
30
This is known as a perfectly
linear relationship
Univariate Linear 20

Regression
10

Multiple Linear
Regression 10 20 30 40 50 60 70 80

Non-Linear
Regression
In this case you can simply calculate the exact value of Y for any given value
of X (no Machine Learning needed, just simple math!)

Syed Ali Ameer

 LINEAR RELATIONSHIPS

Linear Relationships 50

40
In the real world, things
Least Squared Error aren’t quite so simple
30
When you add variance, it
means that many different
Univariate Linear 20
lines could potentially fit
Regression
through the plot
10

Multiple Linear
Regression 10 20 30 40 50 60 70 80

Non-Linear
Regression
To find the equation of the line with the best possible fit, we can use a
technique known as least squared error or “least squares”

Syed Ali Ameer

 LEAST SQUARED ERROR

Least squared error is used to mathematically determine the line that

Linear Relationships
best fits through a series of data points
• Imagine drawing a line through a scatterplot, and measuring the distance between
Least Squared Error your line and each point (these distances are called errors, or residuals)

• Now square each of those residuals, add them all up, and adjust your line until
you’ve minimized that sum; this is how least squares works!
Univariate Linear
Regression

Why “squared” error?

Multiple Linear
Regression • Squaring the residuals converts them into positive values, and prevents positive
and negative distances from cancelling each other out (this helps the model
optimize more efficiently, too)
Non-Linear
Regression

Syed Ali Ameer

 LEAST SQUARED ERROR

Linear Relationships 50
X Y (actual) Y (line) Error Sq. Error

10 10 15 5 25
40 20 25 20 -5 25

30 20 25 5 25
Least Squared Error
30 35 30 27.5 -2.5 6.25

40 40 30 -10 100

50 15 35 20 400
Univariate Linear 20
60 40 40 0 0
Regression
65 30 42.5 12.5 156.25
10
70 50 45 -5 25

50 10 100
Multiple Linear 80 40

Regression 10 20 30 40 50 60 70 80

Non-Linear
Regression STEP 1: Plot each data point on a scatterplot, and record the X and Y values

Syed Ali Ameer

 LEAST SQUARED ERROR

y = 10 + 0.5x
Linear Relationships 50
X Y (actual) Y (line) Error Sq. Error

10 10 15 5 25
40 20 25 20 -5 25

30 20 25 5 25
Least Squared Error
30 35 30 27.5 -2.5 6.25

40 40 30 -10 100

50 15 35 20 400
Univariate Linear 20
60 40 40 0 0
Regression
65 30 42.5 12.5 156.25
10
70 50 45 -5 25

50 10 100
Multiple Linear 80 40

Regression 10 20 30 40 50 60 70 80

Non-Linear
Regression STEP 2: Draw a straight line through the points in the scatterplot, and
calculate the Y values derived by your linear equation

Syed Ali Ameer

 LEAST SQUARED ERROR

y = 10 + 0.5x
Linear Relationships 50
X Y (actual) Y (line) Error Sq. Error

10 10 15 5 25
40 20 25 20 -5 25

30 20 25 5 25
Least Squared Error
30 35 30 27.5 -2.5 6.25

40 40 30 -10 100

50 15 35 20 400
Univariate Linear 20
60 40 40 0 0
Regression
65 30 42.5 12.5 156.25
10
70 50 45 -5 25

10 100
Multiple Linear 80 40 50

Regression 10 20 30 40 50 60 70 80

Non-Linear
Regression STEP 3: For each value of X, calculate the error (or residual) by comparing
the actual Y value against the Y value produced by your linear equation

Syed Ali Ameer

 LEAST SQUARED ERROR

y = 10 + 0.5x
Linear Relationships 50
X Y (actual) Y (line) Error Sq. Error

10 10 15 5 25
40 20 25 20 -5 25

30 20 25 5 25
Least Squared Error
30 35 30 27.5 -2.5 6.25

40 40 30 -10 100

50 15 35 20 400
Univariate Linear 20
60 40 40 0 0
Regression
65 30 42.5 12.5 156.25
10
70 50 45 -5 25

100
Multiple Linear 80 40 50 10

Regression

=
862.5
10 20 30 40 50 60 70 80
SUM OF SQUARED ERROR:

Non-Linear STEP 4: Square each individual residual and add them up to determine the
Regression
sum of squared error (SSE)
• This defines exactly how well your line “fits” the plot (or in other words, how well
the linear equation describes the relationship between X and Y)
Syed Ali Ameer
 LEAST SQUARED ERROR

y = 12 + 0.4x
Linear Relationships 50
X Y (actual) Y (line) Error Sq. Error

10 10 16 6 36
40 20 25 20 -5 25

30 20 24 4 16
Least Squared Error
30 35 30 26 -4 16

40 40 28 -12 144

50 15 32 17 289
Univariate Linear 20
60 40 36 -4 16
Regression
65 30 38 8 64
10
70 50 40 -10 100

Multiple Linear 80 40 44 4 16

Regression

=
10 20 30 40 50 60
722
70 80
SUM OF SQUARED ERROR:

Non-Linear STEP 5: Plot a new line, repeat Steps 1-4, and continue the process until
Regression
you’ve found the line that minimizes the sum of squared error
• This is where Machine Learning comes in; human trial-and-error is completely
impractical, but machines can find an optimal linear equation in seconds
Syed Ali Ameer
 UNIVARIATE LINEAR REGRESSION

Univariate (“simple”) linear regression is used for predicting a numerical

Linear Relationships
output (DV) based on a single independent variable
• Univariate linear regression is simply an extension of least squares; you use the
Least Squared Error linear equation that minimizes SSE to predict an output (Y) for any given input (X)

Coefficient/parameter
Univariate Linear
Dependent variable (DV) (sensitivity of Y to X)
Regression This is just the equation of
y =𝘢 + βx + 𝜀 Error/residual a line, plus an error term

Multiple Linear
Y-intercept Independent variable (IV)
Regression

Non-Linear
Regression
Simple linear regression is rarely used on its own; think of it as a primer for
understanding more complex topics like non-linear and multiple regression

Syed Ali Ameer

 MULTIPLE LINEAR REGRESSION

Linear Relationships Can we use more than one IV to predict the DV?

Multiple linear regression is used for predicting a numerical output

Least Squared Error
(DV) based on multiple independent variables
• In its simplest form, multiple linear regression is simply univariate linear regression
Univariate Linear
Regression
with additional x variables:

Multiple Linear DV
Regression
y =𝘢 + β1x1 + β2x2 + β3x3 + … + βnxn + 𝜀 Error/
residual

Non-Linear Y-intercept
Regression Instead of just 1 IV, we have a whole set of independent variables
(and associated coefficients/weights) to help explain our DV

Syed Ali Ameer

 MULTIPLE LINEAR REGRESSION

To visualize how multiple regression works with 2 independent variables,

Linear Relationships
imagine fitting a plane (vs. a line) through a 3D scatterplot:

Least Squared Error

Univariate Linear
Regression

Multiple Linear
Regression

Non-Linear
Regression
Multiple regression can scale well beyond 2 variables, but this is where
visual analysis breaks down (and why we need machine learning!)
Syed Ali Ameer
 MULTIPLE LINEAR REGRESSION

EXAMPLE You are preparing to list a new property on AirBnB, and want to estimate
(or predict) an appropriate price using the listing data below
Linear Relationships

Least Squared Error

Univariate Linear
Regression

Multiple Linear
Regression

Non-Linear
Regression

Syed Ali Ameer

 MULTIPLE LINEAR REGRESSION

EXAMPLE You are preparing to list a new property on AirBnB, and want to estimate
(or predict) an appropriate price using the listing data below
Linear Relationships

MODEL 1: Predict price (Y) based on accommodation (X1):

Least Squared Error Y =55.71 + (16.6*X1)

MODEL 2: Predict price (Y) based on accommodation (X1) and number of bedrooms (X2):
Univariate Linear
Regression Y =52.59 + (15.4*X1) + (5.1*X2)

Multiple Linear MODEL 3: Predict price (Y) based on accommodation (X1), number of bedrooms (X2), and room
Regression type ([Link] (X3), [Link] (X4), [Link] (X5)):

Y =43.82 + (5.7*X1) + (5.1*X2) + (63.7*X3) + (65.4*X4) + (9.8*X5)

Non-Linear
Regression MODEL 4: Predict price (Y) based on accommodation (X1), number of bedrooms (X2), room type
([Link] (X3), [Link] (X4), [Link] (X5)), and district (manhattan (X6), Brooklyn (X7)):

Y =26.1 + (6.3*X1) + (6.7*X2) + (60.5*X3) + (54.5*X4) + (10.6*X5) + (28.5*X6) + 9.8*X7)

Syed Ali Ameer

 MULTIPLE LINEAR REGRESSION

EXAMPLE You are preparing to list a new property on AirBnB, and want to estimate
(or predict) an appropriate price using the listing data below
Linear Relationships

Least Squared Error

Univariate Linear
Regression

Multiple Linear
Regression

Non-Linear
Regression

Model 1 Model 2 Model 3 Model 4

Sum of Squared Error (SSE): 226,577 224,259 172,201 158,591
Syed Ali Ameer
 NON-LINEAR REGRESSION

Linear Relationships What if the relationship between variables isn’t linear?

Non-linear regression is used when variables have a non-linear

Least Squared Error relationship, but can be transformed to create a linear one
• This works exactly like linear regression, except you use transformed versions of
Univariate Linear your dependent or independent variables:
Regression
Coefficient/parameter
Dependent variable (DV)
Multiple Linear
Regression y = 𝘢 + β*ln(x) + 𝜀 Error/residual

Y-intercept
Log-transformed independent variable (IV)
Non-Linear
Regression
All we’re really doing is transforming the data to create linear relationships between each IV and the DV,
then applying a standard linear regression model using those transformed values

Syed Ali Ameer

 NON-LINEAR REGRESSION

EXAMPLE #1 You are predicting Sales (Y) using Marketing Spend (X). As you spend
more on marketing, the impact on sales eventually begins to diminish.
Linear Relationships

Least Squared Error

Univariate Linear
Regression

Multiple Linear
Regression
The relationship between Sales and Marketing ...but the relationship between Sales and the
Spend is non-linear (logarithmic)... log of Marketing Spend is linear!
Non-Linear
Regression
y = 𝘢 + βx + 𝜀 y = 𝘢 + β*ln(x) + 𝜀

Syed Ali Ameer

 NON-LINEAR REGRESSION

EXAMPLE #2 You are predicting population growth (Y) over time (X) and notice an
increasing rate of growth as the population size increases.
Linear Relationships

Least Squared Error

Univariate Linear
Regression

Multiple Linear
Regression
The relationship between Time and Population ...but the relationship between Time and the
is non-linear (exponential)... log of Population is linear!
Non-Linear
Regression y = 𝘢 + βx + 𝜀 ln(y) = 𝘢 + βx + 𝜀

NOTE: There are multiple ways to transform variables based on the type of relationship
(log, exponential, cubic, etc.), and multiple techniques to model them (more on that later!)
Syed Ali Ameer
 REGRESSION WORKFLOW

Measurement Planning

Remember, these steps ALWAYS come first

Preliminary QA Before building a model, you should have a clear measurement plan (KPIs, project scope, desired outcome, etc.) and an
understanding of the data at hand (variable types, table structure, data quality, profiling metrics, etc.)
Data Profiling

Feature Engineering Data Splitting Model Training Selection & Tuning

Adding new, calculated Splitting records into Building regression models Choosing the best
variables (or “features”) to “Training” and “Test” data from Training data and performing model for a given
a data set based on sets, to validate accuracy applying to Test data to prediction, and tuning it to
existing fields and avoid overfitting maximize prediction accuracy prevent drift over time

Syed Ali Ameer

 FEATURE ENGINEERING

Feature engineering is the process of enriching a data set by creating additional

independent variables based on existing fields
• New features can help improve the accuracy and predictive power of your ML models
• Feature engineering is often used to convert fields into “model-friendly” formats; for example,
one-hot encoding transforms categorical variables into individual, binary (1/0) fields

Original features Engineered features

Social Competitive Marketing Promotion Competitive Competitive Competitive Promotion >10 &
Month ID Revenue LogSpend
Posts Activity Spend Count High Medium Low Social >25

1 30 High $130,000 12 $1,300,050 1 0 0 1 11.7

2 15 Low $600,000 5 $11,233,310 0 0 1 0 13.3

3 8 Medium $15,000 10 $1,112,050 0 1 0 0 9.6

4 22 Medium $705,000 11 $1,582,077 0 1 0 0 13.5

5 41 High $3,000 3 $1,889,053 1 0 0 0 8.0

Syed Ali Ameer

 DATA SPLITTING

Splitting is the process of partitioning data into separate sets of records for the
purpose of training and testing machine learning models
• As a rule of thumb, ~70-80% of your data will be used for Training (which is what your model
learns from), and ~20-30% will be reserved for Testing (to validate the model’s accuracy)

Social Competitive Marketing Promotion

Month ID Revenue
Posts Activity Spend Count

1 30 High $130,000 12 $1,300,050

Test data is NOT used to optimize models
2 15 Low $600,000 5 $11,233,310

3 11 Medium $15,000 10 $1,112,050 Using Training data for optimization and Test data
Training for validation ensures that your model can
4 22 Medium $705,000 11 $1,582,077
data accurately predict both known and unknown values,
5 41 High $3,000 3 $1,889,053 which helps to prevent overfitting
6 5 High $3,500 3 $1,200,089

7 23 Low $280,000 2 $1,300,080

8 12 Low $120,000 11 $700,150

Test
9 8 Low $1,000,000 51 $41,011,150
data
10 19 Medium $43,000 10 $1,000,210

Syed Ali Ameer

 OVERFITTING
Splitting is primarily used to avoid overfitting, which is when a model predicts known (Training) data very
well but unknown (Test) data poorly
• Think of overfitting like memorizing the answers to a test instead of actually learning the material; you’ll ace the
test, but lack the ability to generalize and apply your knowledge to unfamiliar questions

OVERFIT model WELL-FIT model UNDERFIT model

• Models the Training data too well • Models the Training data just right • Doesn’t model Training data well enough
• Doesn’t generalize well to Test data • Generalizes well to Test data • Doesn’t generalize well to Test data (high
(high variance, low bias) (balance of bias & variance) bias, low variance)
R-Squared

Syed Ali Ameer

R-Squared

Syed Ali Ameer

 R-SQUARED

R-Squared measures how well your model explains the variance in the
dependent variable you are predicting
• The higher the R-Squared, the “better” your model predicts variance in the DV
and the more confident you can be in the accuracy of your predictions

• Adjusted R-Squared is often used as it “penalizes” the R-squared value based

on the number of variables included in the model

Total distance between predicted and actual

values, squared (aka squared error)

SSE = Sum of Squared Error = ෍(𝑦 𝑖 − 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑖)2

R2 =1 – 𝑖

TSS = Total Sum of Squares = ෍(𝑦 𝑖 − 𝑦ത)2

𝑖

Total distance between each y value and the mean,

squared (basically variance, without dividing by n)

Syed Ali Ameer

 R-SQUARED EXAMPLE

y = 12 + 0.4x
50 x y prediction (y–prediction)2 y (y– y )2

10 10 16 36 30 400

40 20 25 20 25 30 25

30 20 24 16 30 100

30 35 30 26 16 30 0

40 40 28 144 30 100

20 50 15 32 289 30 225

60 40 36 16 30 100

10 65 30 38 64 30 0

70 50 40 100 30 400

80 40 44 16 30 100
10 20 30 40 50 60 70 80

=
722 1,450
SSE TSS

SSE = ෍(𝑦 𝑖 − 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑖)

2
722
R2 =1 – 𝑖
1– = 0.502
TSS = ෍(𝑦 𝑖 − 𝑦ത)2 1,450
𝑖

Syed Ali Ameer

ADJUSTED R-SQUARED

ADJUSTED R-SQUARED

A criticism of r-squared is that it will never decrease as new variables are added
Adjusted r-squared corrects this by penalizing new variables added to a model
• This measure has no meaning whatsoever, but it helps as a variable selection tool

EXAMPLE Adding a random column to a sample of 100 diamonds

R2 increased
but adjusted
R2 didn’t

In this case, the p-value could also tell us to remove this variable
(other times, a variable can be significant and lower adjusted R2)

Syed Ali Ameer

VARIABLE SELECTION

VARIABLE SELECTION

Variable selection is a critical part of the modeling process that helps reduce
complexity by only including features that help meet the model’s goal
• Do new variables improve model accuracy? (critical for prediction)
• Is each variable statistically significant? (critical for inference)
• Do new variables make the model more challenging to interpret or explain to stakeholders?

EXAMPLE Using all the possible features to predict diamond price

R2 improved to .859 compared to the simple regression model (.849)

If the goal is inference,a single variable model may be a good choice!

“x”still has a negative coefficient

This helps R2 but makes the model hard to explain

“z”has a p-value greaterthan alpha (0.347 > 0.05)

We should drop this variable from the model

Syed Ali Ameer

Residuals Sum of Squares (RSS)

Syed Ali Ameer

MEAN ERROR METRICS

MEAN ERROR METRICS

Mean error metrics measure how well your regression model fits in in the units of
our target, as opposed to how well it explains variance (like R-Squared)
• The most common are Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE)
• They are used to compare model fit across models (the lower the better!)

Average of the absolute distance Average of the squareddistance Square root of Mean Squared Error,
between actual& predicted values between actual & predicted values to return to the target’sunits(likeMAE)

σ 𝑖 |𝑦𝑖 σ 𝑖 (𝑦 𝑖 − σ 𝑖 (𝑦𝑖 − 𝑦ො𝑖

− 𝑦ො 𝑛 𝑖| 𝑦ො 𝑖)2𝑛 )2 𝑛
RMSE is more sensitive to large outliers, so it is preferred over
MAE in situations where they are particularly undesirable

Syed Ali Ameer

MEAN ERROR METRICS

MEAN ERROR METRICS

You can use [Link] to calculate MAE and MSE for your model
• [Link].mean_absolute_error(y_actual, y_predicted)
• [Link].mean_squared_error(y_actual, y_predicted)

The simple linear regression model has an

average prediction error of ~$1,000

The outliers in the dataset are making

RMSE around 50% larger than MAE

This returns RMSE

instead of MSE

The multiple linear regression model

performs better across both metrics

RMSE will always be bigger than MAE

Syed Ali Ameer

 MEAN ERROR METRICS

When should I use each type of error metric?

• Mean Squared Error (MSE) is particularly useful when outliers or

extreme values are important to predict

• Mean Absolute Error (MAE) is useful if you want to minimize the impact
of outliers on model selection

• Mean Absolute Percentage Error (MAPE) is useful when your DV is

on a very large scale, or if you want to compare models on a more
intuitive scale

PRO TIP: In general we recommend considering all of them, since they can be
calculated instantly and each provide helpful context into model performance

Syed Ali Ameer

MEAN ERROR METRICS

MEAN ERROR METRICS

Syed Ali Ameer