DataCamp Supervised
Learning in R: Regression
SUPERVISED LEARNING IN R: REGRESSION
Logistic regression to
predict probabilities
Nina Zumel and John Mount
Win-Vector LLC
DataCamp Supervised Learning in R: Regression
Predicting Probabilities
Predicting whether an event occurs (yes/no): classification
Predicting the probability that an event occurs: regression
Linear regression: predicts values in [−∞, ∞]
Probabilities: limited to [0,1] interval
So we'll call it non-linear
DataCamp Supervised Learning in R: Regression
Example: Predicting Duchenne Muscular Dystrophy
(DMD)
outcome: has_dmd
inputs: CK, H
DataCamp Supervised Learning in R: Regression
A Linear Regression Model
> model <- lm(has_dmd ~ CK + H,
+ data = train) Model predicts values outside
> test$pred <- predict( the range [0:1]
+ model,
+ newdata = test
+ )
outcome: has_dmd ∈ {0,1}
0: FALSE
1: TRUE
DataCamp Supervised Learning in R: Regression
Logistic Regression
p
log( ) = β0 + β1 x1 + β2 x2 + ...
1−p
glm(formula, data, family = binomial)
Generalized linear model
Assumes inputs additive, linear in log-odds: log(p/(1 − p))
family: describes error distribution of the model
logistic regression: family = binomial
DataCamp Supervised Learning in R: Regression
DMD model
> model <- glm(has_dmd ~ CK + H, data = train, family = binomial)
outcome: two classes, e.g. a and b
model returns P rob(b)
Recommend: 0/1 or FALSE/TRUE
DataCamp Supervised Learning in R: Regression
Interpreting Logistic Regression Models
> model
## Call: glm(formula = has_dmd ~ CK + H, family = binomial, data = train)
##
## Coefficients:
## (Intercept) CK H
## -16.22046 0.07128 0.12552
##
## Degrees of Freedom: 86 Total (i.e. Null); 84 Residual
## Null Deviance: 110.8
## Residual Deviance: 45.16 AIC: 51.16
DataCamp Supervised Learning in R: Regression
Predicting with a glm() model
predict(model, newdata, type = "response")
newdata: by default, training data
To get probabilities: use type = "response"
By default: returns log-odds
DataCamp Supervised Learning in R: Regression
DMD Model
> model <- glm(has_dmd ~ CK + H, data = train, family = binomial)
> test$pred <- predict(model, newdata = test, type = "response")
DataCamp Supervised Learning in R: Regression
2
Evaluating a logistic regression model: pseudo-R
2 RSS
R =1−
SS T ot
2 deviance
pseudoR = 1 −
[Link]
Deviance: analogous to variance (RSS)
Null deviance: Similar to SS T ot
pseudo R^2: Deviance explained
DataCamp Supervised Learning in R: Regression
2
Pseudo-R on Training data
Using broom::glance()
> glance(model) %>%
+ summarize(pR2 = 1 - deviance/[Link])
## pseudoR2
## 1 0.5922402
Using sigr::wrapChiSqTest()
> wrapChiSqTest(model)
## "... pseudo-R2=0.59 ..."
DataCamp Supervised Learning in R: Regression
2
Pseudo-R on Test data
# Test data
> test %>%
+ mutate(pred = predict(model, newdata = test, type = "response")) %>%
+ wrapChiSqTest("pred", "has_dmd", TRUE)
Arguments:
data frame
prediction column name
outcome column name
target value (target event)
DataCamp Supervised Learning in R: Regression
The Gain Curve Plot
> GainCurvePlot(test, "pred","has_dmd", "DMD model on test")
DataCamp Supervised Learning in R: Regression
SUPERVISED LEARNING IN R: REGRESSION
Let's practice!
DataCamp Supervised Learning in R: Regression
SUPERVISED LEARNING IN R: REGRESSION
Poisson and
quasipoisson
regression to predict
Nina Zumel and John Mount
Win-Vector, LLC counts
DataCamp Supervised Learning in R: Regression
Predicting Counts
Linear regression: predicts values in [−∞, ∞]
Counts: integers in range [0, ∞]
DataCamp Supervised Learning in R: Regression
Poisson/Quasipoisson Regression
glm(formula, data, family)
family: either poisson or quasipoisson
inputs additive and linear in log(count)
DataCamp Supervised Learning in R: Regression
Poisson/Quasipoisson Regression
glm(formula, data, family)
family: either poisson or quasipoisson
inputs additive and linear in log(count)
outcome: integer
counts: e.g. number of traffic tickets a driver gets
rates: e.g. number of website hits/day
prediction: expected rate or intensity (not integral)
expected # traffic tickets; expected hits/day
DataCamp Supervised Learning in R: Regression
Poisson vs. Quasipoisson
Poisson assumes that mean(y) = var(y)
If var(y) much different from mean(y) - quasipoisson
Generally requires a large sample size
If rates/counts >> 0 - regular regression is fine
DataCamp Supervised Learning in R: Regression
Example: Predicting Bike Rentals
DataCamp Supervised Learning in R: Regression
Fit the model
> bikesJan %>%
+ summarize(mean = mean(cnt), var = var(cnt))
## mean var
## 1 130.5587 14351.25
Since var(cnt) >> mean(cnt) → use quasipoisson
> fmla <- cnt ~ hr + holiday + workingday +
+ weathersit + temp + atemp + hum + windspeed
> model <- glm(fmla, data = bikesJan, family = quasipoisson)
DataCamp Supervised Learning in R: Regression
Check model fit
2 deviance
pseudoR = 1 −
[Link]
> glance(model) %>%
+ summarize(pseudoR2 = 1 - deviance/[Link])
## pseudoR2
## 1 0.7654358
DataCamp Supervised Learning in R: Regression
Predicting from the model
> predict(model, newdata = bikesFeb, type = "response")
DataCamp Supervised Learning in R: Regression
Evaluate the model
You can evaluate count models by RMSE
> bikesFeb %>%
+ mutate(residual = pred - cnt) %>%
+ summarize(rmse = sqrt(mean(residual^2)))
## rmse
## 1 69.32869
> sd(bikesFeb$cnt)
[1] 134.2865
DataCamp Supervised Learning in R: Regression
Compare Predictions and Actual Outcomes
DataCamp Supervised Learning in R: Regression
SUPERVISED LEARNING IN R: REGRESSION
Let's practice!
DataCamp Supervised Learning in R: Regression
SUPERVISED LEARNING IN R: REGRESSION
GAM to learn non-
linear transformations
Nina Zumel and John Mount
Win-Vector, LLC
DataCamp Supervised Learning in R: Regression
Generalized Additive Models (GAMs)
y ∼ b0 + s1(x1) + s2(x2) + ....
DataCamp Supervised Learning in R: Regression
Learning Non-linear Relationships
DataCamp Supervised Learning in R: Regression
gam() in the mgcv package
gam(formula, family, data)
family:
gaussian (default): "regular" regression
binomial: probabilities
poisson/quasipoisson: counts
Best for larger data sets
DataCamp Supervised Learning in R: Regression
The s() function
> anx ~ s(hassles)
s() designates that variable should be non-linear
Use s() with continuous variables
More than about 10 unique values
DataCamp Supervised Learning in R: Regression
Revisit the hassles data
Model RMSE R2
(cross-val) (training)
Linear ( 7.69 0.53
hassles)
Quadratic 6.89 0.63
(hassles 2 )
Cubic ( 6.70 0.65
hassles 3 )
DataCamp Supervised Learning in R: Regression
GAM of the hassles data
> model <- gam(anx ~ s(hassles), data = hassleframe, family = gaussian)
> summary(model)
## ...
##
## R-sq.(adj) = 0.619 Deviance explained = 64.1%
## GCV = 49.132 Scale est. = 45.153 n = 40
DataCamp Supervised Learning in R: Regression
Examining the Transformations
> plot(model)
y values: predict(model, type = "terms")
DataCamp Supervised Learning in R: Regression
Predicting with the Model
> predict(model, newdata = hassleframe, type = "response")
DataCamp Supervised Learning in R: Regression
Comparing out-of-sample performance
Knowing the correct transformation is best, but GAM is useful when
transformation isn't known
Model RMSE (cross-val) R2 (training)
Linear (hassles) 7.69 0.53
Quadratic (hassles 2 ) 6.89 0.63
Cubic (hassles3 ) 6.70 0.65
GAM 7.06 0.64
Small data set → noisier GAM
DataCamp Supervised Learning in R: Regression
SUPERVISED LEARNING IN R: REGRESSION
Let's practice!