Source: Business Data Science: Combining Machine Learning and Economics to Optimize, Automate, and Accelerate Business

Decisions, 1st Edition

2.2. Logistic Regression

Binary responses arise from a number of prediction targets:

Will this person pay their bills or default?

Is this a thumbs-up or thumbs-down review?

Will the Edmonton Oilers win or lose this game?

Is the writer a Republican or Democrat?

Even when the response of interest is not binary (e.g., revenue), it will sometimes be that your decision-relevant information is
binary (e.g., profit versus loss) and it is simplest to think in these terms.

Recall the general linear model specification: [y | x] = f (x′β). When the response y is 0 or 1, the conditional mean becomes

E [y|x] = p (y = 1|x) × 1 + p (y = 0|x) × 0 = p (y = 1|x) .

Therefore, the expectation you're modeling is a probability. This implies that you need to choose the link function f (x′β) to give
values between zero and one:

p (y = 1|x) = f (β0 + β1x1 … + βp xp ) .

Logistic regression uses a logit link function:

ex′β exp [β0 + β1x1 … + βp xd]

(2.16)

See Figure 2.8 for a picture of the logit function. To see how this link works, consider extreme values forx′β. At large negative
values, f(−∞) = 0/(1 + 0) = 0 and y = 1 has zero probability. At large positive values, f(∞) = ∞/(∞ + 1) = 1 and y = 1 is
guaranteed. Thus, the logit link maps from the "real line" of numbers to the [0, 1] space of probabilities.

© McGraw-Hill Education. All rights reserved. Any use is subject to the Terms of Use, Privacy Notice and copyright information.
 Figure 2.8 A logistic link function.

To interpret the β coefficients, writing p = p(y = 1|x) and a little algebra shows that

log [ ] = β0 + β1x1 … + βp xp .
p
1−p

Thus, logistic regression is a linear model for log odds. The odds of an event are the probability that it happens over the
probability that it doesn't. For example, if an event has a 1/4 probability, then its odds are "one in three," or 1/3. You should get
used to thinking about uncertainty in terms of odds—much of your modeling will occur on this scale. Following the same logic
as we had for log linear models earlier, eβk is then interpretable as the multiplicative effect[1] for a unit increase in xk on the odds
for the event y = 1.

For our first logistic regression example, we'll build a filter for email "spam"—junk mail that can be ignored. Every time an email
arrives, your inbox performs binary regression: Is this spam or not spam? We'll train our filter by fitting logistic regression to
previous emails.

As training data, [Link] has for 4600 emails (about 1800 spam) indicators for the presence of 54 keywords or characters
(e.g., free or ! ), counts for capitalized letters (total number and longest continuous block length), and a spam variable for
whether or not each email has been tagged as spam by a human reader.

Notice that email #1, which contained the word free and had a block of 61 capitalized letters, was tagged as spam. Email
#4000, with its more modest sequence of 26 capital letters, is not spam.

Logistic regression is easy in R: you use the glm command as you would for linear regression, except that the response is
now binary and you add the argument family='binomial' . The response variable can take a number of forms: numeric ( 0/1 ),
logical ( TRUE/FALSE ), factor (e.g., win/lose ), or even a two-column binary matrix. In the email data, we havey = 1 for spam
and y = 0 for important emails.

© McGraw-Hill Education. All rights reserved. Any use is subject to the Terms of Use, Privacy Notice and copyright information.
Note that the formula " y ~." is shorthand for "regress y on to all variables in the data." Take a look at one of the large positive
coefficients in our fit:

Thus, the odds that an email is spam increase almost 5 times if that email contains the word free . On the other hand, we see
next that if the email contains the word george , the odds of it being spam drop by a factor larger than 300.

This is an old dataset collected from the inbox of a guy named George. Spammers were not very sophisticated in the 1990s, so
emails containing your name were most likely not spam.

When you run the spam regression, R warns that fitted probabilities numerically 0 or 1 occurred . You need not worry: this
just says that the regression is able to fit some data points exactly—for example, a spam email is modeled as having a 100%
probability of being spam. This situation is called perfect separation; it can mess with your standard errors but is largely benign.
It is a symptom of overfit, which is covered in Chapter 3.

As with linear regression, prediction for logistic regression is easy after you've fit the model with glm . We call the predict
function on our fitted glm object and provide some newdata , with the same variable names as the training data, at the
locations where we want to predict. The output will be x′β̂ for each row x of mynewdata .

Of course, these are not probabilities. To get those, you need to transform through the logit link asex′β̂ /(1 + ex′β̂ ). R's predict
function lets you add the type='response' argument to get predictions on the scale of the response (i.e., in [0, 1] probability
space).

The first email (true spam) has an 88% chance of being spam, while email #4000 (not spam) has a 15% chance of being spam—
in other words, an 85% chance of being important email that George wants to read.

Logistic regression is very similar to linear regression. You still use glm , and you just need to adjust your thinking to odds
instead of means. In the next section, we introduce the important ideas of deviance and likelihood, which tie the estimation
techniques behind linear and logistic regression together into a single framework. Gaining this unified view is essential for our
later work on penalized models and machine learning.

© McGraw-Hill Education. All rights reserved. Any use is subject to the Terms of Use, Privacy Notice and copyright information.
[1] What happens if βk = 0? Nothing!

© McGraw-Hill Education. All rights reserved. Any use is subject to the Terms of Use, Privacy Notice and copyright information.