# 5803
Classi cation
Binary logistic regression
fi
# 5803
Multiple linear regression
In a regression model, we are interested in
predicting a quantitative variable (i.e. y ∈ ℝ):
⊤
ŷ = b + x w
= b + w1x1 + ⋯ + wp xp
# 5803
Classi cation
In classi cation, we are instead interested in
predicting some categorical variable
• binary classi cation: y ∈ {0,1} (or y ∈ {−1, + 1})
• multi-class: y ∈ {1,2,…, K} for K ≥ 2
multi-label: y = [y1, y2, …, yq] with
⊤
•
yj ∈ {1,2,…, K} for j = 1,…, q
fi
fi
fi
# 5803
Agenda
• Logistic function
• Binary cross-entropy loss
# 5803
Logistic function
# 5803
Toy example
# 5803
Why don’t we use MLR?
We already have a model that can predict a
quantitative response. Why don’t we use it here?
# 5803
Bernoulli model
• Since the true Y is either 0 or 1, we need to make
sure ŷ = 0 or 1.
• Our idea is to model the conditional probability
p(x) = P(Y = 1 | X = x) and determine the output of
ŷ by the value of p(x).
# 5803
Logistic function
• Note that p(x) is a probability, and thus 0 ≤ p(x) ≤ 1.
• One way to achieve this constraint is trough the
logistic function:
1
σ(t) = . (1)
1 + e −t
• Here t is a transformation of the features x; e.g.,
t = b + x⊤w.
• The function σ(t) is also called sigmoid function.
# 5803
# 5803
Properties of the logistic
function
• De nition:
t
1 e
σ(t) = =
1 + e −t 1 + e t
• Range: 0 < σ(t) < 1
• Derivative:
d
σ(t) = σ(t)(1 − σ(t)) (2)
dt
quotient rule
fi
# 5803
• Re ection and symmetry:
1 e −t
1 − σ(t) = = = σ(−t)
1 + e t 1 + e −t
Inverse:
(1 − p)
−1 p
t = σ (p) = log
fl
# 5803
Binary cross-entropy
loss
# 5803
Logistic regression by least-
squares
We might estimate θ = [b, w]⊤ by least-squares
estimation which minimizes
n
1
( )
2
R (θ) = (
⊤
)
n∑
yi − σ b + x i w .
i=1
# 5803
Likelihood function
Let X1, X2, …, Xn be a random sample. The likelihood
function is given by
L(θ | X1, …, Xn) ≜ f(X1, …, Xn; θ) .
iid
When X1, X2, …, Xn ∼ f(x | θ), the above equation
becomes
n
∏
L(θ | X1, …, Xn) = f(Xi; θ) .
i=1
# 5803
Maximum likelihood estimation
• Find the parameter that is
“most likely” to observe your
data.
• Maximizing the likelihood
function is equivalent to
maximizing the log-likelihood
function (for computational
issues) since logarithm is a
monotone function.
[Link]
maximum-likelihood-estimation-c7b4342fdbb1
# 5803
Likelihood functions for logistic
regression
Assume Yi | Xi = xi are independent sample from
Bernoulli (p(xi)), where p(xi) = σ (b + x⊤i w). The
likelihood function becomes
n
L (θ) = p(xi) [1 − p(xi)]
yi (1−yi)
∏
,
i=1
and the log-likelihood function:
n
ℓ (θ) = yi ⋅ log (p(xi)) + (1 − yi) ⋅ log (1 − p(xi)) .
∑
i=1
# 5803
Binary cross-entropy loss
Maximizing ℓ (θ) is equivalent to minimizing −ℓ (θ):
n
1
−ℓ (θ) = − [yi ⋅ log (p(xi)) + (1 − yi) ⋅ log (1 − p(xi))] .
n∑i=1
binary cross-entropy loss
# 5803
Binary cross-entropy loss
In most software implementations, the BCELoss is
often formulated as
BCE(y, p)̂ = − [y ⋅ log (p)̂ + (1 − y) ⋅ log (1 − p)̂ ], (3)
where p̂ = σ (b̂ + x⊤i ŵ ).
# 5803
MLE for logistic regression
The MLE for θ becomes the solution of
n
1
θ ̂ = arg min BCE (yi, pî ) .
θ n ∑ (4)
i=1
The above minimization is usually carried out by
some iterative algorithms, e.g., the gradient
descent algorithm.
Regularizations
We can also incorporate regularization terms such
as elastic net into equation (3). This will lead us to
regularized MLE:
n
1
̂ = arg min BCE (yi, pî ) + λ (α∥w∥1 + (1 − α)∥w∥22) .
∑
θλ,α
θ n i=1
# 5803
Logistic regression by scikit-
learn
# 5803
Summary
• Logistic regression is derived from Bernoulli
model
• logistic (sigmoid function)
• cross-entropy loss
• MLE
# 5803
To be continued
• Classi cation by logistic regression
• Evaluating the performance of a binary classi er
fi
fi
# 5803
Readings
• Chapter 19.1-19.4 of Learning Data Science