4.
Classification:
logistic regression
Antonio Bahamonde
Departamento de Informática
Contents
• Classification and logistic regression (Main notes, chapter 2)
Classification
• We will focus on the binary classification problem in which y can take on
only two values, 0 and 1. (Most of what we say here will also generalize to
the multiple-class case.)
• (i)
For instance, if we are trying to build a spam classifier for email, then x may be some
features of a piece of email, and y(i) may be 1 if it is a piece of spam mail, and 0 otherwise. 0
is also called the negative class, and 1 the positive class, and they are sometimes also
denoted by the symbols “-” and “+.”
• Given x(i), the corresponding y(i) is also called the label for the training
example.
Logistic regression
We could approach the classification problem ignoring the fact that y is
discrete-valued. It doesn’t make sense for hθ(x) to take values larger than 1 or
smaller than 0 when we know that y ∈ {0, 1}. To fix this, let’s change the form
of hθ(x)
logistic function or the sigmoid function
Before moving on, here’s a useful property of the derivative of the sigmoid function, which
Logistic regression
Let’s endow our classification model with a set of probabilistic assumptions,
and then fit the parameters via maximum likelihood.
Logistic regression
Gradient descent update rule
Logistic regression
Gradient descent update rule
it looks identical than LMS; but this is not the same algorithm, because hθ(x(i)) is now
defined as a non-linear function of θTx(i). Nonetheless, it’s a little surprising that we end
up with the same update rule for a rather different algorithm and learning problem.
Logistic regression. Conclusion
• La regresión logistica es un algoritmo de aprendizaje automático que se
aplica a tareas de clasificación binaria. Aprende (construye) una hipótesis
que para cada entrada x estima las probabilidades de que pertenezca a
cada una de las clases.
• En este tema se vio como se deriva el aprendizaje de la hipótesis de la
regresión logística. Para ello se propuso estimaciones de las
probabilidades mediante una sigmoide (función lógistica). Los parámetros
de la hipótesis se aprendieron mediante descenso del gradiente de la
función de máxima verosimilitud de cada tarea de clasifiación (conjunto
de entrenamiento).
Logistic regression. Conclusion
• Logistic regression is a machine learning algorithm that is applied to
binary classification tasks. It learns (builds) a hypothesis that, for each
entry x, estimates the probabilities that it belongs to each of the classes.
• In this topic it was seen how the learning of the hypothesis of logistic
regression is derived. For this, estimates of the probabilities were
proposed through a sigmoide (logistic function). The parameters of the
hypothesis were learned by gradient descent of the maximum likelihood
function of each classification task (training set).
Separation rule
y = 1 ⬄ h (x) > 0.5 ⬄ 𝛳Tx > 0
Separation rule
y = 1 ⬄ h (x) > 0.5 ⬄ 𝛳Tx > 0
𝛳
Separation rule
y = 1 ⬄ h (x) > 0.5 ⬄ 𝛳Tx > 0
xi ∈ {0, 1}; x0 = 1
𝛳Tx = 15* x1+ 25*x2+ 30*x3+ 30*x4 –50*x0
Multi-class classification
(softmax)
Multi-class classification
In this case, p(y | x; θ) is a distribution over k possible discrete outcomes and
is thus a multinomial distribution. Recall that a multinomial distribution
involves k numbers 𝝓 , . . . ,𝝓k specifying the probability of each output
1
Multi-class classification
We introduce k groups of parameters θ ,...,θ , each of them being a vector in
1 k
Rd. Intuitively, we would like to use θ Tx, . . . , θ Tx to represent the
1 k
probabilities. But
θ Tx, . . . , θ Tx are not necessarily in [0, 1]
1 k
Their sum is not necessarily 1
Multi-class classification
t with k =2
Binary is tha