ITCS 6156/8156 – Machine Learning
Spring 2024 – Lecture 2
Supervised Learning & Linear
Regression
Christian Kuemmerle
Types of Machine Learning:
• Supervised Learning:
Constructing computer programs that learn from experience to perform
well on a given task.
E.g.: Predict spam/no-spam based on dataset correctly labeled as spam/no-spam.
• Unsupervised Learning:
Given data, try to discover common patterns/ structure or sub-spaces.
E.g.: Cluster news articles by topic
• Reinforcement Learning:
Try to learn from delayed feedback
E.g.: Robots learn to walk; learning to play chess well from playing chess
Mathematical formulation of the pipeline
Dataset:
𝒟 = {(x 1 , y1), …, (xn, y n)}, xi ∈ ℝd, yi ∈ 𝒞 (xi, yi) ∼ 𝒫 ( X , Y ) i . i . d .
Hypothesis:
e.g., a neural network-based
h : ℝd ↦ 𝒞 classifier that maps image to label
of cat or dog
Hypothesis class E.g.:
- a large family of NNs with
ℋ = {h} different parameters
- all linear models
- all quadratic models
MathematicalFormulation
8:d-dimensional feature space
xi:inpat/feature
vector of
in the sample/data point
-
y::
ith Label/response variable output variable
· 2: set oflabels setofoutputs
·
P(x,y):data distribution
Examples: Binary classification e=
40,1) span (no span
·
Multiclass Classification classification
C (1. k) image
·
=
.
.,
.
Regression e=R [Link]
of
temperatore
patient
·
heightofperson
·house
price
Examples of hypothesis
Inductive bias (i.e., assumptions) encoded in the hypothesis class
Ex: is a linear function h(x) = sign(w⊤x); Ex: is nonlinear h(x) =
ℋ contains all possible linear functions sign(w⊤(ReLU(Ax)));
ℋ contains all possible one-layer NN
Do we need to make assumptions on the data?
No free lunch theorem says that we must make such assumptions
Informal theorem: for any machine learning algorithm 𝒜 , there must
exist a task 𝒫 on which it will fail
We use prior knowledge (i.e., we believe
linear function is enough) to design an ML
algorithm here
House Prices in Portland