MachineLearning

Lecture 1 Introduction

Given N points Xi y i Xv y2 1X了 73 I in

iii
Geometry [Link]
sum
ofabsolute errors L iF Iwxitbyil
The line that minimizes the error would be

argmini
0，5
wb

Optimization Solve
恐占Élwxitbyīl

lal is not differentiable for all a

Absolute ER
valuesie are not differentiable everywhere
Sowemodify the Loss function to
Li Ěiwxitbyīi
Solve
no ⾔ 0

⾼L Ě 2 [Link]Ěxit Nb⼀点yi o
b i Ěyi [Link] xiij [Link]
Rewrite Loss function
L ⼆点 lwlxi [Link] __j 12
录L Ěilwlxi xl lyi [Link] xjzzlwlxi xllxi i lxi [Link] 0

w Ě lxi [Link]
無 xi [Link]
Theability to predict is Called Generation

Recipe Task Data Set Loss Model Optimization algorithmsGeneralization

 Lecture 2 Probability
babilitymeasures Start with a set ⼝
A subset X En is called an event
A probabilitymeasure R takes a subset and returns a real value

P 2 R E is the power set i.e all Subsets

of n
l P is a function that takes Subset ofn and returns a real value
0PcX ET forany X En
Push 1
lplxnYi RX RYiifxnY

Probability X Ee
[Link]
When it is discrete and finite it is possible to enumerate all elements
of a subse
Distributions For we can implement a probabilitymeasure D with anotherfunction p
any X En by
letting
Rx ⼆意 PM
Thefunction p [Link] a probability mass function or discrete probability dis
tribat
Dp ⼝ R
Ofpcw El
forany w En
Ewanplw 7

Example n 1，2，3，4.5.61

⼼ 以 吸 ⼤⼝
Tina
⼀
的
It's valid type_correct to write Pc917 or Pc 1.4
Pin PcIi 213，4，5，671 ⼆7
i

Pill 21 pc i p 2 ⼆⾔
11is an event I is not
P and p are different

et Comprehension A shorthand [Link] with constrains

Pcw 3 P4
w 311
Pcw 3 PYwiw 31
Pcwis even ⼆PclwiwEh4，611
Thevariable name doesn't matter
Always askwhat is random
 11 l 1 1 9

continuous 在 ten F is a Cumulative [Link].a

obability [Link] 0，1
zin_ 2
3
F is monotonic i.e
1an
Fix
Fyi if
下⼼
xg 㾩
A probability densityfunction pis defined as pcu ⼆ 装⼼ or Fix ⼆ pandu
We can construct a probability measure Pbyletting
Rack b ifpandu⼆Fib Fca
si R and Piz Rtakes a subset R as of input

Gaussian
[Link] T exp⼈⾔ [Link]
i ______________

iiiiiii
CDF
it PDF

We say a is drawn from a Gaussian

if
a Nun a了
业
Plan⼆ 合加 expl [Link]
xpectatīon Definition
EIXI [Link] EIxI 㮺如 ⼼

EIXI is not a function of x but a functionof p

A better notation would be
Exnpcx X

Notation plx p is not the name of the function as opposed to fix

When we have multipledistribution the convention is to [Link] name to
distinguish [Link] ply plz
Usekeywords a pcZ a
arguments
[Link] ply a
Plx a tpYx x al p takes a pointin n not a subset
 11 1 X

M⼼吐 JointDistribution PIX y

[Link] pix ⼆点pl y
x
Random Marginaldistribution or
Variables conditional distribution
N'T 管
哥
Notation plx ⼆点[Link] pxla ⼆点
_Pxyla.bz
⻢
等
的
pylx 器 [Link] ⼆
台

3agesRule pcylxn
yp [Link]
管哥 x
器
台 ⼆
器
是吕
gg
x and
independence
y
are independent
if
[Link] [Link] forany xerox and yEm
of
Bythe definition conditional probability⼝
⼆
pcylx 兴
名 ⼆
器
管 Ply
x and
y are independent
if given x does not changetheprobabilityof y
independent [Link]
and E Exty E x if xand y are independent
E y
[Link] Ex pnnIEy pyIXtyII [Link] Ey
pyiyI EU
xpectation

yI [Link] are independent

ndom Variables Wedefineevents as subsets and probability measures as a functionthatmapssubsets to
real values
A probability distribution is a function that maps individualpoints to realvalues
A variable is random variable if it is associatedwith a probability measure

If an V10，1 then a is random

If an Nco l then a is random
some realvalue [Link]
If Ex Nco l then mt E is random
for [Link]
[Link]了 yn.NU2 oi then xty [Link] mini
If V10，1 and uz [Link]
If u End [Link]
[Link] 1
⼆
 ome Mx七 EIEI

fetplxsd nction
1tfxtfxi lTaylor
nt_generating

[Link] E E E
1 吉E区 [Link]区2
Milo EIXI M x co E区2
If Mx 七 Myct then x and yhasthe same probability distribution
MGFofaGaussian Suppose x [Link]
Mxct E t 18⽐ ⾼ expl Eilx
tild f
expl fx2 [Link] dx

赢 ⼀些 tn
鉴蕊
it sumsto1after integration
亗竺
瑬紧
canpullit out ofthe integration

iexpl EW
littuii [Link]
Linear Suppose xi [Link] and X Nwnoi [Link]

tanx ombinatio
Gung ⼆ [Link]
[Link] et _Mxnuzfttii
Max taxztast [Link] Max It Max ⽐ ˊ_ˋMan
xnlt [Link]
it Mxdat Mxnlant

[Link]
lamtaunt ee
aunttitlaiiiaziintaio
anxu [Link] aunt un [Link]
ain't hit ta a
[Link] on _tai jnixi Youtube⼤学
爱丁堡⼤学
E IE tax2
E etaxI.EEetax2
⼆[Link] etauniio
etlaun aun iiaioi aiai
Wehave a [Link] [Link] [Link]
 dependence and [Link] _xn arecalledindependentandidenticallydistributed [Link] samples if [Link] ⼀ 加 are mutua
dnticauydistributed independent andare drawnfromthe same distribution

anssian Normally 楍 exp ⼀

器I
坐
stribution

知⼼
武
The discreteprobabilitydistribution of a random variablewhichtakes the value
Distribution with the probability p and the value 0 with theprobability ftp
Suppose we have a single trial

thetrial can result in one of two possible outcomes labelledsuccess and

failure
Pcsuccess ⼆ P
Mhmzp
variance a
Mip pup
Likelihood
Trying to findthe optimalvalueforthe mean or standard deviationfor a distribution
of
bunch observed measurements
given a

Maximum
If we flip a coin 500times and get300heads estimate theprobabilityofgettingheads
icon Assume [Link] Bernoullirandom variables x xn with probabilityB to behands
The likelihoodof Bis
PIX 加 ㄨ了 [Link] Ǚ Pi ⼆点 fill B 1 Xī
Themaximumlikelihood estimator of B is thevaluethat maximizesthe likelihood
L Lgplx ⻘ xilogptlixiilogll
xn ⼆
singing u
pi argmaxpxiu
[Link] tu [Link]'tchangethevalueof
不⼏ o argmax
岩⼆点 给 ⼀ 晋 ⼆点 I
奭 藏
然
 Lecture 3 Linear Regression

Ěǐiiin
nanny
_on points
origin

Think of line as a plane in higher

gher
dimensions
gzwttb
i dimensions
Ělnixitbyi
2

nearRegression S ⼼ y.i lxmynhidatase

xixn [Link] iiiiii
groundtruth label goldreference
[Link]
y
⼆⼼ b
f WEIWEI linear predictor hyperplane
widIT weights
beR bias
w b parameters

Given S
[Link] [Link] fndw such that the mean_squareerror MSL
Givendatapointswant

The act
[Link]
is called
ǚiǖǖi
offinding w training

Thegoal of linear regression is to solve

2
⻰感 Whitbyi
悠
The optimal solution Satisfies
⾔ 此
⾔ _vector if real
⼀ number

⾼⼆⾔之惑⼼xitbyi 三⽆惑 uixitbyi ĒlwlĚxi Nb Ěyi o

2b Ē y i nixi o
 big it
y

Substituting
big 咬⼀ in L then
ㄥ ⽆惑⼼
xitgniyi 2
惑
⼆⻰ ⼼必⼀万
lying
win
⼆⻰感 vector T Ti
Thus
点
takederivativewithrespect 憥
试点
to avector
úxijxi

iiiiīiī
⼀点 i'ㄨ i
y

⼆ Ělxxw Xy
WE
INT'Xy
we Steps [Link] _minusmean is actuallycatering [Link] now has meanzero
x_x 不 不I

gifynrg
Centering

write cxyyi
f ⼼toCan
pack datapoints
just the way
òmputing the
MoorePenrose
⼼
ˊ_
化
big_ni
Sending

Features
yznixtb i b Y i Y w'Tx ⼆ ⼼中⼼
Fitting [Link] to appending 7 to x andfitting fx not
The1 canbe seen as a featureindependent the input
of
Suppose we have a datapoint x IxIn 2 3 IT
The data pointafter appending 7 becomes
IT 听 2 x 3 IT
Thedatapoint after appending1 and quadratic terms becomes
2
中化 T Xc 2 3 1 2 X2 x 3 3 九四⻛2 2x 3
Thefunction
fixing x is a polynomial linearin no not features or data

We call中 a feature function

In general 中canbe any function
 [Link] we ow have It w

Instead
ofX⼆区 n_n XNI we have 0 [Link]
Theoptimal solution
for linear regression becomes w 10017
的
Alinear regression model is linear in the parameters w not features
A linear regression model can fit an arbitrary nonlinear function
What are the'right'features
Whatdoes it mean forthe program Glx
w we write withdata to be correct

cannot get a perfect fit because

ProbabilisticAssume we noise
of
interpretation In particular we assume the noise is additive and Gaussian
In other words giiWY Xi t Ei where E N 0，1 Zeromeanandvariance 1
IfEineMo l then givN ⼼中化⻔ 1 addconstanttoGaussianvariablejustshiftmean
Nowwe have xīand pairs likelihoodfollows the distribution
yi
Pyīlxi 后 eptiyi_WY Xi 7 no variance in it because variance is one

The of w is
likelihood
log
gipgīlxi log惑病expl iyi [Link]
doesnt changeargmaxw

yi [Link]
器
⼆
点点
䉀等 痴七⼗MSE likelihood
maximizew meansminimize
maximize w

MSEcanbeseen as an Gaussiannoise modelwhereyouassumethe noise are Gaussian

andtherestofthethingare standard linear regression
type ofthing andyou
just
derivativemaximum likelihood back MSE
youget
[Link] error
j Ě gi wycx.nl2
Themaximum likelihood estimator is the optimal solution for MSE
Once
yousee MEthen trigger theremustbe
Gaussian somewhere

Thecomplexity computing 4中⼼ 1g is 0W where N is thenumber samples

of
of
Theruntime is not particularly suitable
for largedatasets
of
Instead solving mind exactly could we
find an approximate solution
In exchange could we have an algorithm that scales better than oil
Not all problems haveclosed form solutionsfor 器 anyways
nearregression The [Link] error can be written compactly as

ingmatrixcalculus Lilltwy吃
Wecanexpandthe [Link] error as
L 11妔 ylliiloiw [Link] ⼆⼼中yTkloiw
yTwtgi
ing optimal solution 我⼼ y iidoiw
[Link] uioly
4⽇ wofwD [Link]
lfIEItIff
业
manorial
WE1997年g

A measure oflengthgiven thesquare root ofthe squares Denoted

ly
a vector F [Link] aus is 11 112 Nāttntat
by11112
the two norm of ⼆
Thetw
norm
of an mxm matrix A is defined by
vector that is
notthe Zero vector
竺 岳ǐ
品贤
where is an m [Link]
 Lecture 4 Classification
[Link]

Geometry Xt N wTxt bio where x 红

⼼如 ⼼ whichside is negative or positive
depends on wherethe
以 be originis andwhere wi
pointing to
sx

Binary 扣 可 if with so
Classification 个 if uixtbzoisgnlnixtb
Theplane nixtb 0 Separates the two classes
Thefunction f labels one class as o and the otherclass as 7
Theclass is called binary classification becausethere are two classes

ero one longy 17

0
if gty ⼆⽇gty
otherwise

Think
gasathe prediction andgas the label
We suffer lossof 7 if we predict the label wrong
In the binary case long y ⼆卫
然⼀ same sign or different signs

lassifcationS [Link] XN yah data set

X X⼝ x IT input
Id I features
y ground truth label gold reference
xtb linear Separator linear predictor hyperplane
f ⼆
W 吐⼝ w Id I
7
weights
bERibia
⼆
[Link] parameters
 Given
Sfx y [Link] fndw Such that the zero one loss

LÈĚlol fun yi is minimized

The act finding w is called training
of
In thebinarycase
hit Ěloilnixtb yi it Ěfyīmixitbso

Slightly changing wand b doesn't change theloss

Theloss value only changes when the hyperplane flips the signof a datapoint
and it either increases by l ornone at all
Thelossfunctionwith respect to wand b is like step functions flat everywhere wi
discontinuity whenthe value changes
Finding theoptimal wand b is inherently Combinatorial and hard

Probabilistic
Making predictions based on signs
2preach
fxnflifuixtbc [Link] ⼆
SgaCwTtb
Defining probabilities ofit1
classes

ltTxtb pcg
X ⼆
ply
llxi 7 [Link] 11 ㄨ

igmoid
Function ⽔S ⼆

When s
命
[Link] 1
When ss [Link] so

probabilistic
ply ⼆411x in luixtb ⼆ 1 ⽐ ⼀ 如
approach When uixtb [Link] ya 11x 1
When uixtb s [Link] y 11x
f
Pifnixtbc [Link] ⼆Sgnlwixtb
 ply it1 1X ⼆

ltexp ixtb
ply
Divideby
[Link] ⼆
毕筠华 1

fnixtbjtl
Plylxk [Link] her 1 or 1

oglikelihood Given data set

wandb
[Link] Mr you I the Ukdihoodofwand b is
of
L log惑plyilxi ⼆ 点 g
ii z
loglltexpl
The
gilwixitb
Ěfyī[Link]
loss
flat and is hard
co is to optimize
L
loglikelihood ⼆点 Txītb has curative
10glltexpl [Link]
However unlike linear
region
㒫 o 哥⼆
Lecture8 donot have closed form solutions

Classification Suppose we have a labeled data point [Link]

Losses Zeroone 6的

Iynixtbk hgh
[Link]
Notation Thelogloss notation log can be misleading
ply x
he Is y the ground truth or is it a free variable ofthedata
Variablesareboundto one
What it really means is logplg [Link] points
Or 6gplgzyīlxī given a pair lxī yi in a dataset

fix if ⼼ 1 0
f
Features

sgncwixtb tlifuixtbz flxjzg

 1 11 1
It1 if uiocx 9 1 1 1

pcylxiltexpl pcylxliltexpl

wo [Link]
example

Linear A linear classifier is linear in the parameter w not in the feature

Classification A linear classifier can have arbitrary nonlinear features

have two classes

inmyn dominator
[Link] nniiii

plg [Link] Then bout and 1 are in dominator

and in numerator wehavethe onematches
binary the input

lassífcaniji
嘉华器 Y1
⼆ 灬 in this case_
中灬

嚣
Ǘniiiim㡭
[Link]
ending theclass yEY gEY
youhighestfix ⼆⽉ .es
twi
yc

rob
ability if wigs wI 中化 on
constant
inna
[Link]年化 o
iflww n ⼀⼼
f 1 if 中仪 20

1y
1
if nicola co

f ⼼如 ⻅
 LogLossin thebinarycase
[Link] 州
Logloss in the multiclass cass
[Link]
binary Classification multiclass Classification

1 ifWElxko
fix 1 1 īfwhx [Link]

yE Mii Pcg1x
ficx
纸 以 管
三 器⼼ Gives
[Link] 䊺
probabiligdntributo zpl. s
1 23 ㄒ 0109 0.24 0.67IT
ofmmax
soft ax 100200300 i 1087 ftp IT
Sfmax always returns a probability distribution
When the dynamic
becomes sharp
the
range of is large the result
input ofsoftman
values are veryclose to one or Zero

Prove Claim 三点㗊器 7 when to

That means
the max
三点 器ie 0 when T 0
forany aj that is not

We have

噐器灬
⼆

explaitmexp [Link]
TlJ
o because whenT
am is the largest and [Link] co
 Lecture 6 Information Theory
How to define the amount of Information
Let IN denote the amount of informationfor event x
Desired
property Icx
of
Monotonically decreasing function of probability
If plx T
Icx
x

If plx o If x happens it wouldbemuchvalues

Adding of independent events
If pcxnpzplxpcy [Link] Icx Ig
Candidates
of Inn log 或1
Choice
oflogarithmicbase
log偷 bits
gel剜 nats

tow to define similarity between two distributions

⼆ Euclidean distance
Pdx us
Pgly
1点筘器器管 器 䲜Iitg
tunnelCoding
1凾 ⽐
国
We want to send a message with minimal numberof bits
We don't know the message ahead time
of
ending 1 Coin 1 bit per message
Coinflips 2 coins 2 bits
permessage
If it's a fair coin pcHF PCT ⼆ 三
If there are two fair coins pcHH pcHT
Thenumber of bits to encode a variable x i
pcTH p TT 本
⼆

log前 10gzpcx
Lowprobability events need more bits whilehighprobability events need
fewer bits
login bits are equivalent to hogpix nats
Entropy The entropy of a distribution
p is defined as
Hcp H1 ㄨ ⼆ Ex

[Link] or iiii
pcxiElogpcxi
Note that of iij
Hex is not a function x
Theentropy can be interpreted as the expected number of nats needed to

Entropy of For a coin with probability u being head its entropy is

a coin
ulogu [Link] log 1 u
The entropy peaked at a o5
In general the entropy of a distribution is higherwhen the distribution
closer to uniform
Entropy can be seen as a measure
of

ff uncertainty fi
The conditional entropy

Hcxly
of
⼆
x given y is

Exy [Link]
If x and
y are independent

Hlxly Exy [Link] 学

Exy [Link] 臖
Exgnpix I logpix I
⼆

[Link] logpix I
⼆
⼆ Hlx

Knowing something reduces the entropy in general

EH x
Hcxly
The proof requires a basic fact
log t s t 1 so
fort
Or
logt 31 t
for too
Proof HM 类
Hcxlyi Ex [Link].I Exy [Link] E log呫
⼆
Exy [Link]兴器
不Ex g [Link] 1 咍
器 I
[Link]
⼆⼋五年pcxy 烮
器 ⼆三[Link]

1 EP以 ⾔Ply o

Mutual Since Hcxlg EH cxl the extra information Hcx Hcxly we know
Informationabout x given y is called the mutual information
I [Link] Hcx ⼀
Hcxly Hey H 的1
⼆
Ex yopix y I log咒
器
Cross Recall that the entropy Ex [Link] can be interpreted as drawing
Entropy message x from pix and sending it with logpix nats
This assumes that we know p What happens if we do not
We estimate p with some other distribution q
The expected numberof nats under p ofencoding messages with
distribution q is the cross entropy
Hip q Ex
pcxsElogqcx. denote
N joint
Bi the notation p qisalso used to entropy [Link]

Weneed more
than the true
nats
if we encode messages with a
distribution
distribution qother
p
Hcp E Hp q
The proof uses the inequality log t Et l again

Hcp q Hcp Ex pcx.I logqcxiI Ex

Proof
[Link]. E
⼆

Exnpcx I log幾
Exnpcx 1 幾
1 引 ⼼ 死 ⼆0
 allback_LeiblerTheextra nats of encoding with the wrong distribution is the Kullback Leibler
divergence divergence

KL pnq Hcp q Hcp

⼆Exnpcx 10g 器
KLcpnqiz [Link]
PHP ⼆ 0
KL divergence is often used to measure the distance between two distributions
However in general [Link] Toovercome we use 9盥
之
Mutual Read that IN y ⼆ HM Hcxlgi Hiyi [Link]
Information
In ⽐⽐ ⼼以
Ex y [Link] Eloi礜 I
meaningweassumex andy are
I 灬y [Link] pmpy independent

Mutual information can be interpreted as the number of extranats if we assume

x and y are independent

rossEntropy Recall that in multiple class classification

ndLogLoss
pcglx explwjollxi [Link]

Thelogloss is
6g pljx 味中化 log 新[Link]
ixil wherey
is the label

Given a datapoint
[Link] wecan represent the ground truth as a distribution
ply 正guy onehotdistribution if for
Thecross entropy between the ground truth and the learned distribution is
Eynpy Eloglglx 新 pyI
logpylx.
新⻢ logPly
E
⼼
__y

logphil x
Hence the
logloss is also known as thecrossentropy loss
 Lecture 7 Optimization1
Convex Ln
Optimization
录 Tw
ii
Suppose
f Rds R
The goal is SOhe
ninfa
Note min fi E fyi for any y
We want to find 姅 such that fix myfix
The point 对 is called the optimal solution or the minimize of f
There might not be a minimizer or there mighthave many

convexfunctions A function
f is convex
if
fcaxtcl [Link] x
fcy
for every x and 0East
y
⼈ For f Rs R
if器 ofis on
For
TD visualization fRd [Link] ㄨ ⼆Hix is amatrix
If His
semidefinite
c f is convex
i 以1v30foreveryvector

[Link]. l
xxxtu
Propertiesof Iff is convex then
convexfunctions
fix1 3f y tixy ⽽ fly function is supported
by
forany x and
y
hyperplane everywhere

Proof fcxxtllmgssxfxstu [Link]

2fytflytxixyn
fyexf fyitflytacxyn
吣 [Link] fix
[Link] [Link]
when o_o then it 2

nation
y
thedotproduct between the direction and gradient
 fxitcg [Link] fix 㣈
Efg
Tutorial 7 i Recall ⽗兄 的 ⼗⼀ ⼆
Dufy i Úzfy
Substitute v x y
How Ky The direction vector
gg fix
⼆
tly xi [Link] a line of y
Remember the slope⼆ 点 㠭器淡
器笊 Think 对⼼ as the slope normalvector
andy x as the horizontal displacement
should be interpreted as
fix y xifx ⼥
currentvalue ⼀ horizontal displacement slope which is the way you describe a line
Why do take x T
we transpose y
Because
yx is a vector and fix is also a vector The hyperplane is
defined as all the possible vectors are perpendicular to fix
the normalueco
That's why there's a dot product to measure if a vector is perpendicular to
the normal vector
 definit [Link]
sufficient

iii 箈 Tfcx
condition When we write
Ff x Yo we say that is Positive semi

2
convexity
of The Squared distance list [Link] is on ins
guarddistance 等 2 27 o

then [Link] also convex

Affinetransform If f is convex
gcxxtllxiyizfcacAxtbitll
reservesconvexity
xicAytbi sxfcAxtbitctdifcAytbi
[Link]
onnegative If f 后 are ⼼⼼ [Link] Bkfk is alsoconvex when A pk
eigHedsumof
[Link] ⼆时 2 1my t
Rfk 2 1 2y
[Link] pkci nf
myung
[Link] Bkfk⽐1 1 2 Bfg t tpkfgs
2fix 12
fly
convexity of The meansquared error ⼆is
L ⻰ Ě ⼼中灬 yi
2
MSE
WeKnow that the squared distance is convex
Use the affine transform andnonnegative weighted sum to obtainthe meansquare
error

Optimality Hf is Convexand 妙 0
condition
对minimize ⼆

at M
then 对 is the r
off
⼥
ProofSuppose 对必 ⼆ Foranyx p the对化
fix fit textile fix
OptimalsolutionThemean_squared error is
of MSE L⼆⻰点 ⼼中 X⼀ yi
2

Thesolution to 沯 0 isnt 中中Tidy

Because L is convex in no ut is the
global minimum
 wealready knowaffinetransformpreservesconvexity So we don'tneed to worry
Convexity of The logloss in the binary case is ⼼中⼼and yi isonly a multiplier
LogLoss L⼆点loglltexpl [Link]
Wejustneed to show lcsi [Link] is convex in s
Use affine transform and nonnegative weighted sum to obtain the logloss

柰 ft in
iii
1
嘉 ⻬
家煎
⼀

品笑
煎 夔 庇江 炏
trongconvexity A function f isµ strongly convex if

forany x and y 䶛 器品恐会 in

function is supported
多parabolaeverywhere
Quadratic Lonerbounded
Lowerbound

A function is L

LThitziffiifykniiier fr
pschit [Link]

anyy x and
In words the function Values can
onlychange little for points that are
close
If you change x by a Little bit the functionvalue doesn'tchangemuch

off is L Lipschitz then we

say that fish smooth
mothness When the gradient
In otherwords f is L smooth if
Hefly 对⽐111 ELNg x11whenpoints areclose thegradient
⼀

for any x and y

doesn't change much
L smooth also implies
fcyiefxitcy [Link] xy 吃
Proof fyi fxiI [Link] 化⼋ X 址parabola 加 泓 卿 啾啾
Byconvexity fifty 对 g by everywhere

fyi fa cxyifgxnfcyi ⼀对 [Link] y x

Emfy 112 ⼼ 1111yxy
EL Ily
0uadratic

yiǔg
 Lecture 8 Optimization 2
Forlog ⼼了
⼼ log 1 expgi⼼中
we cannot solve 先⼆0
matesolutions We
zroxi
say that 分 is an approximate solution
if for a given E 0
optimization

Note that it is
ficlose f a
in function value not close in input

Gradient descent is an iterative algorithm

radiant consisting of the steps
seen Wtt ⼆ Wt Lcwe Jt
Thevariable is called the step size and can dependon t
gt70
how
faryou want to fun the gradient thenormalvectorofthe

iii
riterative Howmany updates do we need to achieve

fcxti [Link]
it
an approximate solution

We want to express E as a function of t

[Link] Shine
Results a fxti [Link]
Linear
fx n
[Link] faster
Quadratic
fcxts fcx
sciforoerc sublinea fx
E
u
0 ⽆
[Link]
or t 0 吉
Linear
[Link] sci for Ocr
E Oc it tz OnlogÉ
⼆ or
Quadratic

fxti ftiscriforo.rs1
[Link]
a
[Link]
 11对⼼ ⼜
fyillELllxyllfixkfytix [Link]
of Based on mplications
of
definition smoothness and gradient update

moothnes
yi
fiiifiixt
choose E Xt
iiiiiiin
flxtni [Link] LyinAl
Xt 1 ⼆ ⽓t
[Link]
Xt 川⽓
f N t 11
Jt 1
Lgt Nfl Xt 川⽓ L smoothmaking
sure
In other words [Link] youalways
makingprogress
Theexpression [Link] a maximum 在 when gt⼆⻰ and reaches 0 when gt
Choosing any gtE Ii t is able to Strictly decrease the objective
Forsimplicity we choose yi Ì for the rest of the analysis

Based on the definition

of strong convexity
miffns
Strong fx [Link] yix fy'tTHX7 䂬
Thebest x on the righthand side is knen
y Ènfy 对ycyfy
Wehave
fix fly ÈuHeyEHi 11for any x and
y
In particular fly _fix 在 对y HE_
words In how
the gradient norm at any givenpoint tells us
far we are from
theoptimal value

Guarantee
of If we do gradient descent on a L smooth andin_strongly convex fncti
[Link]
Given
illoflxtill fcx
radiantdescent
n fcxssfcxt [Link] 1 在11对1X ti HE
Ènofy112 fxt fly E f Xt fix 吉 f 妙 [Link]
⼀

1 吉 妙 fix f
1 吉⼋
Theconvergence rate is linear_ nni [Link]
If we do gradient descent on a Lsmooth convex function
[Link] a 唑等
forget
The convergence rate is sublinear
No proof needed

LogLoss The logloss in binary case

L 惑log ltexpc [Link]
 W he sown s

og

radiant on 毙 ⼆ 点 燓不怨 管派 c
[Link]
loss
itexpi j
⼆点 ll c

golcxn iu
of
size
hedataset
n hu
For mean_squared error recall that the solution for 器
w 中中⻔ 中 y
⼆0 is

Computing 14474y takes 0 N3

For gradient descent on log loss computing the gradient itselftakes 04

tochasti [Link]
下 Sam t from data set S
descent 2 wttiiwt gt [Link] l is singleton lossfunction_onlymeasure
Rr Sample Lz loss [Link] wig 灬 _y i 10910的 atgivenpot
Per Sample log loss tcwix y [Link]
3 Go to T until the solution is satisfactory

[Link] is now random because Xt are random

yt
The expectation
Ex gus it mix y 7L ⼼
where US is the uniform distribution over the samples in S
aurantee for If we do SGD on an K smooth convex function
[Link] Ex y us I LIE EL cut Mil ⼠兵

[Link]
i I
radiantdescent where

tcwtjxy 112 11Exy [Link] HE forany t

⼼
Ūt ⼀
Noproofneeded
The runtime is Oct independent
of the data set size No

ub
gradient A sub
gradient at x is a vector
g that satisfies his
set
ofg
3 fy fix tcgxig
for any y andthe set of sub gradients at x is denoted as Ǜǐi
Obviously Pfcx E Han if Tx
exists
Convergence theorems can be ported to sub gradient descent
Examples
fixX x2 is 2Strongly convex
fl M is Convex and 7 lipschitz
This also implies that mean_squared error is Strongly convex function
fix ⼆ 11姙 is 2Strongly convex
glx fix t ⽐吃 is strongly convex if f is convex
 Lecture10 Neural Networks 1
hamster

cognitionwith
eruption

cìsionBoundary
ycx ⼆ [Link] o l x 式2X ⼀器
⼆⼀ [Link]
linear
discriminant
2D

eason
0
Boundary
[Link]
linear
discriminant
3D

asianBoundaryDecision Boundary⼆ [Link]

linear
discriminant Dimension Decision Boundary
line
⽅
3 plane
hyperplane 巤赢
[Link] 0

NB W is a normal vector to the hyperplane_

of
raining A discriminantfor a two class problem
zupa
[Link]
eruption a⼼ ⼆⼼九⼗W ⼼义
nor correction where w we wT T 义 1 xT T
[Link] use wand x to denote wand x from now on

y X ⼆gla凶 ⼆
gcwīx
where
gia ⼆正 a o 1
if
1 zif oa Eo
a

ga activation transfer
D 11x y
function ⼆ Hia
ga forperception
Training set 了
[Link] [Link]
y i 10 7 targetvalue or label
Initialize w
Modify w if Xi was misclassified
utc wtgyi [Link] _learning rate
NB lnnmixi nixitgcgi

ycxiilllxi
raptron's eometrg. r
夕化字 wi
[Link]
correction
ycxinxilogal
gixijoyx yi
0 0
yi 1
1 1 0
[Link]

heperception Incremental online Perception More Th Someone notstab

algorithm
for i wcl1 arning
N
dgfffhm
because
everytimeyougetsomeerrors
Sometime its oscillatedoesn
wtgcyi [Link]
Batch Rrceptron algorithm converge very well
Vsumio More efficient stable take more time
for i 1 N
Vsum Vsumtlyi
 yUsu leper med me

Whatabout convergence
Theperception learning algorithm terminates if trainingsamples are linear
separable

nearly Separable
us
nearly
nonseparable

optionstructures Wi 以 T
and gcxigcacx.
gcwTx xs 1 x 加
Boundaries asim where
gia if a so
1 if a to Cf sigmoid function

213
nnnānnnn
doesnt changetheLine

NB A one node neuron constructs a decision boundary whichsplitsthe inputspace

into two regions

you a
logicalFunction

Question i
findthe weightsfor each function
Perception
or XOR
rceptro [Link]

and
elisionBoundaries

Eachedge is a node

gwixtw. et
⼆
[Link] y
workwith 三
ultipleoutput yklx glwixtwko
odes

剡州 11 11
iwiiii gg
K outputnodes y y N Bi we
wi
sometimes use
y
to denote j forsimplicity
ForXn⼆ Xno Xin T
jnk⼆g 总Wkdxkd glanky an ki 裘Wkdxnd
Limitationsof Single
layer perception is just a linear classifier
eruption Multi layer perception can form complex decision boundariespiecewiseLinea
but the Perception training algorithm is not applicable
data are Linearly non_separable
Training doesn't stop
if
Weights w are adjusted for
misclassified data only Lcorrectly classified da
are not considered at all

owcanwesolvethe problem
of training error
Use the least Squares criterion
for training
2
Ezlw zijn yn
Replace with a differentiable function
gl
What about removing gl in the hidden layers
big W
gcnixnn gnzgcwmixni [Link]

Question Show networks with linear hidden nodes reduce to single

layer netnor

Replacing with a differentiable non linear function

gc
g logistic sigmoid function

ga [Link]
⼆
⾯

Mapping ⼀内 to co 1
Use a threshold ⼆ 0.5 for binary classification
忐gcangiain gcani [Link]
utofNN
hresholdfunc us

gmoidfunc it

of
yNetworks
Universal Approximation theorem
and Univariate function and a set
ofaffinefunctionals can uniformly
approximate anycontinuous function
of n real variables with support in the
unit hypercube only mild conditions are imposed on theunivariate function
业
A single output node neural network with a singlehidden layer with a finite
neurons can approximate continuous functions
unconstrained
Lecture
9 Optimization 3

Optimization my Law

n example 在 如
Law
roblemwit [Link].lk1
constrains

is an example
of
a constrained optimization problem
Theinequality IN1124 is called a constraint
Solutions that satisfy the constraints are called feasible solutions

Representing We Can Write the optimization problem as

m
constraints Llw V 111w⼼ 1
where
u if SEO
f if s so
This does not change anything both problems are equally hard or easy to
solve

often the We ⼼ 印 mine

m [Link] [Link]
Constraints
with
myLaw ⼊ 11w112 1

forsome ⼊了
Note that ⼩ E V_ S for all s

Lagrangian In general if you have an optimization problem

my Law
[Link] so
the Lagrangian is defined as
Lcwltxhcw
for ⼊了
The value ⼊ is called the Lagrangian multiplier
Dualfunction If Ǔ is feasible solution meaning that he ⼼ so
a then
⼈⼼ 热 了 EL ⼼
答
Thereshould be a lowest possible left hand side
myIcu ⼊[Link] ⼗九 hw EL co
We call
g ⼩ ⼆ myIcu ⼊hcw Elint
the dual function Always underestimatetheoptimal value

We can see that

for any ⼊
E Lcwx
gc⼊
where WX is the optimal Solution
[Link] subject to hcw so
Theproof is the same argument that
Cw EL cut ⼊hunt Link
gc⼩ myLc with a
⼆

In otherwords thedualfunction alwayshas a lowervalue than the optimal

value

Since
gcx stint
for anyEL
⼊
Cui
where ⼊仁 [Link]
gci maximize
gix
Theproblem
gn
拟
器
⼊
⼊ 20
is called the dual problem

Thedualproblem can be written Compactly as

笳 fix ⼊hcx

Forevery feasible solution i hi so

For every feasible solution ⼥ to make fix it ⼊⼈⼼ as large as possible
⼊has to be zero
For infeasible solutions ⼊ o
lunigram Row Row Row your boat gently down the stream
model MerrilyMerrily luring Merrily life is but a dream

Then are 18 words

Intuitively
pcrow ⼆ ⾔ panty pcis ⼆ È
There are 13 unique words
We refer to the set
V
of unique words
row your boat down the stream
gently manylife is but a dreamy
as the vocabulary
We assign each word v a probability R
The probabilityof a word is
Pcw 茈pfiw
We assumethat each word is independent of others
This assumption is but can go really
obviously wrong
far
The likelihood of A given the data is
GPvector
⼼ wa log杰plwi logĚ 豇pǚwi
Since B is a probability have the assumption
we
憑⺠丑 ilog R
唟R 1

We arrive at the optimization problem

惑忌卫 i log R
答
St 唟R 1

Its lagrangian is
F ⼆点忘卫 logput⼊ ⼼ ⺠只
Solving the optimality condition gives 器 ⼆点㐬卫w_wi É 卫 以 2

7_ 点 耘⼀点 _iiiniiiii
[Link]
⼆

忘义惑⼯ i 1 ⼊⼆⾔ˇ 上 i N
⼆
 B ZuEV2⼀⼆⼆wi 1 1 1 [Link] lad check
Bk I ⼭__wi manytimes K occurs

Projection no
Ti
v
nuncos0 11
吤器叫 in
The projection of x onto w isn
If we have N data points x 加了 then the sum of the csquared
with
projection is

The sum
Ěfup ⼆

of squared projection can be seen as the spread

of data
[Link]
I xiw
Maximal
Projection Ǚy
xijiì XTw
Wewantto find the maximum direction to project
The optimization problem is
my 哓兴
The problem is scale invariant multiplythe solution
bya sealerdoesn'tchange
⼀
iii 怒 ⼆ ⼉ the solution
we don'tcare abouttheleng
The problem is equivalent to 4 w

my wTXXTws.t.mn2 1

The Lagrangian is
F i i w ⼊ 1 11W们
Finding the optimalSolution gives
是 ⼆
XXTXIMW 2⼊w 0

ㄨㄨㄒ w ⼊w
It turns out that ⼊ is an eigenvalue and w is an eigenvector

Plugging the solution backto the objective

Why ⼊
器 ⼆⼊

Since the
goal is to
find the maximal projection this is now equivalent to
 I [Link] genuine f XX

The term
i is
called the Rayleigh quotient
The optimal w is called the first principal component
 Lecture11 NeuralNetworks2
R sigmoid function_larger weight its sticker I 5
change the weight slopeof sigmoid function changes

Universal algorithm theorem

A singleoutput node neural network with a singlehidden lager with
a finite neurons can approximate continuous functions

Output of his du a
Single的 a network with a Single output node Logistics
igmo [Link]
activation function
pity where 点 f
ctNation ⼆ a⼆ Wi Xi
y g
fi m
⼆
⼀ 三品 知
interpretation
of output X X

Guinea
robability
of Ci ry
two class problem with classes
p p
C and Cz The

posteriorprobability
un
ofclassC
㦛 [Link]
京器 [Link]
⼆
1 ⼆

Approximation

f posterior
snobabilities
Training Training set D [Link] l XN where
yi E To ⻔
yN
single layer Error function
ural network En Ew 三点ljn yn 2

with MSE 2
it gcuixn y n

i Ěcgl感 wixni y n
2

Definition ofthe training problem as an optimization problem

兜 EmsECW
Optimization problems
No analytical solution
Employ an iterative method requires initial values
eg Gradient descent steepest descent Newton's method
Conjugate gradient method

Gradient descent

vector form winwtwi [Link] o w is vector

Online stochastic gradient
upiǜifiiiiǎ
ccf Batchtrainingaccountat
descenti
Sample xn yn from the dataset and
time

I wi
global minimum
Solution
Ii

try differentstartingpoints
raining EMSEl ⼼ ⼆点En where En ⼆
Íljn yniiilgcwixnis [Link]
ng gun
USEconti Where
in ⼆ gca an ⼆ Éwixi 器 Xní ⼀ ⼆

⼦
㦛 器 i
cjn gnio i
cjn [Link] X ni
trainingsampleof ith dimension
iggy
ugayǖ a
gun fin
Another Training problem with the mean squared erm CMSEs criterion with the
running Sigmoid function 2
Criterion ⼀ EmsEcw i
ljnyn 余 gcan
rossentropy
写
答 ⼆点 ljn [Link] gcanicl

gca evro [Link]

forsuch a that got0021
[Link] error
Eric w 尤感lynlogjntci I
五
ynil.gl1 jn
EH ⼼ 点
for multi classes ⼆
Eīyilogjì
It can be shown that
哲兴 ⼆ ⻰ j n yn X ni

Other Th

嚣 i
activation n
functions Mapping 0 内 C 1.1

ReLUCRectified Linear Unit

ga [Link]
Several times faster than tan h output
 ly g L l ro
[Link]
use Leaky ReLn instead

ngklayer y 加 ⼆gcǐix two

etwork
ith multiple yklx ⼆glWEX t Wko
output
nodes K output nodes y
For ⼆
xn Xno XnDJ
gk 徽 州 黜点 瀏
y让 g
⼆ 三
品 Wkdxnd gcank 点Wkdxnd
an k⼆
gg Wi

Training set D⼆个 x

y [Link] 了
where y n you ynk and yuk E To I了
Error function
EmsEcw ⼆⼗点 llj [Link]
⼆
点En where En⼆三n jnyn HE 三点link up
training with gradient descent
Wait wki
g
kicyso [Link]
of Ei 惑 [Link]
error function
gnk ⼆
g Lank
single g ank⼆ 感Wkixni

器 籡 然 然 ⼆

⼆
ljnk [Link] g'tank X ni

Outputs with Sigmoid activation function

f output
fffffton
nodes⼀ problem 惑yk
K
g Fy
aK ⼆ c [Link] ⼆
⿊ Wkíxī
Sfmax activation function⼆
器
gk 幸其 all
 Properties
ii
of
KE I
of max function

0Ey differentiable
ii in ye
y

PCCklxsizbca
k0f
multilayer
wawii
lk Multi layer Perception MLP
Hidden to output weights

gi .Input.to_hidden
weights
wie wi 7 蕊

Écgiynk
2
hederivatives En
ffunction
the error
与 ⼆
an K ⼆ 杰Wǜznj
9CanK
[Link] [Link]⼆点 ni'xni

簽 靠 撡 撡 ⼆
⼼

El
ji_guk g'lGuk Znj

蕊 器 ⼀

i Ü
⼆
器
惠蕊 字 h bnjsxni
⼆ 1点link y nk g'cane nii J h'sbug X ni
Errorback
飍 靠 Ii
⼆
t
ropagatio
ljnk ynklgcankizn
[Link] ⼆ 装⽔

器
⼆
噵 恭 ti 惑ljnk
ynkigcankjwihibnpx
h'lb
.nl Xni
 1 1 1

tactical neural network with softma

layer go
1st layer with sigmoid activation functions
[Link]
Computation
g X 0 W x m
2nd layer with a softmax activation function
g [Link]
Cross Entropy loss function
L ⼆ ⼀三
yilogjii logjez [Link]

directed graph comprising

Represents computation as a simple of
[Link] and matrices
⼆ Automatic differentiationCNE

Forward pass compute xz Yo j L

Backwardpass compute 点 先 是 亲
Computational

graph
entropy
layer
L
logjcz [Link] C is the true class a 4

亲 录 ⼀
⼆ I log
Ee ac ⼆
哥 ⼼
⼀ 卫i c
j 卫 in

器 g ⼆

[Link]

Derivatives
rule

ectors

of
derivatives
 Lecture 13 Neural Network 3
Joblenswith
multi layer
auralnetworks
with EBP
 Lecture14 Highdimension stats
dune ⽐ 以 me
of a ball with radius r is
volume is scaled
蕊
concentration
expnenw
[Link] d [Link]
we
shrinks
sǜiiiiiiig
by
a small amount E the volume

cyd fi_cl [Link] et

vu zci

the last inequality uses 1 x se

As d becomes
large V111 a r
is only a small fraction
of Vcr

The volume ofballa ball in

high dimentions concentrates on the
thin crust of
If uniformly [Link] of that point will likely end up at the
crust
of ban
The volume of a unit ball is cnn

fine
of a unit
Vu ⼆
器
T t ti ⼀⽂
⼩2不 É X
VII so when d so

Corners
of
he unit cube if
可
Volume near Pick a north direction
the equator Pick a so the width of a slab
灰⼼ 么me above is
about_e.dz
The volume is concentrated at the equator
Two random Dick a random vector u inside the unit ball
actorsinside Set u to be north
a unit ball Since most of the volume is concentrated at the equator another
random vector v will likely lies on the equator
The dot product civ is likely to be o Likely to be ortho
gonna
Thevolume is Concentrated at the Crust So Hull and Hull are
likely to be close to 1
The distance
of u and v is
likely to beabout 后

llx
yllillxllfzxgftllylit wrmofForanyx
[Link] closet
andom ing
Gaussian P 1 eiiexpli
vectors In words for anfine Nco I Ml is about da
 Lecture 14 Generalization 1
Generalization is defined as being approximately correct on unseen
data most of the time

Generalization There exists a common unknown distribution D where both t

yet
training and test data are drown from
The training set S
drawn from D
x yes i [Link] cxn 了
yu īndwìtiiǖp
Thetraining error
for a loss l and a program h is defined as
Lslh ⽆忌 [Link] i
If we have a test set S then Ls⼼ is the error on the test set or
test error
for short for a programme h

The generalization error

for a program h is defined as

⼼ a Ex
p [Link] I
Thetest error Lsich a program h is an estimate
error LDch
of the generalization
of
The goal is tofind a program h with low generalization error LDch
m
I learning A Y H me take m Samples

algorithm In words a learning algorithm is a function that takes a dataset

size m and returns a function fromthe hypothesis class H
of
A hypothesis class H is the Set
of
possible programs a particular
of for
For example a linear classifier is H x win WER
Ino
thereexists such that
A hypothesis class His PAC learnable with A frfr any
distribution D

㤠裂华 for
Approximately all q so and 0 8 ET Such that
aprogram bestprogram howfarawayfromthebest
p
The dataset
LpAcs
sisiimi
is alsorandom
[Link]器
的
of
want theprobability
awayfromthebestisSmu

恐纠 b ch is the best a program of this form c in H can do

 s he e ra
is the appromycre. S part
the probably
confidence probability

Suppose E01 and 8 o 05

o

Wecan say that our learningalgorithm A Can achieve at most 1 worsethan the
best of otherprograms this from 959 of the time
any of
ofreelunch Suppose Ix1 2m Forany learning algorithm A there is a distribution Dand
theorem
fix o 1 I such that f 0 but
Bin ILDAcs ⻰ ⽯⻰
The 2 and 10 are arbitrary constants
In words foreverylearningalgorithm there exists a task that it will fail_
Tradeoff When we compare to 㘩 best in H we are comparing
say we only
ga [Link]
complexityand
DSandsi notfrom same Distribution
Failure case 1 D H not largeenough minLinh not lowenough
generalization When His large min m H LDch becomes lower
When His the universe all function we cannot learn nofreelunchtheorem
of
H needs to be about the right size
Hcanbe a large but the range of A needs to be about the right Size
Forexample We can only run a finite number Steps with stochastic
of
gradient descent so the range we can explore is limited the
by
的⼼ ⼆ algorit m ⼼ ⼗
decomposition
⽓
器 箭器
垒念
应竿 之管 error

Approximation error is due to the Choice of H

Estimation error is due to not finding the best program
of H
boutTraining Since we
onlyhave a dataset S we can only minimize Lsch n
Minimizing Lsch is called empirical risk minimization ERM
If [Link] do we knowanything about LochEau
minimizing trainingerror
 ifgiiiiii
that
Uniform We say that H has uniform convergence property
Convergence D for all Go and reset such that for every h EH we have

trainingerror tg
[Link] error

Uniform Convergence assures that the training error and generalization error

ǔff
are not
fartofrom each otherall
This has
requirement
happen
for 笑⽈
h EH the uniform part and a strong

Awe
state
forany [Link]
In particular
LDChERM E ni b chi t Za
If It has uniform convergence property then H is PAClearnable
with ERM
 Lecture 15 Generalization2
AClearning
A灬

m 最 ⼼

裴
Uniform
Convergence
ch

Nofreelunch All functions

theorem

Error The generalization error can be decomposed into

Deompim
LDcln [Link]
chi
ǔibch's
Estimation error can be controlled
if we do ERM and have uniform
convergence
Approximation error can be controlled
by changingthe sizeof H
Generalization
Many not all generalization bounds have the following form
Bounds With probability 1 8 for an h E H
⼀
1
b ch ELscn 1亞 舉
n is number of
samples
CH is a capacity measure of H the sizeof ft
There is a family
ofuniform convergence results
ample Howmanysamples do we need to achieve a certain error
Complexity How large should n to get to E
⼩巠 顨
⼩ EE
In other words CCH
n o

VCgenerdizati.nlbounds_rn.nu
apnik Chervonenkis generalization
sounds 8 的⼼ ⼀
⼼ EU ⼼ z 占
d is called VC dimension
For linear classifiers H x [Link] [Link] H p 1
Formultilayer perceptions with p edges VC [Link] Ocplogp
These results are independent
of
learning algorithms
In particular it is independent of
how ERM is done

Capacity Shattering Norm Margin

measure
of
H

Shattering Given n data points there are 2 ways label them FI of 7

A set of n points is shattered by His there is an arrangement
n points such that classifiers in H can produce all 2
of
waysof labeling
VC dimension is the largest number
of points that H can shatter
halteringpoints We can shelter 3 points with a line in 2D
2D However n
we cannot Shatter 4points with a line in 2D
The VC dimension
of
lines in 2D is 3
In general line classifiers with p parameters have VC dimension p 1
We can again shatter 4 points witha er MLP in3D
day
Neural networks have larger VC dimensions than linear classifiers_
 ⽐ generalization bounds

毙⾔毕n
[Link] chi Els ch 2 di
When His large minhEH Lsch can be low
When H is large d becomes
large
Capacity
eneralizatīon
tradeoff

We can only do ERM for a limited number of cases

Optimization
Wix XT'Xy in linear regression
forexample
Recall that the convergenceof an optimization algorithm tells as how ma
iterations we need chow large t Should be to get to
Lschti [Link] E

The number gradient updates is often divided

of iterations or
by
the number of
training Samples
A pass Through a dataset is called an epoch
Optimization We care about generalization of Zero one loss not the Crossentropy or
the log likelihood
Cross entropy or the log likelihood are Called surrogate losses
Surrogate losses are easier to optimize than the task loss and usna
have some connection to the task loss
For example log Loss is easier to optimize them zero one loss and i
a smooth approximation of Zero_one loss

Error imitation error

Decomposition Mismatch between the surrogate loss and the taskloss
Controlled
by the optimization algorithm
Estimation error
Controlled
Controlled
if we do ERM and uniform Convergence
 g cap g f a che z
of che ng
set

Approximation error
Controlled
by capacity of H
Under
fitting A model is under fitting
training
if there is anothermodel that has a lower

A model h is under
The better f
fitting
is unknown unless we find if there is
fit such that [Link]
All models are underfitting with respectto ERM
When people fitting they simply men there is
say a model is under
room to improve the training error

Over
fitting A model is
if there
fitting
over is anothermodel that has a highertraining
error but a lower test error
A model his fitting
over
if there is f such that Ls
Lsf s chi and

Better
if Ls'ch
f is unknown unless we find it
Models Can over even
fiterror is though
the gap lls ch _Lich 1 between
training and test not large
When people
say a model is overfitting they simply mean there is
a large gap between training and tent error

We minimize training set S i.e doing ERM

in practice a Surrogate loss on the
Wecanonly do ERM approximately most
of the time becauseof optimizatio
difficulty
Suppose training gives us hi
We use a test set S and measure task loss Ls in to approximate
generalization error
We hope ⼼ is small when Ls his small

Testset Test error on a test set is used to approximate generalization error

Testset is supposed to beconsidered as an independent data drawnfrom
the unknown distribution
Sometimes we have hyperparameters not learned from data we needto
tune
for example the step size in stochastic gradient descent
What's the problem of
using the test set to tune hyperparameters
 1 up a based on
Useandthe
tests chen
and the
a com
site
bel Step you
to becon independent
outcome of
the test size test needs
Sothe more use the outcome the lessindependent

thesis
typo Suppose we have a hypothesis class in 2D
un gzaxt
[Link]
iiiiiix x

lil
1x 1
 Lecture16 Generalization 3
With Probability 1 8 for an hey

1tab
Capacity confidence
generalization
aug ch Esch
Ascapacity ofH increases mirisch drops but theSecond terms goesup

agehypothesis Compare ⼆
classes fl the set oftwo_layerneuralnetworks with 512 hidden units
tlzi the set of all two_layer neural networks
fl has a finite VC dimension while th hasinfinite
It is much easier c or tempting to reduce training error
by increasing the
hypothesis class

FailureCase Compare
2 W2 0.206 0.317
wq 30.69 93.21 2.65 3.29 0 [Link] 857I
Thelearned weight are either too large or to small for degree9
What if instead we optimize
⾄11WIFEwho
LsCW t Hwy2
新
笓 Tw Last who 7w Last
X [Link]
iWtiwt
The term EllWIR is called L2 regularize [Link] gtNWt
yt7wLLW kegularizatien
It is alsoknown as weight decay
Theexpression
Lsu tEllWIR
is the Lagrangian of
ginLscw
[Link]
lwllE TheLzregularizer
has an effectof controlling the capacityof the
hypothesis class
Compare
flyx [Link] WER M
fl 1x ⼼中 x Hulk
BI
Shatering Given in datapoints there are
A setof n points is shatteredby ft if there is
2
ways of label them
an arrangement
Itof 17
n point
such that classifiers in tf can Produce all 2 labeling
VC dimension is the largestnumber points that ft can Shatter
of
Rademacher Rademacher complexity in binary Classification on a dataset S is defined
complexity a
Raft ⼆ 区
器 去点 mx I iiiifiiiǜi
where one In [Link] chosen 器 瀏 了
爕
In words Rademachercomplexity measures how well a class of classifiers
correlate with random noise
Rademacher Complexity in
as
binary classifications for u points is defined
RsCH ⼆
Espn I Rs MI
dematcher With probability ⼤⼦ for all chhE tft ⼩熙
generalization LDch ELS Rnch
bounds With probability It for an hEM
ch Lsch t Rsch 3 藇
If Six 此 ⼼ 了 and fix mix nuns Bi
ǙǙǏǛĚ
Lin if
hypothesis class is bounded
fireRslH ⼆区 嘫 Ed 噩
Rsc
bounded
withbounded is H
坛器 a ⻄ now 讪 款 伛
⼀

⾿是墢 ⾅ x的 11九111
叫
ability If we replace a data point in the dataset do youget a very dffu
classifier
We say that the leaning algorithm is stable is changing a datapoint
does not change the classifier much

If S is the dataset then S is the same dataset with

data pointreplaced with another random datapoint
the i th

Stable learning algorithmsdon't

overf ES
DnILDIAcsn [Link] Eiwcn ItcAcs Xi
yi t Acs xi
y
 y

䧇
唯 Acs ⾂ Ecxy [Link] y I
EsIEcxynDIllAcS lxi yi II
Es IS His Es IE nun It Acs Xi
if I
pschizloss If the losslipschizcontinuous
is P
1 ec x is Acsill
Acs
Xi
yi Acs yi EpHAis
Weonlyneed a bound on HAS以 ⼀Acs 11

pschizand If a function is ⼊ ⼀strongly complex

tonglyconvex
in x_x 112
Efxi flxspnx [Link]
where I
is the minimize
If we can bound [Link] then we can have bound on 11x_x411
We will then let x Acs and 如 ⼆Acs

Lzregulizer this ⼊⼀Strongly convex

Lsu till wli is ⼊ ⼀strongly convex if Lsu is convex
Adding a Lz regular makes learning stable

If wechoose Acs argmǚfscw t Emi we get

HAis Acs HE噐
In theend we
笸 in bus [Link] Et
hypothesis 6咖
lasslimited 军 ⼆ the setof all two_layer neural networks
2g the flu ⼆ the set of all two_ layer neural networks with boundedfrmB
learning the the set of all two Layer neuralnetworkssearched with t
lgorithm updates
gradient

hi has finite VC dimension while the last two has bounded Rademacher
complexity
 Lecture 17 universal approximation
Universal Forany Eso given any Lips chit function f xEl 叫 1，1 there is a
Approximationnetwork
g such that lgcxi [Link] E forany
The number of nodes needed to achieve this is Ocd Tenralnetworks ispowfn
Polynomials are universal approximations
Decision trees are universal approximations
Universal approximation does not explain
why neural networks so special
i
DepthSeparation There exists functions which Can be approximated withsmall depth 3networks
but cannot be approximated with depth 2 networks without using 0129nodes
Functions to show these results tend to oscillate a lot
Some believe that the results are pathological anddonot happen in practice

VCdimension Infinite
offunction
a sin
iiiiiilfhfnho.u
[Link]
Universal Whatcan be implemented with polynomial numberofnodes
Approximation Any Turing machine that runs in T operations Canbe implemented with a
neural network
of depth OCTwith a total OCT nodes
Recall that VC dimensionneural networks is Oil El logl El where
of
E is the numberof edges in the network

and
ness
of Training a layer 3node neural network is NP complete
2

aiming The proof converts instances of an NP complete problem intodata points

the training set we solve the NP Couplet
neuralnetworks
If we can minimize the loss
Problem
of

Approximating ERM is NP hard

Theloss is not necessarily convex
ERM is hard for
neural networks

[Link] Overparameterization means using a lot more nodes than the numberof points
Over
parameterization helps optimization
interpolation Fitting a dataset to training error zero is called interpolation_

Qi Since neural networks can approximate almost everything

why is the UCdimas
still bounded
E matters
Q2Ifthe hypothesis class is too large do ERM training error is o

Why no free lunch theorem doesn't exist

If do SGD with finite numberof steps norm is bounded becauseof
 Lecture 8 Dricipal Component Analysis PCA
d 12 d
Tension
dig
Reduction j Dimensional

léducion
I
m Z

I
Applications
and
Considerations

rmcīpd Xm i Em [Link]
Component Zmxk ⼆Xmxd Udxk
Analysis Z 21 Ez Zk X x ⼈ ㄨd V u in ⼀ uKI
Zi Xu

Rinipal are a sequence

components projections of
the data mutually of
uncorrelated and ordered in variance
The columns u ua U are orthonormal so that
of 0 if and
only uiuj
and uiui 1
if it
j
Maximum Xmxd is Zmxk K d
Variance Zmxk ⼆Xmxd Udxk
Formulation Z 21 瓦 Zk X x ⼈ ㄨd V u in ⼀

uk ZiXu myVarEiI
myVarIXu. [Link]
u [Link]
⼆ [Link] xTx NxcovarianceofX
[Link] uiu ⼆⼗

Using the Lagrange multipliers method

Lcu ⼊1 ⼆UTExu [Link] in 1
Fu ⼆2Exu 2⼊ ⼼ ⼆0
 X Ii g g

To maximize Var瓦⼝ we need largest eigenvalue andeigenvector as

Var瓦⼝⼆ uizxu ⼆ uh u ⼆⼩ theycome inpairs
[Link]
For Ex there are d eigenvalue_eigenvector pairs
ed
fi 吉吉 vd

The first principal direction u must be eigenvector of Ex that corespon

to largesteigenvalue en
E Xui Z ⼆ Xv
Whatabout otherprincipal components
Zz K Xuzik Ī Xu k
Each new principal direction u i should
maximize VarEti I
beorthogonal to all other uj extracting something newfrom X
For Zz Zz ⼆ Xuz

my Varia my
uixu S.t.HU211
uzi 0 1
E Xuz z X V2
be Ex that correspond to second
Because u must eigenvector of
largest eigenvalue Ca

Summary XmxdA
SZmxkckcd ZmxkEXmxdUdxki

XmxdVdx
[Link]
亚 X Z加 x [Link] VKI
where columns
of Vdxk are the eigenvectors of Ex 5ㄨ
 Lecture19 K means Clustering
不blem Aim i Identify Clusters data points in a
of multi dimensional space
mount Suppose we have data set 化 xz XNT as N observations of
a d dimensional variable x
Our goal is to partitiondata set into a known numberofclusters
say k
We formalize the idea by introducing ddimensional
can vectors
MkEll KI to represent each cluster
Specificgoal Given a K find an assignment data points to
2roble. g of
mum clusters and the Set of vectors Uk to represent these cluster
The assignment rule crnk 1 if Xn is in cluster k and all pics
are unknown
Ideally we points in each cluster to be close to
want the
each other and far from points in other clusters

A proposal i Minimize the distortion function i.e the Sum

of
Squared distances of each data point to its closestvector tk

Cmeans J ⼆点 Ě rnkllxn Mc.tl2

solution I Given K randomly select Me 1 ⼀⽔
2 Minimize j with respectto tnk keeping ink fixed
3 Minimise with respect to Me Keeping the rinkfixed
j
4 Repeat step 2 cExpectation and Step3 cMaximization Steps anti
Convergence that is JJ CE

Step 2 Minimize j with respect to rnk keeping ink fixed

J is a linear function of rink Also terms with u are independent
Simply rnk 1 for the closest Cluster K i.e whichever k that
llxn
gives the smallest value MKN
rnk⼆
[Link]
To otherwise
 Step3 Minimise j with respect to Me Keeping the rinkfixed
J is a quadratic function of Me and can be minimised by setting its
derivative with respect to
due to Zero that is 装k 0

8三点 三点 rnkllxnwui ⼈
fk Stu
⼆
点么k X El X2 Xn fuk 0

ĚrnkXn ⼀点 rnkrkio dnkzEnrnkx znrn

1 Random initialization
initialize
lowt pk 2 es are initiated to a subset
ofdata
3 Repeat clustering for various initial andselect the best set of us
4 K mean tt
 Lecture20 Gaussian Mixture Models
Context Kmeans Solution
J 砻 hkllxn [Link]
云
Mk ⼆
器 ⻜

models 1 Let's assume we want to generate below data with 2 Gaussians

Mfffj
a generative [Link] xn Select one of the [Link] probability in assuming
Setthe
Ein
3 Sample data xn from this Gaussian
fifty 三⼝ N ma 云

1 We described data with a generative process

2 In a clustering context all data points with Znk⼆⼗ are in cluster K
3 But we need to learn IinferIcalculate we Ek from the observed dat
But this is a circular argument
1 Trivial to calculate the component parameters we Ek
if rule Znk 1
we knowthe assignment
2 Trivial to workout the assignment rule Znk 1
if we know the Component parameters Un Ek
Mixture
of Complex probabilities can be approximated with a linear superposition of K
Gang Gaussian densities
三点 [Link]
pcx
We define El Zi Zu 1 where Zkfo 11 and Ektk ⼆⼗

We know that pcx Z ⼆

Plz pl IZ and plxr
X
Zk⼆⼗ Ezplzplxlz
7点砖
Tk i
k
.pl0E 不KEI and 三点17⽐⼆1
plz
2
P X12 ⼆万 品N l X lMc Ek F
 Another
key quantity is
pczlx [Link]

⼆ 三
器器器 ⼀
引
812 1 is the responsibility that component k takesin explaining the observation x

maximum Suppose we observe [Link] xi Xz XNI Assume that data points are
kelihood
Luton toGMM
drawn independently the likelihoodfunction
of all Ndata points is
lot ideal Pila f 三 ⼆
点 在NIXnltc.TK1

And so the leglikelihood will be

Li log Pl X law 三 ⼆点 leg 惑 [Link] nlMc Ek I

We can estimate [Link] and Ek by differentiating L with respect to
these variables and gradient based optimization
using

Expectation The EM method can be used to overcome challenges

Maximization likelihood
using Maximum of
If
EM GM Ms EM derives a lower bound B on the likelihood L that is BEL
Instead L directly EM maximizing B
of maximizing
Use Jensen's determine B
inequality to

logEpa If I 3
Epdgfcz I
The logarithm
or equal to
of
the
the expected value
expected value
of
of [Link]
fz is always greaterthen
LM_ Let's define Ink to be positive and satisfying 三品 去 k 1
[Link] is some probability distribution over K components for the
Gans n th data T
T ⼆
笀点 log惑 [Link]
Ě log 在 NcxnlMe Ek 等
点 log惑 Ink 1Mt
⼆
飈
砒
⼆点 Enk I
器等 1
班 Jensen's 你后⼼ 作到 了 㼢 Igf I
inequality
L ⼆点leg Enk
l 炎 1
等 1 B
2点 End log
会
B ⼆点惑hklgnk⼗点惠[Link] 無惑[Link]
EM is an iterative process maximizingthe bound B until convergence
For each update we take the partial derivative
parameters Set it to zero and above
of
bound B wrt

EMSolution E 5吅 M step
for Gmm 不⼆ ⻰ Ei Ink
hi
管
䶛⿏⽣
到 ME
㠭
站
器 ⼀

⼭ 如 北叮
Ek ⼆ 呈笑千氵志
 Lecture 21 i Expectation Maximization
aural latent Two Sets of random variables X and Z
ariableModel X captures all observed variables
Z captures an unseen hidden latent unobserved variables
Joint Probability model is parameterised by OE 0 as
pc X Z10
EMkey It
is hard to optimise for
marginally_likelihood
motion Typically it is easier to optimise the loglikelihood
for
d ng
complete
logpix10

my log PIX E 10

Jensen's Theorem
Inequality If f Rs R is a convex function and x is a random
variable then

Forexample fcx
Efx f 正⼼
x2is a convex function and Ex22 Ex
如 2
Ex oix
o

Cull
bac ing Fordiscrete probability distributions p and q on the same probability
space X
Divergence The KL divergence is defined by

KLlpllqiEx [Link]袈 磊阰 10g 器

The KL divergence measures the distance between p and q but
The KC divergence is not a metric
The KL_divergence is not a symmetric

KLCDnqp [Link]
up
KL Pdp 0
overbound Let q X be any discrete probability function on

Z fr
argnd
likelihood
logcpcxon lgEpix.tl0
log垚 ⼼ 等
1ㄍㄠ

evidence Evidence
i
Lower Bound

In EM we maximise the [Link] q and 0

ÒEM ⼆
[Link] 0
430
Reformulation
910 ⼆ 主 qagpqy
[Link] 嵆 等
器
⾔9⽇ log 望 专
qczilogpcxl i
KLIqlzsllpcZIX.no1 logPMO

logpcxla
tcqoitkLIqllp EM
[Link] an initial 00d
2 Expectation Step
Let q Z Ep LE l X 0 以 givingthebestlowerbound at001d
⽐
⾔
辔品型笇
丁⼼ 㥃 ⼆

3 Maximization 9
qui arg 了⼼
哔
 log up

ma

2
um
[Link] Likelihood estimation is easy
the relevant random variables
In caseof missing dataand or
if we observed the valuesofall
latent variables then Maximum
Likelihood estimation becomes hard
3 In suchcases it often Simpler but not the
EM algorithm
is a
lug faster to use

4 EM alternates between inferring themissing values given theparent

Estep and then optimising theparameters giventhe filledindata M
observed data
5 EM monotonically increases the loglikelihood
Lecture 22 Support Vector Machines
 Lecture 25 Statistical dependencies
dependencies t in
[Link] PanPcg
PcXIg pix
X
ty to denote the independence of x and y
if 化 xv xnj上 的 yz yml then
DIX 加 ⼀
[Link]
⼀
gm
Independence implies factorization
Forexample [Link] [Link]
Lafter [Link]
suppose
plz
[Link]
Original domain is XxYx E factorization domain is Xx Yand Z
 Practice Paper ⼼ 是⼆ 点素哲⼆ 点素 ⼀
两7
2 2
Qzyii EC 1
点溔素 点 ⼗点点
a B
藏
三 i ⼗
号t FI
榧
⼼
辯靠 禁避素 ⼼与诈 䳊⾔ t i ⼆
礁影态
⼩
点 纞蘸 点景点澔 礁 ⼀⼆ 是 1 -2

嗡嗡 ftp
溔景州⼗ 点传 ⾔ t

Q3 ca KLIplzlxilllplz IE Ezyz灬 10
g 器
⼆ 点 Ez pczixnllogpcxilzi logP IT

⼆ 点 Ez pulxi ly 叫坚
器
⼀点 EZ pczixnII 还
gl
-

志瓦 logPIX I ⼆ 点 logplxī
⼩会