RrvstON
dutlcal Wunam.
Buep daaxning
dhe most fundamental unit f antiliial neunal netwt
alled at<icial neuwon
newons = newal eels - newral procesting unt
filjial
Meloloch-
Pits Neunen :
+divided into pantr -
tæptakes an input i pojom an aqgiegatin
makei a dciim
value, f
Aactatety et hhibtey inguh
be inhibitoy ot ercitately
+ hu inputs san eitherthoite that hove mathum fet
Snhilitoy inputi' ae input
making iespecbive f otet
on Hneldecistan that oill make the newtn
+brcitatyhy ünputs" ane NOT the ones
thti onn, but they might re t When
dine on
conbined toqethen
ye fo1}
MP newen :
AND unchion newisn wOua
#dn input ane
bry. fine wahen all
hve:
DA tre g(1)23
Jum > theta, the xNewton would fie ANY
ihee
tt won't
ORfuncbon
yeomesie Snteupuctatim Cr funcdion
a,+7)=1 -- show that all those inputt
whose output nohen passed throuyh the (o.
ABovE
OR function MP function tie oN ot
that (ine and dl the input point that:
lie. eELOW that ine ane going to (O,0)
outut O.
AND function
yeometia ntupnetation AND fancion
Sn this cast, the deuisien boundaxy
equabon is a,+ X 2. tee, all inpute
point that Sie oN et above iust (b),
output 1 whn passed the #ND fonetion
bounduy otks
MP neuxon . 9t fits! The dectsion
y+2--0
(o,o)
TAUTOLOGY(alnays oN)
with 3 inputs :
Becim beundany
(Pane)
OR function
day points:
many unique functions can we
+Aow
inpub
binavy t
Each binany input can take. vabueu o o
bunavy inptt thexe ae a pezible input combirat
.AM n con be eitheuol
. E sAch these combinatitnd, the oubput
Total numbex of wigue boolean tunctions,
that can be fetnid with n binauy inpat iui
Ge ne2 => :4(input combiratiöns)
posible o aepxeSent all the boolean untiens
+98 it
with MP newcen ?
ND t Can't. Betaisi they ave Tevt wotking utng
bireny inputi (o/4) ired weisht, thueshold funetion
Canot model yoR (not Iineaxly sepenable)
plane caudh hat point (o.0,0) Cies on one sde
ie ned a
and stmatring 4point ie on othe Bide of plane
Linax Japaraility : auh thst
that all
all tnputiandwhicd.
Thue erb a Lint/plane ine/plant
one side e< the
poduce 1 die on
input which podute
a 0 tie on either
Puoblem with lon-booltan data :
Cailuatiam
Yuneit nepuntation
Plant
Pudietion : wr+b >=0 (4ceept)
wxtb <0 (Rejeet)
imensitrs
dimibatien MP NewNen and doution :
-boolean (Real) inputs doean't wotk
" Aon
" hand coding 4 threholdng
on- (ineór functions
Ovenconina the imitations of MP newnon ,fank Raserblat.
pxopoked 'tht CLASSICAL PERCEPTION NODEL, Mghty artilica.
newnon in 19r8
Moe qenexaliyed computatioral model than m? newAGn
whee'weiiti and threholdu Cancan bebe ceant oveh time.
feueptren Modet
nuneuiual weight fot inpdu and a
Sntuoduction 4
wransm ot Jenxning thii wetht nput au nO
mechanm
lintitd to boolean values.
lonqex
allons to fnd aut decision bourdany tlinelanc/kyerlant)
#allowed trput - n to be nn-boolean,
alwtd to de<ne tnypottonce of each input (z)uing
gtaen Medel Repuesentatin :
i-l
i=0
=0 ;;< 0
=0
W we
oheue, , =4 and w, -
*de, w, (btas) epueert he puejudice
henly bri ho wight one dearnid Peoxy chooAh:
wüahtd lead to tniasnect clasii ation
fogtaen deasing l_ethim:
W
(-0
implits point bu
je he
on this line, vecte
Wand x ane otthogonal
'p', in +ve dpace ’ w'x>=0 dx90
fox point
fo4 point 'p' in-v space wy<-0 3 d>0
that the angele)
fox aeP t! wX<0 then it meons
ws gteate than n
brtween this x and the cwwent
we want o to be less than 9o)
(but
= Wt
COs (onew) > cosa
Thus Cyohy il be less than d and
his is. ex acty ohat we want
dgothmn : Puceptxon JeaningI;algothm
Poput ith dabel
with dabl.o:
NA input
Kandomiy :
nitialze w
chile lconvergance do
pick sandom. epUN ;
and z <0 then
if xeP (=0
W W+X
ixeN and w; 20 then
W= W-2)
end
.nd
ohn allthe input ae cauilied
the alyothm conveiges
L=0
consider trwo vectoy W wand t
Hence,
lheoxem :
Bnlnition -- Tvo uet Pand point in on 1-dn
doote axe ca lled ahsoutly intowly sp oiabi,
if n41 al no's nWp W uch that
evexy point (*y 73n) eP sati<ies
ualiiu
Pxopositin
I the aet fand Naxe inie avd in cany sepuab,
aparabt,
lkavningnumbu
the pucptronyinit upolat he weigt
algoithmtincs:
vect w a of
tn otheu OBMds : he vectos in Pand Nane
tested cycically ene altu he othe, a weijht vectet
after a finite nmbe f ateput whiet
Wis fournd
Can separate the two steps
optimal oution which exist but
w is some
Wt dont know
at time sep t we inspected -the
Alow gpo.4e 0
point t: and sound that
we mak e a consecbien W = WtP:
ange betwien * and |
þ be the
humexatot
.
(s-min wetvi)
w,
(W}-+ f)+8
2 lby nduction
CosB W. W
tunexator > wt. W, + k/8)
Senom inat WH12
(:N'e4)
<(IW-l+1) +1
can tecome
* a4 k(number e! coNHectiens) incuease cos B
bourded by a
* bt since casb 4, k must be
maximum numbe
* Thu, theue can a finite iumber cotuetons
only be
) to'w' ard alsoxithm oill convnge
Aebian deauning Rule :
Wnaupenvied taxning
sbxength the eonnecbon ohen input a and oupat
ae both active
"No notien o< "coet' ot "inconnetl' outputjut
CoNelabon
Wsedin biologital model Aonauporvised nthotnks
Rucataon Jeavndng mude :
dupevsed leaxning alzothm:
updates weights baed on euuot between pudichd
ond tanpet ouput.
whene
leaxaing satt
t ’ taxget output label)
yactial cutput (usually oott)
’ input
updated woiyht :
dunmany
Hebbian- "newxons that fine togethu wie toge hu
(purely coxelatin bnsed)
bosed n
Puepbuen : "leaun Juom mistakes' (coNYect
destned autput)
’ needs tanget output
-’ bosed on eet cocction
SiljeNene betwen poch and enalion in ANN:
tpedh:
Gne epoch i tne complete pas thuough the nte
tyatning dataset H means that the model
idt tht nodel has deen 6veny tkaiing example
teuatisn:
Cne itovatin is ene ypdat e the modell povomahy
paramehu
he numbex o< itevatins per epoch depends m the
batth sige
data sige =lb00 Sampls
Training = l00 damples
batch "adize
then:
- t000 damples - lo batehes
Iepoch - 0 tvationt
do, epoch
Snplenert AND Yunction sing debhian ude fou bipolax
inputs and taget and we algha =0-01
4 gl3<o
=0
tayget weijht hanges Fínal weijht
Saput
-1 -1 -D"D| -00 00)
001
-1 -1 -0"0
-0202.
-1-1-1
N+02
0-01O-0! o-03 0-03
1
-1 -1 -001--01 D01 D-02 0-0Z 0-04R
1-114+b010-o1-o0
0-010-03D0s
1-1o:01
W 0:04
Dectsion Boundaxy, o
+72=0
fuceptxen eauning kule:
fet each input a, compu gla)yin Zwtb
Ary thvesheld functien:
9 Yin z0 then Youk 1
elue' You -1
only 4 puedictions s incetaeit:
Updat wetht
Rpeat until all puedictond math tangeBs (connegence)
felta Rule:
Sntialige W, Wb 0, d- D"1
coonpute
PHedicted output : !,1gl»20
-1 othewise
Update ule (based on aanó op not thxesholded o)
(t-y)z
bnew bold t e(t-y1
[gunot thueholded ofp us he H0 value belde adhatia
otbian keauning Rule :
Compute
rly thrcikold: 20 1
ele Youk -1
key point:
+Convewgen ce suelens to the peint in vaining vohee 6.
he
noereotweightsbetomel
step sstable
changings siynianty
mininal
and te
(los)
dggtegatin function : vonpute1 the weijhted sum
inputa
* dctivatioN functien : takes net input and puoduce
the output of newen
Bias can be thought af a hoo much levtble hy
pexceptron b: at is sinilar to he constant b
nn
s< a lincax funition y axtb
; alo ws us to more the ine p and down to
it the puediction oith data ete.
* PexceptnOn is a newal netvetk with, ony ene neuYO,
and can only ndeustand. lineax elatioskips betaceon
he injut and output data pxoided
vMaltilayex peueçtxon wa
was dee laped to tactlk dhe
limitationu e! ncn- linear data.
4 is anewal netwotk where mapping between
inpus and ou put is non-tneae
*ftd ferwanding -’ netwoxk
pauu input dada throyh
to corpute ctelut
the
koipxopgain using gtadents
values that oxplain he stengh (degtee of
Cnpettanu) the connetion betoten any
-wo neLOns.
bias - constant value added to cum product dly
input voluu and uegechie weihtb i wed to
aceleae /delay the actiatien of a gieu nae
tunting numbex pauametuin ANN models:
fed foward Newal Nehootl
Hully comected netoo)
Auribev a< panametus = connection between layers t
biass in evey layers
(:Hh +ho)+ (h+o)
pimizes Methods
Cptimi_atin Tehitges : ptinizatin teckiniqus--
Buanback1 analytical
chedking stend oder sulficiny cady
o) diicutty in, postible tolation fo the
)dfieulty in finding all
non liniat epuatin
Numeuical optim'yatten techni_uet numvical mehod
) ohing optimi_ation uoblamssoluuing
tion
to obtain nean to eract
9 dilfexent alyetthimi 'ave wed
clasfied into a ypes
alsetthmu aue nainly
Dinect search method
a) funticn alone to seanh)
(usu the cbjective
b) fadint brsed method f fonctions karck
to
dervats
luu the
Jet optimu)
Unonstyained tptiniz atiern : one whese gou
ú
dn wntbnstyained optimigation pxoblm
concened with he objectiie fnction
nly have to be
you to optimige functien
None of the vauiables in he objectie
aMe ConstHained
Constrained ptimigatin :
This u c[aid to occu when one ot moee sf the
Vauiabler in the obiectihre func ton is constained
by some function.
Hence it will have an objectine nctiens and
a set cf constaint
ioaiunt tscent
optimigatisn puocedune that ian be usd with nnanu
ingle
he salua paameteu (o-) l
d ts find miinizu a cici funcion
that mininiu dhiebie funcian tcax/lau)
¿unttien (4) that
"is bests ,wed ihen the parametu carnet be calculatod
al_e:thint
hatie fetnmula :
nutxical alyothm to fnd the minimum dan
nony
lttonatained puoolem , the itatie fstmula wd i:
Khene
al is puevicus tbuaticn point
in atep k
d) u dection ef dercent in stepk
tonditien: 4(k+)) <(a)
(hek whether tie given functiens have a ducent di
m - (a,-) alomy the gien dine cthions d and d,
d -(61)
d 23) and d,
dol:
2= (d-1)
dh (-23)
,2
+433
01 4(a) -a4)+ 6(2)34
tA)-(2) 4(-1)=0
v la,-)= (34,-to)
-) To check desent dóuechen dixecen
vfla)d<0
vfd 34-(2) +(-10)3 ==-8 <0
.. d, is descent duection
fo do ( 1)
vf - 34(1)+ (-0)1 4 >0
. da is nota descent dinection
. lind a minimum fo the funtien fy)- (a-1)# +(y-)2
dtaxting yuom point (b3) wing atepest destent mthod
choose the teumination panamete E- D-0001 and
Step length is d - 0c0
Hay) (x-1)+ (y-2)2
(oY) (13)
d= 0"00! E 0-000|
Set itenation he
| terat untl
) (2%Yo )=(b3)
v(13) - [2 (-)» a(3-a)]:Jo.a] Jteratien- o
2 =1 - 0-D001(o) = 1
4, = 3- 0"0001(a) 998
Heratien-4
-Q998 - 0" b00l/1.996)
996004
Ye (-o"002) + 0·oo4'
(y22 onveses
Gropping vatue q:
minímum potnt (h)
minimum vaue
"ypa
yadient dicent :
kateh gtadtent ducent the gfadient
t compute
wied to panameters
ántiue datasct iaMpect to
les functien with
udaB per epotk
only 1 weisht
dinettisn (fullinfo)
’ acnath lauye dakauet)
dlow and menLoky- intersn (fot
veuy functien neduces' Snootkly
toat
computing ghadienti en the
updates
’entine wight aftr
datauet.
&Brchasti gadlient dzscent
Medel parametens ane upated uing he godist
of the los funttien 'wnt ingle thaiñing brample at
each ttiuatien
updats wei_hts wsing one sample atatime
-’ fast and mete memety- elticient than falGD
- Can be noisy and may csc illate
3" Mini- Bath atadientdescent
cenpusmise betveen 66rD and sGDi instad wins
Jul dataset/single example MBG Dupoate themca
paramehs winj
data catled a nni -batck
lls a god balance bw pufomane Áacracy
Gptimigeu ae olethnu /uehedd, ued to adiust the
wei_hti o neural nctwedu to minintie los tunin
slent epinizeru elfect the sped, atabillty and nal
) S5D - baic optimiger, Senaitie to leaNning sate
)Acam - adpative moment estimatien
a) Msprop- Root Mean squave Paopasation
- ixes hdastad by wing movin_ avuaye
Cquard gkadient
- adpata leauning sate fot ea dk pakamet
) agtad
- Adaptir 6uaditnt #yetithm
Memnkun dmorther &faute leanning:a
momenhum helps accaleat gtadtuot n Hghk diteçh
eading to fash convyenie
add a team. Yelocity that canis fotnard pest
gtadiont to sDMeoth the ypdate,pnoceis
Nithout nom entum
with Monmenkum
ddaptin deaxning kate :
giadient desent,
gn standad g'adient desenti you we afired laning
But semetimes -
some weihks leavn too faut caue nstab:thy
cthes leantoo slow ’slon slow conveGene
td patie lavang aate tednques adjuut the leauning
Hate indiidualy tot each pavamet based on tt
hitetteal gkadieta
Adagiad:
Apdapts leanuning nate to each panaeto baed en
sn
dum af past squaned gadient sat deueres
-qeat tot Apanse
- data but leaning
too much oveu tie
tdagtad by wing an erponentially
bwipuoves squned gadib1t
wtighted movin aveaze o
puoblems
wokks well tn non -sationaxy