References

Dr. Chetana Gavankar has over 27 years of Teaching,

[Link]
Research and Industry experience. She has published
[Link] papers in peer reviewed international conferences and
journals. She is also reviewer for multiple conferences
[Link]
and journals. She has worked on different projects with
[Link] multiple industries and received awards for her research
[Link]
work . Her areas of research interests include Natural
Language Processing, Information Retrieval, Web Mining
[Link] and Semantic Web, Ontology, Big Data Analytics,
Machine learning, Deep learning and Artificial
Intelligence.

Thank you!!

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

This is in your brain

Session Content

Neural Networks and Neural Language Models

• Units in Neural Networks
• The XOR problem
• Feedforward Neural Networks
• Applying Feedforward Neural Networks to NLP tasks
• Training Neural Networks: Overview
• Neural Language Models
• Introduction to large and small language models (LLM and
SLM)
• Prompt Engineering By BruceBlaus - Own work, CC BY 3.0,
[Link]

56
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 X
At its heart, a neural unit is taking z= z=a weighted
b+ w·x+wi xib sum of its inputs
neural network is a singleNeural
computational unit.
Network A unit
Unit takes Neural unit
mbers as input, performs some bias term
computation on tional
them,
This is not in your brain and term in the sum called a bias ter m. i Given a set of inputs x1
As defined in [Link]
a set of corresponding 7.2, zis just1a...w
realnvalued
and anumber.
unit is taking a weighted sum of its inputs, with one addi- Often it’smoreconvenient tow expressthisweighted bias sum b,using
so the weighte
vector notation
alled a bias term. Given a set of inputs x1...xn, a unit represented
has
y vector
Finally, instead of using z, a linear function of x, as the output,
as:linear algebra that a vector is, at heart, just a list or array of number
from
eights w1...wn and aOutput
bias b, sovalue
the weighted sum z can be • Take weighted sum Xwillplus
of inputs, a biasto the output of this
a apply
we’ll a non-linear
talk about z infunction
terms of af to z.
weight We
vector w, refer
a scalar bias b, and an inpu
X z = b+ w i xi
the activation value for the unit, a. Since we are just modeling a sin
z= b+ wi xi
Non-linear transform σ
activation
(7.1) x, and we’ll replace the sum with the convenient dot product:
i
z
i
activation for thenodeisin fact thefinal output of thenetwork, which we
todefined
nt to express this weighted sum using vector notation; recall
∑ it’smore
Weighted sum
a vector is, at heart, just a list or array of [Link]
Thus
call y. So the value y is
convenient expressas:z=
thisw·x+ b
weighted sum using vector
ms of a weight vector w, a scalar bias b, and an input vector • Instead of just using z, we'll apply a nonlinear activation
Weightsvector w1 w2 w3 from b
biasalgebra
linear that a vector
function f: is, at heart, just a list or array
of
sum with the convenient dot product:
As defined in Eq. 7.2, zis just a realyvalued
= a =number.
f (z)
Input layer x1 we’
x2 ll talk
x3 about
+1 z in terms of a weight vector w, a scalar bias b, and
z= w·x+ b (7.2) Finally, instead of using z, a linear function of x, as the output, neur
x, and we’apply
ll replace the sum
a non-linear with
function the
f to z. convenient dot
We will refer to theproduct:
output of this fun
s just a real valued number. We’ll discuss three popular non-linear functions f () below (the sigmo
activation
sing z, a linear function of x, as the output, neural units 57
the activation
and the rectified
BITS Pilani, Deemed to be University under value
Section
BITS for the
linear unit, but
ReLU)
3 ofPilani
Pilani, UGC Act, a. Since
1956
Campus we are just modeling
it’s pedagogically a single
convenient to su BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

on f to z. We will refer to the output of this function as

the unit, a. Since we are just modeling unit,sigmoid
a singleFunctionsthe
activationfunction
sigmoid for thenodeisin
since wefact
saw zin
it = Chapter
thefinal w·x+ bthenetwork, which we’ll g
output of5:
Non-Linear Activation
in fact thefinal output of thenetwork, which we’ ll generally
Final function the unit is computing
call y. So the value y is defined as: 7
defined as:
As defined in Eq. 7.2, z is just a real valued number. 1
y = a = f (z) y = s (z) =
a = f (z)
We’ve already seen the sigmoid for logisticFinally, instead of Eq.
regression:Substituting using 7.2z,intoa linear
Eq. 7.3function
gives 1+use−the
of x, as the
z output of aoutp
neu
apply
ular non-linear functions f () below (the sigmoid, the tanh, a non-linear
We’ll
function
discuss three
f to
popular
z. Wefunctions
will refer to the(the
output oft
ReLU) but it’s pedagogically convenient to start with the Thesigmoid (shown in non-linear
Fig. 7.1) has anumber f () below sigmoid,
of advantages;
1modeling ait ma
we saw it in Chapter 5: activation the activation and thevalue for
rectified the
linear
into the range [0,1], which
unit, a.
y = sis(w·x+
ReLU) but Since
it’s
useful in
we
b)squashing
= are
pedagogically just
convenient to start
outliers toward 0 orw
Sigmoid
activation
sigmoid for thenodeisin
sigmoid function fact
sinceas
we thefinal
saw it in output
Chapter 5: 1+
of exp(− (w·x+ which
thenetwork, b))
y = s (z) =
1
(7.3) differentiable, which we saw in Section ?? will be handy for learnin
1+ e− z call y. So the [Link] y is defined
7.2 shows a final as:schematic of a basic neural unit. In th
in Fig. 7.1) has a number of advantages; it maps the output 1
ch is useful in squashing outliers toward 0 or 1. And it’s takes 3 input values x1, x2, and y = xs3(z)
, and
= computes−z
a weighted sum
we saw in Section ?? will be handy for learning. y =w3a, = 1+
f (z) e
value by aweight (w1, w2, and respectively), adds them to ab
passesThe thesigmoid (shown
resulting sum in through
Fig. 7.1) has anumber function
a sigmoid of advantages; it mapsinthe
to result a
59 into the range [0,1], which is useful in squashing outliers toward 0 or 1. A
We’ ll discuss three popular non-linear functions f () below (the sig
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 y y to get an intuition. Let’s
valueby aweight (w1, w2, and w3, Let’s walk through
respectively), addsthem antoexample just
abias term b, and then
aa unita sigmoid
with thefunction
following weight a
vector and
a bias:0
Final unit again passes the resulting sum through to result
An in a number
example between
σσ and 1. σ
w = σ[0.2, 0.3, 0.9]
zz y b =zz 0.5
y
Output value ∑
∑a • Suppose aa unit has: ∑∑ input vector:
What would this unit do with the following
σw σ
• w = [0.2,0.3,0.9]
Non-linear activation function w w11 w
w22 w33 bb • b = 0.5 ww11 xw w=22 [0.5,w330.6,bb
w 0.1]
z
z
xx11 xx22 xx33 • What happens with input x: xx11be: xx22 xx33
+1
+1 ∑ +1
+1
Weighted sum ∑ • xThe= resulting
[0.5,0.6,0.1] output y would
Figure
Figur e7.27.2 AA neural
neural unit,
unit, taking
taking 33inputs
inputsxx11,, xx22,, and
and xx33 (and
(and aabias
biasbb that
that weweFigure
represent
Figur e7.2
represent 7.2as asAA
aa neural
neural unit,
unit,1taking
taking 33inputs
inputsxx11,, xx22,, and
and1 xx33 (and
(and aabias
biasbb
inputWeights
bias w1some =wconvenient
yweight forwan
sconvenient
weight
weight for
for an
an input clamped
clamped at at +1)w1 and
+1) andwproducingw3 an
producing
2 b output
an output y. y. We
We include
include weight
some 2 (w·x+
for an b)b =clamped
3 input
input clamped at
at +1) = producing
+1) and
− (w·x+ b)
and producing an
an output
output y. y. We
− (.5⇤.2+ .6⇤.3+ .1⇤.9+ .5)
=
We incl
inc
intermediate
intermediatevariables: the
theoutput
outputx1of
of the
thexsummation, of xthe x2 a. x3
1+ e 1+ e
variables:
Input layer summation,
2 x3 z, z, and
and+1the
the output
output of the
1
intermediate
sigmoid,
intermediate
sigmoid, a. variables:
In
In
variables: the
+1theoutput
output of of the
the summation,
summation, z, z, and
and the
the output
output
this
thiscase
casethe
theoutput
output of
of the
theunit
unit yy is
is the
the same
same asas a,a, but
but inin deeper networks we’ll
deeper networks we’
this
this
In reserve
case
casethe
llpractice,
reserve theyythe
output
to
output of
of the
to sigmoid unit
theis notybythat
unit is
is the
the same
commonly same asas
useda,
a, butas in
but in deeper
an deeper netwo
netwo
activation fu
mean
meanthethefinal
final output
output of
of the
theentire
entirenetwork,
network, leaving Figure
leaving aa as 7.2
asthe A neural
theactivation unit,
of taking
of an 3 inputs
an individual
mean x ,
final
finalx , and x (and a bias we represent as a
that is very similar but almost always better is the tanh functionoo
the node. output of the entire network, leaving a as the activation
activationtanh individual
mean the 1
node.2 3
output of the entire network, leaving a as the activation
weight for an input clamped at +1) and producing an output y. We include some convenient
61
intermediate variables: the outputtanh issummation,
of the a variant z,ofand
BITS Pilani, Deemed to be University under
thethe
Section
sigmoid
output ofthat
3 ofPilani
UGC Act,
ranges from
the sigmoid,
1956
a. In -1 to +1: BITS Pilani, Deemed to be University under Section 3 ofPilani
UGC Act, 1956

Let’s Let’s Let’s Let’s

BITS Pilani, Campus BITS Pilani, Campus

Let’s walk through an example just to get

walk through an example just to get an
thisan the output Let’s
intuition.
caseintuition. y is theLet’s
suppose
suppose
of the unit we
we have
walk
have
same walka through
through
as a, but in deeperan example
annetworks
example just to
justreserve
we’ll get
to get an intuition.
an
y to intuition. Let’s
z
e bias:e−
−bias: z
unit with
unit with the
thefollowing
following weight
weight vector
vector and mean the final output of the entireunit
and bias:
bias: unit withleaving
with
network, thefollowing
the following weightofvector
weight
a as the activation vector and
and
an individual node.
An example An example y =
ez + e− z
w == [0.2,
w [0.2,0.3,
0.3, 0.9]walk through an example just to get an intuition. Let’s suppose
Let’s
0.9] w == we
w [0.2,
[0.2, 0.3,
0.3,
have 0.9]
a 0.9]
0.5 unit with the following weightThe
bb == 0.5
simplest
vector and bias:activation function,bband perhaps
== 0.5
0.5 the most common
ReLU tified linear unit, also called the ReLU, shown in Fig. 7.3b. It’s
• Suppose a unit has:
What would
would thisunit
unit do
do with
with the
thefollowing
following input
input vector:
vector: • Suppose
w = xwould
when
What a[0.2,
isunit0.3,0.9]
has:
positive, anddo
thisunit
unit 0with
do otherwise:
with thefollowing
following input
input vector:
vector:
What this What would this the
•w = [0.2,0.3,0.9] • bw == [0.2,0.3,0.9]
0.5
•b = 0.5 xx == [0.5,
[0.5,0.6,
0.6,0.1]
0.1] • b = 0.5 y== [0.5,
xx = max(x,0)
[0.5, 0.6,0.1]
0.6, 0.1]
What would this unit do with the following input vector:
• What happens with the following input x? • What happens with input x:
The resulting output y would be: Theresulting
resulting output
output y would be:
• The resulting output y would be:
x = [0.5,0.6,0.1] • Thexx == [0.5, 0.6, 0.1] y would be:
[0.5,0.6,0.1]

y = s (w·x+ b) = 11 = The 11
resulting output y=would 11
y
be: = s =(w·x+
.70 b) = 11 = 11 =
y = s (w·x+ b) = 1+ e−−(w·x+ b) = 1+ e−−(.5
(w·x+b) (.5⇤⇤.2+.6
.2+ ⇤
.6⇤ .3+.1
.3+ ⇤
.1⇤ .5) = 1+ ey
.9+.5)
.9+ −
=
− 0.87 s =(w·x+
0.87 .70 b) = 1+ e−−(w·x+ (w·x+b)b) = 1+ e−−(.5(.5⇤⇤.2+.6
.2+ ⇤
.6⇤ .3+.1
.3+ ⇤
.1⇤ .5) =
.9+.5)
.9+
1+ e 1+ e 1+ e 1+ e 1+ e
1 1 1
Inpractice,
practice, the
thesigmoid
sigmoid is isnot
not commonly
commonly used y = sas
used (w·x+ b) =
an activation
activation [Link]=practice,
practice,
A1+ function the sigmoid is = commonly
not = used
.70 as an activation f
In as an e− (w·x+In
1+ function. b) A e− (.5the
function ⇤.2+sigmoid
.6⇤.3+ .1⇤.9+is.5)not 1+ e− 0.87 used
commonly as an activation fu
anh
anh
that is very similar but almost always better is the
that is very similar but almost always betterInispractice, tanh
the tanh function
function shown
tanh
shown
tanh
that
thatin Fig.
is
in is
Fig. very
7.3a;
very
7.3a;similar
similar but
but almost always
almostfunction.
always A better
better is the tanh function
is the tanh function
tanh is
isaavariant
variant of
of the
thesigmoid
sigmoid that
that ranges
rangesfrom
from -1 -1 to the
to +1:
+1: sigmoid is not commonly
tanh is used
isaavariant
variant ofas an activation
of the
thesigmoid
sigmoid that function
that ranges
rangesfrom
from -1 -1 to
to +1:
+1:
tanh tanhBITS Pilani, Deemed very tanh
thatto beisUniversity under
BITSsimilar
Section 3 ofPilani
Pilani, UGC Act, but
1956
Campus almost always better is the tanh function shown in Fig. 7.3a;
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
h an example just
Let’s
Let’s walkto get
walk an intuition.
through
through Let’sjust
an example
an example suppose
just getwe
to get
to have a Let’s
an intuition.
an intuition. Let’s suppose
suppose we
we have
have a y=
weight
g weightvector
vector
unit with
and
and
with the
bias:
bias:
thefollowing
following weight
weight vector
vector and
and bias: 1 1 1 ez + e− z
unit An example ybias:
= s (w·x+ b) = = Non-Linear = Functions = .70
1+ e− (w·x+ b) 1+ e− (.5⇤.2+ .6Activation
⇤.3+ .1⇤.9+ .5) 1+ e− 0.87 besides
ww == [0.2,
These activation 0.3,
functions
[0.2, 0.9]
0.3,have0.9]different properties that make them useful for The simplest activation function, and perhaps the most commo
sigmoid
ww == [0.2,
[0.2,
For In 0.3,
0.3, 0.9]
0.9]
practice, the sigmoid
func- is not commonly used as an activation function. A function
different language applications or network architectures.
ion has thebnice b ==properties
0.5
0.5 of being smoothly bbdifferentiable
example, thetanh
ReLU tified linear unit, also called the ReLU, shown in Fig. 7.3b. It
=tanh0.5that and is very
mapping similar but
outlier almost always better is the tanh function shown in Fig. 7.3a;
= 0.5 when z is from
positive,
values toward the mean. The rectifier function, on the other hand tanh has is anice
variantproperties
of the sigmoid that ranges -1 to and
+1: 0 otherwise: Most Common:
do • Suppose
ohatwith
result
with thefrom
the following
What would a
it unit
being
following
would thishas:input
very close
input
thisunit vector:
unit doto linear.
do with
with the In the sigmoid
thefollowing or
following input tanh functions,
input vector:
vector: very
What
high •valueswof=z result in values of y that are satur ated, i.e., extremely close to 1, ez − e− z
[0.2,0.3,0.9] y= z −z y = max(z,0)
(7.5)
and have derivatives
xx == very
[0.5,
[0.5, close
0.6,
0.6, to
0.1]0. Zero derivatives cause problems for learning, e + e
• b = 0.5 x == [0.5,
[0.5,0.6,
0.6,0.1] 0.1]
because as we’ ll see in Section 7.4, we’ ll trainxnetworks by propagating an error
signal• backwards,
What happens with input
multiplying x: (partial derivatives) from each layer of the function, and perhaps the most commonly used, is the rec-
gradients The simplest activation
would
would be:
be:
The resulting output y would be: U totified also called the ReL U, shown in Fig. 7.3b. It’s just the same as x
• The
network; x resulting
gradients that are
= [0.5,0.6,0.1]output
almosty0 would
cause be:
the error ReL
signal linearand
get smaller unit,
smaller
until it is too small to be used for training, a problem called the when x is positive,
vanishing gr adientand 0 otherwise:
11 1 11 11 1
1=ReLU 11
[Link]fiers
= ==
s (w·x+ don’ tb)
have= this problem, since
==
= the derivative of = .70
.70 for high = = .70
.70
−−(w·x+ y z=b)iss1 (w·x+
(w·x+ b)
e−−(.5
(.5⇤=.2+
⇤ ⇤.3+
.2+close
.6⇤.3+−.1
.1 ⇤
⇤.9+
(w·x+ .5) = 1+
.5)
b) 1+ e−−−(.5⇤
0.87
0.87.2+.6 ⇤
.6⇤ .3+.1.1⇤⇤ .5) = 1+ e−− 0.87
.9+.5) y ==max(x,
0.87 0) (7.6)
ee
values of b) 1+ ethan
1+
rather very 1+
.6
1+ e e−
to 0. .9+b)
(w·x+ 1+
1+ e ee
− (.5 ⇤ .2+ .3+ .9+ 1+ e ReLU
disisnotIn
not practice, the
commonly the sigmoid
used as an
an isisnot
not commonly
commonly
activation [Link] as
AAas an activation
functionactivation function.
function. A tanh
A function
function Rectified Linear Unit
Incommonly
practice, used as
sigmoid activation function.
used function
an
XOR
almostthat
anh
almost
anh problem
that isvery
always
always
is very similar
better
better isthe
is
similar buttanh
the
but almost
tanh
almost always
function
function
always betterin
shown
shown
better isFig.
in
is the tanh
7.3a;
[Link]
the function shown
7.3a;function shown in
in Fig.
Fig. 7.3a;
7.3a;
sigmoid
sigmoid tanh
that
that
tanh isranges
is aa variant
ranges
variant of the
from
from
of the sigmoid
-1 sigmoid
-1 to
to that ranges
+1: that
+1: rangesfrom
from -1-1 to
to +1:
+1: 66
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Early in the history of eneural −networks it was realized thatezzthe − − z of neural net-
power
z − e− z
z z e−
y = e− e y = e −from e zcombining (7.5)
works, as with the
y =
real
z
neurons
The− z
that
XOR inspired
problem them,
y comes
= e z + e− z (7.5) (7.5)
these
(7.5) Perceptrons
units into larger [Link] ++ ee− z ez + e− z
One The
of thesimplest
most cleveractivation
demonstrations of the need
function, andfor multi-layer
perhaps the networks
most was
commonly used, is the rec-
nhefunction,
proofThe and
by simplest
Minsky perhaps the
andactivation
Papert most
(1969) commonly
function,
that and
aMinsky
single used,
perhaps
neural is
unit the
the rec-
most
cannot commonly
compute used, is the rec-
function,
tified and perhaps
linear the
unit,of also most
called commonly
the ReLU, used,
shown is the rec-
in Fig. 7.3b. It’s just the same as x
eLU
called the
tified ReL U, shown in Fig. 7.3b. It’sand Papert (1969)
just the same as x
x areIt’s just the same as x
some very simple functions
linear shown
unit, alsoits input. Consider
called7.3b. the
the ReLU, task of computing elementary
LU
alled functions
ogical
the
when ReLU,
x is
of positive,
two inputs,
in
and
like
Fig.
0
AND,otherwise:
OR,
It’sXOR.
and
justshown As
in Fig.
thea reminder,
same 7.3b.
ashere
d 0 otherwise:
when x• is Can neural units
positive, andcompute
0 otherwise:simple functions of input? • A very simple neural unit
0heotherwise:
truth tables for those functions:
y = max(x, 0) (7.6) •(7.6)
Binary output (0 or 1)
y = max(x,
AND 0) y = max(x,0)
y = max(x, 0) OR XOR
(7.6) •(7.6)
No non-linear activation function
x1 x2 y x1 x2 y x1 x2 y
0 0 0 0 0 0 0 0 0
0 1 0 0 1 1 0 1 1
1 0 0 1 0 1 1 0 1
1 1 1 1 1 1 1 1 0

This example was first shown for the per ceptr on, which is a very simple neural
unit that has a binary output and does not have a non-linear activation function. The

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Easy
Earlyto
in build AND
the history or OR
of Early
neural thewith
innetworks itperceptrons
history was
of neural
realized
networks
that theitpower
was realized
of neural Not
that
net- possible
the power tonet-
of neural capture XOR with perceptrons
works, as with the realworks,
neurons
as with
that inspired
the real them,
neurons comes
that inspired
from combining
them, comes
thesefrom combining these
units into larger networks.
units into larger networks.
One of the most cleverOnedemonstrations
of the most clever
of the need
demonstrations
for multi-layer
of the
networks
need forwas
multi-layer networks was
the proof by Minsky and the proof
Papertby(1969)
Minsky thatand
a single
Papertneural
(1969)unit
thatcannot
a single
compute
neural unit cannot compute
some very simple functions
some ofvery
itssimple
input. Consider
functionstheof its
task input.
of computing
Consider elementary
the task of computing• Try for yourself!
elementary
logical functions of twological
inputs,
functions
like AND, of two
OR,inputs,
and [Link] AND,
As a reminder,
OR, and here
[Link] a reminder, here are
the truth tables for those
thefunctions:
truth tables for those functions:

AND OR AND XOR OR XOR

x1 x2 y x1 x2x1y x2 y x1 x2x1y x2 y x1 x2 y
0 0 0 0 0 0 00 0 0 0 0 00 0 0 0 0
0 1 0 0 1 0 11 0 0 1 0 11 1 0 1 1
1 0 0 1 0 1 10 0 1 0 1 10 1 1 0 1
AND 1 1 1 1 1 1 1OR
1 1 1 1 1 01 1 1 1 0

er ceptr on This example was firstThis

per ceptr on shownexample
for thewas first shown
per ceptr on, which
for the
is aper
very
ceptr
simple
on, which
neural is a very simple neural
unit that has a binary output
unit that
andhas
does
a binary
not have
output
a non-linear
and doesactivation
not have afunction.
non-linear
The
activation function. The
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Why? Perceptrons are linear classifiers Decision boundaries

• Perceptron equation given x1 and x2, is the equation x2 x2 x2

of a line
• w1x1 + w2x2 + b = 0 1 1 1

• (in standard linear format: x2 = (−w1/w2)x1 + (−b/w2) ) ?

0 0 0
• This line acts as a decision boundary 0 1
x1
0 1
x1
0 1
x1

• 0 if input is on one side of the line

a) x1 AND x2 b) x1 OR x2 c) x1 XOR x2
• 1 if on the other side of the line

XOR is not a linearly separable function!

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
s realized that the power of neural net- Early in the history of neural networks it was realized that the power of neural net-
Solution to the XOR problem Solution to the XOR problem
d them, comes from combining these works, as with the real neurons that inspired them, comes from combining these
units into larger networks.
the need for multi-layer networks was One of the most clever demonstrations of the need for multi-layer networks was
a single neural unit cannot compute the proof by Minsky and Papert (1969) that a single neural unit cannot compute
sider the task of computing elementary some very simple functions of its input. Consider the task of computing elementary
• XOR can't be calculated by a single perceptron • XOR can't be calculated by a single perceptron
OR, and XOR. As a reminder, here are logical functions of two inputs, like AND, OR, and XOR. As a reminder, here are
• XOR can be calculated the
by atruth
layered
tablesnetwork
for thoseoffunctions:
units. • XOR can be calculated by a layered network of units.
ReLU y1 y1
XOR AND OR XOR
y x1 x2 y 1 -2 0 x1 x2 y x1 x2 y x1 x2 y 1 -2 0
0 0 0 0 ReLU h1 h2 +1 0 0 0 0 0 0 0 0 0 h1 h2 +1
1 0 1 1 0 1 0 0 1 1 0 1 1
1 1 0 1 1 1 1 1 0 -1 1 0 0 1 0 1 1 0 1 1 1 1 1 0 -1
1 1 1 0 x1 x2 1 1 1 1 1 1 1 1 0 x1 x2
+1 +1
ceptr on, which is a very simple neural
per ceptr on This example was first shown for the per ceptr on, which is a very simple neural
e a non-linear activation function. The unit that has a binary output and does not have a non-linear activation function. The

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

The hidden representation h Feedforward Neural Networks

y1
1 -2 0
h1 h2 +1
1 1 1 1 0 -1
x1 x2 +1
x2 h2 • Can also be called multi-layer perceptrons (or MLPs) for
1 1
historical reasons

0 x1 0
h1
0 1 0 1 2

a) The original x space b) The new (linearly separable) h space

(With learning: hidden layers will learn to form useful representations)

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
Binary Logistic Regression as a 1-layer Network
Fully connected single layer network
(we don't count the input layer in counting layers!)
y y
Output layer σ Output layer
1
s s n
s
(σ node) (y is a scalar) (softmax nodes) y is a vector
w w w b (scalar)
W b
(vector) 1 n W is a b is a vector
matri
Input layer x x + Input layerx x xn +1
vector x 1 n 1 scalars 1

77 78
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Two-Layer Network with scalar output

softmax: a generalization of sigmoid
Output layer y is a scalar
• For a vector z of dimensionality k, the softmax is:
(σ node) U
hidden units
(σ node) Could be ReLU
Or tanh

• Example: W b
Input layer
x xn +1
(vector)
1

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Two-Layer Network with scalar output Two-Layer Network with scalar output

Output layer y is a scalar Output layer y is a scalar

(σ node) U (σ node) U
hidden units j hidden units
Wji
(σ node) (σ node) Could be ReLU
Or tanh

W b vector W b
Input layer Input layer
x i xn +1 x xn +1
(vector) (vector)
1 1

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Two-Layer Network with softmax output Multi-layer Notation

Output layer sigmoid or softmax

(σ node) U y is a vector W[2 b[2]

]
hidden units j ReLU
(σ node) Could be ReLU
Or tanh

W b W[1 b[1]
]
Input layer
x xn +1
(vector) x i xn +1
1
1

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Multi Layer Notation Replacing the bias unit
y
a
σ
z
∑
• Let's switch to a notation without the bias unit
w1 w2 w3 b
x1 x2 x3 +1 • Just a notational change
1. Add a dummy node a0=1 to each layer
2. Its weight w0 will be the bias
3. So input layer a[0]0=1,
– And a[1]0=1 , a[2]0=1,…

85
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Replacing the bias unit Replacing the bias unit

• Instead of: We'll do this: Instead of:

We'll do this:
x= x1, x2, …, xn0 x= x0, x1, x2, …, xn0 y1 y2 … yn
2
y1 y2 … yn
2

U U

h1 h2 h3 … hn
1
h1 h2 h3 … hn
1

W b W

x1 x2 … xn
0 +1 x0=1 x1 x2 … xn
0

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
Use cases for feedforward networks Classification: Sentiment Analysis

• Let's consider 2 (simplified) sample tasks:

• We could do exactly what we did with logistic
1. Text classification regression
2. Language modeling • Input layer are binary features as before
• Output layer is 0 or 1 σ
•State of the art systems use more powerful neural U
architectures, but simple models are useful to consider!
W
x1 x
n

89
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Sentiment Features Feedforward nets for simple classification

σ
σ U
2-layer
Logistic
Regression W feedforward
network
W
x1 x
n
f1 f2 fn x1 x
n
f1 f2 fn
• Just adding a hidden layer to logistic regression
92

• allows the network to use non-linear interactions between

features
• which may (or may not) improve performance.

91 92
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Even better: representation learning Neural Net Classification with embeddings as input
features!

σ
• The real power of deep learning comes from U
the ability to learn features from the data
• Instead of using hand-built human- W
engineered features for classification
• Use learned representations like x1 x
n
embeddings! e1 e2 en

93 94
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Issue: texts come in different sizes Reminder: Multiclass Outputs

• This assumes a fixed size length (3)!

• What if you have more than two output classes?
• Kind of unrealistic.
– Add more output units (one for each class)
• Some simple solutions (more sophisticated solutions later) – And use a “softmax layer”
1. Make the input the length of the longest review
• If shorter then pad with zero embeddings
• Truncate if you get longer reviews at test time U
2. Create a single "sentence embedding" (the same
dimensionality as a word) to represent all the words
• Take the mean of all the word embeddings W
• Take the element-wise max of all the word embeddings
x xn
• For each dimension, pick the max value from all words 1

95 96
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Neural Language Models (LMs) Simple feedforward Neural Language Models

• Language Modeling: Calculating the probability of the next • Task: predict next word wt
word in a sequence given some history. • given prior words wt-1, wt-2, wt-3, …
• We've seen N-gram based LMs • Problem: Now we’re dealing with sequences of arbitrary
• But neural network LMs far outperform n-gram language length.
models • Solution: Sliding windows (of fixed length)
•State-of-the-art neural LMs are based on more powerful
neural network technology like Transformers
•But simple feedforward LMs can do almost as well!

97 98
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Neural Language Model Intuition: training a 2-layer Network

U Backward pass

Forward pass
W

Training instance x1 xn

99 100
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Intuition: Training a 2-layer network Reminder: Loss Function for binary logistic
regression

• A measure for how far off the current answer is to the

right answer
• Cross entropy loss for logistic regression:

101 102
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Reminder: gradient descent for weight updates Where did that derivative come from?

ISTI C REGRESSI ON • Using the chain rule! f (x) = u(v(x))

• Intuition (see the text for details)
y Derivative of the weighted sum
a Derivative of the Activation
σ Derivative of the Loss
d z
wt+ 1 = wt − h L( f (x;w), y) (5.14) ∑
dw
w1 w2 w3 b
s extend the intuition from a function of one scalar variable w to many
x 1 x2 x3 +1

ecause we don’t just want to move left or right, we want to know where
mensional space (of the N parameters that make up q) we should move. 104
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Summary
How can I find that gradient for
every weight in the network?

• These derivatives on the prior slide only give the updates for • For training, we need the derivative of the loss with respect
one weight layer: the last one! to weights in early layers of the network
• What about deeper networks? • But loss is computed only at the very end of the network!
• Lots of layers, different activation functions? • Solution: backward differentiation
• Solution in the next lecture: • Given a computation graph and the derivatives of all the
• Even more use of the chain rule!! functions in it we can automatically compute the derivative
of the loss with respect to these early weights.
• Computation graphs and backward differentiation!

105 106
BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

• Neural Probabilistic Language Models

Probabilistic Language Models

• Probability of a sequence of words: P(W )  P(w1, w2 ,..., wt 1 , wT )

• Conditional probability of an upcoming word: P( wT w1 , w2 ,..., wt 1 )

• Chain rule of probability: P( w1 , w2 ,..., wt 1 , wT )  P(w1 ) P( w2 | w1 ) P( w3 | w1 , w2 )...P( wT | w1 , w2 ,.., wT )

T
• (n-1)th order Markov assumptionP(w1 , w2 ,..., wt 1 , wT )   P(wt | w1 , w2 ,..., wt 1 )
t 1

• Each p(wi |wi−4 , w i−3 , w i−2 , w i−1 ) may not have enough statistics to estimate
• we back off to p(wi |wi−3 , w i−2 , w i−1 ), p(wi |w i−2 , w i−1 ), etc., all the way to p(wi)
107

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
• Neural Language Models (LMs) • Simple feedforward Neural Language Models

• Language Modeling: Calculating the probability of the next • Task: predict next word wt
word in a sequence given some history. given prior words wt-1, wt-2, wt-3, …
• We've seen N-gram based LMs • Output : probability distribution over possible next words.
• But neural network LMs far outperform n-gram language
models
• State-of-the-art neural LMs are based on more powerful
neural network technology like Transformers
• Problem: Now we’re dealing with sequences of arbitrary
• But simple feedforward LMs can do almost as well! length.
10 11
9 • Solution: Sliding windows (of fixed length) 0

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

• Word Embeddings

•• Why Neural
Neural LMsmodels
language work better thanwords
represent N-gram LMsprior context by
in this
• one-hot vector representation , e.g., dog = (0,0,0,0,1,0,0,0,0,....), cat = (0,0,0,0,0,0,0,1,0,....) their embeddings, rather than just by their word identity as used
• Represent each of the N previous words as a one-hot vector of length |V| one-hot vector , i.e., with one in n-gram language models.
dimension for each word in the vocabulary •• embeddings
Using embeddings allows neural language models to generalize
• word “toothpaste”, supposing it is V5, i.e., index 5 in the vocabulary, x5 = 1, and xi = 0 i  5,
better to unseen data.
Training data:
• We've seen: I have to make sure that the cat gets fed.
Embedding matrix One-hot vector

• Never seen: dog gets fed

Test data:
• I forgot to make sure that the dog gets ___
• N-gram LM can't predict "fed"!
• Neural LM can use similarity of "cat" and "dog" embeddings to
generalize and predict “fed” after dog

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
• Neural Language Model • Training the neural language model

Equations:

• Two ways:
• Freeze the embedding layer E with initial word2vec values.
– Freezing means we use word2vec or some other pretraining algorithm to
compute the initial embedding matrix E, and then hold it constant while we
only Modify W, U, and b, i.e., we don’t update E during language model
training
concatenate 3 embeddings for the 3 context words
to produce the embedding layer e
• Learn the embeddings simultaneously with training the network.
11
– Useful when the task the network is designed for (like sentiment
3 classification, translation, or parsing) places strong constraints on what makes
a good representation for words.

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

• N-gram vs NLM

Large Language Models

• Compared to n-gram models, NLM can

– handle much longer histories
– generalize better over contexts of similar words,
– Are more accurate at word-prediction.
• NLM are
– much more complex,
– slower
– need more energy to train,
– less interpretable than n-gram models, so for many (especially smaller) tasks
an n-gram language model is still the right tool.

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Large Language Models

Large Language Models

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Large Language Models Benefits of using Large

Language Models

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 LLM Use Cases

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Prompts and Completions

Prompting and Prompt Engineering

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
In context learning and In context learning and
zero shot inference Few shot inference

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

DL Model Types Generative AI

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Generative AI Applications Generative AI- AI Assistants

Notification Assistant Contextual Assistant

FAQ Assistant

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Generative AI- AI Assistants

Generative AI- PaintBox

Personalized Assistant Autonomous Organization ofAssistants

• Assistant knows you much more in • Group of AI assistants that know every
customer personally
detail
• Eventually run large parts of company
• Quickly checks a few final things operations—from lead generation over
before giving you a quote tailored to marketing, sales, HR, or finance
your actual situation.

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 Generative AI- CodeAid Real-life challenges in NLP tasks

• Deep learning methods are data-hungry

• >50K data items needed for training
• The distributions of the source and target data must
be the same
• Labeled data in the target domain may be limited
• This problem is typically addressed with transfer
learning

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus

Applications of Transfer Learning Fine Tuning

• Image Classification • Fine-tuning takes it a step further by allowing

• Names Entity Recognition the pre-trained layers to be updated.
• Sentiment Analysis • Beneficial when the new dataset is large
• Cross Lingual Learning enough and similar to the original dataset on
• Gaming which the pre-trained model was trained

• Image Recognition
• Speech Recognition

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus BITS Pilani, Deemed to be University under Section
BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus
 References

CH-7 - Speech and Language Processing by Daniel Jurafsky

[Link]

BITS Pilani, Deemed to be University under Section

BITS 3 ofPilani
Pilani, UGC Act, 1956
Campus