Natural Language Processing

06. Neural Networks

Juan José Alegrı́a

April 22, 2025

1
Table of contents

1. Linear Models Recap

2. Neural Networks

3. Neural Network Training

4. The Computation Graph Abstraction

5. More on neural networks

2
Linear Models Recap
Linear models

• A linear model is defined by:

f (⃗ x · W +⃗
x) = ⃗ b
x ∈ Rdin
⃗ W ∈ Rdin ×dout ⃗
b ∈ Rdout

• For binary classification, we can use:

• ŷ = sign(f (⃗
x )), if we are only interested on the predicted class, or
1
• the sigmoid function, ŷ = σ(f (⃗
x )) = , if we are interested on
1+e−⃗x·⃗
w −b

the probability or confidence on the decision

• For multi-class problems, we can use the softmax function, which transforms the
x·W +⃗
(⃗ b) [i]
output vector into a probability distribution: ⃗
ŷ[i] = e
P (⃗x·W +⃗b)[j] , and then predict
j e
using the argmax function.

3
Linear models

• Linear models cannot represent some functions, such as XOR

• Nevertheless, we can map the values using a non-linear function, and then we
can solve the problem

4
Linear models

• In some cases, we can find the mapping function relatively easily. For instance,
for the XOR problem, we can use ϕ(x1 , x2 ) = [x1 × x2 , x1 + x2 ] and then train a
classifier to solve ŷ = f (⃗
x ) = ϕ(⃗ ⃗ +b
x) · w
• But we would like to find the ϕ function during training, i.e., to be a trainable
function.
• We would like something like

ŷ = f (⃗
x ) = ϕ(⃗ ⃗ +b
x) · w
xW ′ + ⃗
x ) = g(⃗
ϕ(⃗ b ′)

• This is indeed a neural network architecture

5
Neural Networks
Introduction to Neural Networks

• Very popular machine learning models formed by units called neurons.

• A neuron is a computational unit that has scalar inputs and outputs.
• Each input has an associated weight w.
• The neuron multiplies each input by its weight, and then sums them (other
functions such as max are also possible).
• It applies an activation function g (usually non-linear) to the result, and passes it
to its output.
• The nonlinear activation function g has a crucial role in the network’s ability to
represent complex functions.
• Without the nonlinearity in g, the neural network can only represent linear
transformations of the input.

6
Neuron

0
Source:[Goldberg, 2017]

7
Neural network

• We can connect many neurons, creating a network (hence the name neural
networks)
• Neurons are arranged in layers, reflecting the flow of information
• The bottom layer has no incoming arrows, as is the input to the network
• The top-most layer has no outgoing arrows, as is the output of the network
• The other layers are called hidden layers.
• When each neuron in one layer is connected to all neurons in the next layer, we
are talking about a fully connected layer. A network comprised of such layers is
called a feedforward neural network (also called multilayer perception (MLP))

8
Feedforward Network with two Layers

0
Source:[Goldberg, 2017]
9
Feedforward Network Neural Networks

• The feedforward network from the picture is a stack of linear models separated
by nonlinear functions.
• The values of each row of neurons in the network can be thought of as a vector.
• The input layer is a 4-dimensional vector (⃗
x ), and the layer above it is a
6-dimensional vector (⃗ h1 ).
• The fully connected layer can be thought of as a linear transformation from 4
dimensions to 6 dimensions.
• A fully connected layer implements a vector-matrix multiplication, ⃗
h =⃗xW + ⃗ b.
• The weight of the connection from the i-th neuron in the input row to the j-th
neuron in the output row is W[i,j] .
• The values of ⃗
h are transformed by a nonlinear function g that is applied to each
value before being passed on as input to the next layer.

0
Vectors are assumed to be row vectors and superscript indices correspond to
network layers.

10
Fully connected layers as vector-matrix multiplications

 
�h1 �h2
W1,1 W1,2
�x = [�x1 , �x2 , �x3 ] W = W2,1 W2,2 
�h = �xW W3,1 W3,2

W32
�xW = [�x1 ∗ W11 + �x2 ∗ W21 + x3 ∗ W31 , �x1 ∗ W12 + �x2 ∗ W22 + x3 ∗ W32 ]
W11 W12
W22
�h = [�h1 , �h2 ]
W21
W31

�x1 �x2 �x3

11
Neural Networks as Mathematical Functions

• The Multilayer Perceptron (MLP) from the figure is called MLP2 because it has
two hidden layers.
• A simpler model would be MLP1, a multilayer perceptron of one hidden layer:

⃗
ŷ = NNMLP1 (⃗ xW 1 + ⃗
x ) = g(⃗ b1 )W 2 + ⃗
b2
x ∈ Rdin , W 1 ∈ Rdin ×d1 , ⃗
⃗ b2 ∈ Rdout , ⃗
b1 ∈ Rd1 , W 2 ∈ Rd1 ×dout , ⃗ ŷ ∈ Rdout

• Here W 1 and ⃗
b1 are a matrix and a bias term for the first linear transformation of
the input.
• The function g is a nonlinear function that is applied element-wise (also called a
nonlinearity or an activation function ).
• W 2 and ⃗
b2 are the matrix and bias term for a second linear transform.
• When describing a neural network, one should specify the dimensions of the
layers (d1 ), the input (din ), and the output (dout ).

12
Neural Networks as Mathematical Functions

• MLP2 can be written as the following mathematical function:

x) = ⃗
NNMLP2 (⃗ ŷ
⃗ xW 1 + ⃗
h1 = ⃗ b1
⃗
h2 = g 1 (⃗
h1 )W 2 + ⃗
b2
y = g 2 (⃗
⃗ h2 )W 3 + b3
xW 1 + ⃗
y = (g 2 (g 1 (⃗
⃗ b1 )W 2 + ⃗
b2 ))W 3 + b3 .

• The matrices and the bias terms that define the linear transformations are the
parameters of the network.
• Like in linear models, it is common to refer to the collection of all parameters as
Θ.

13
Representation Power

• [Hornik et al., 1989] and [Cybenko, 1989] showed that a multilayer perceptron of
one hidden later (MLP1) is a universal approximator.
• MLP1 can approximate all continuous functions on a closed and bounded subset
of Rn .
• This may suggest there is no reason to go beyond MLP1 to more complex
architectures.
• The result does not say how easy or hard it is to set the parameters based on
training data and a specific learning algorithm.
• It also does not guarantee that a training algorithm will find the correct function
generating our training data.
• Finally, it does not state how large the hidden layer should be.

0
For intuition about this theorem, see [Nielsen, 2015], Chapter 4.

14
Representation Power

• In practice, we train neural networks using local search methods.

• We also use hidden layers of relatively modest sizes (up to several thousands).
• The universal approximation theorem does not give any guarantees under these
conditions.
• However, there is definitely benefit in trying out more complex architectures than
MLP1.
• In many cases, however, MLP1 does indeed provide strong results.

15
Activation Functions

• The nonlinearity g can take many forms.

• There is currently no good theory as to which nonlinearity to apply in which
conditions.
• Choosing the correct nonlinearity for a given task is for the most part an
empirical question.

16
Sigmoid

• The sigmoid activation function σ(x) = 1+e1−x is an S-shaped function,

transforming each value x into the range [0, 1].
• The sigmoid was the canonical nonlinearity for neural networks since their
inception.
• It is currently considered to be deprecated for use in internal layers of neural
networks, as the choices listed next prove to work much better empirically.

17
Hyperbolic tangent (tanh)

2x
• The hyperbolic tangent tanh(x) = ee2x +1
−1
activation function is an S-shaped
function, transforming the values x into the range[−1, 1].

18
Hard tanh

• The hard-tanh activation function is an approximation of the tanh function which

is faster to compute and to find derivatives thereof:

 

 −1 x < −1 

hardtanh(x) = 1 x >1

 x 

otherwise.

19
ReLU

• The rectifier activation function [Glorot et al., 2011], also known as the rectified
linear unit is a very simple activation function.
• It is easy to work with and was shown many times to produce excellent results.
• The ReLU unit clips each value x < 0 at 0.

ReLU(x) = max(0, x)

• It performs well for many tasks, especially when combined with the dropout
regularization technique (to be explained later).

20
Activation Functions

• As a rule of thumb, both ReLU and tanh units work well, and significantly
outperform the sigmoid.
• You may want to experiment with both tanh and ReLU activations, as each one
may perform better in different settings.
• The figure from below shows the shapes of the different activations functions,
together with the shapes of their derivatives.

0
Source:[Goldberg, 2017] 21
Embedding Layers

• In NLP the input to the neural network contains symbolic categorical features
(e.g., words from a closed vocabulary, character n-grams, POS tags).
• In linear models we usually represent the input with sparse vectors e.g., as the
sum, average, or the concatenation of one-hot encoded vectors (the sum or the
average can produce bag-of words representation).
• In neural networks, it is common to associate each possible feature value (i.e.,
each word in the vocabulary, each POS tag category) with a d−dimensional
dense vector for some d.
• These vectors are then considered parameters of the model, and are trained
jointly with the other parameters.
• The mapping from a symbolic feature values such as “word number 1249” to
d−dimensional vectors is performed by an embedding layer (also called a
lookup layer ).

22
Embedding Layers

• The parameters in a word embedding layer are simply a matrix E ∈ R|vocab|×d

where each row corresponds to a different word in the vocabulary.
• The lookup operation is then simply indexing: v1249 = E[1249,:] .
• If the symbolic feature is encoded as a one-hot vector ⃗
x , the lookup operation
can be implemented as a vector-matrix multiplication ⃗x E.
• The embedding vectors are combined before being passed on to the next layer.
• Common combination operations are: concatenation, summation, average.
• A word embeddings matrix E can be initialized with pre-trained word vectors
trained from unlabeled documents using specific methods based on the
distributional hypothesis such as the ones implemented in Word2Vec (to be
discussed later in the course).

23
The Embedding Matrix

One-hot-encoded word vector Embedding Matrix

1 × |V |
|V | × d
 
�x = [0, 0, . . . , 1, · · · 0] −1.8 2.3 ... 3.1 abduct
 .. .. .. .. 
 . . . . 
 
E=
 3.3 −2.1 ··· 4.6  dog
 . .. .. .. 
abduct dog zumba
 .. . . . 
4.2 1.9 ··· −3.3 zumba

�xE = [3.3, −2.1, . . . , 4.6]

24
Neural Network Training
Neural Network Training

• Suppose that we have defined our architecture. For instance, a feedforward

network with two hidden layers. This architecture is parameterized by the set of
weights w and biases b of each layer. Let’s denote the collection of parameters
as Θ, and ⃗
ŷ = NN(⃗ x ; Θ) the output of the neural network.
• We also have a training set: a collection of input values ⃗ x1:N = ⃗
x1 , . . . ,⃗
xN and
the corresponding desired outputs ⃗ y1:N = ⃗y1 , . . . ,⃗
yN .
• Let us define a loss function L(⃗
x1:N ,⃗
y1:N ; θ) that measures the error of our
model on the training set.
• The training goal is to find θ that minimize the loss:
Θ̂ = argminΘ L(⃗ y1:N ; Θ). Note that ⃗
x1:N ,⃗ x and ⃗
y are fixed, they are not
parameters of the model. The only values we can control are the weights and
biases (i.e., Θ).
• How can we find the optimal values for Θ?

25
Neural Network Training

• We know that we can use gradient descent to iteratively update the value of each
∂L
parameter: Θi ← Θi − η ∂Θ (ŷ, y) (for all parameters Θi and with a learning
i
parameter η).

∂L
• But how can we compute ∂Θi ?

26
Neural Network Training

• Maybe we can compute the loss directly for each parameter Θi , either weight or
∂L L(Θ1 ,...,Θi +ϵ,...,ΘM )−L(Θ1 ,...,Θi ,...,ΘM )
bias, with the formula ∂Θ ≈ ϵ .
i
• But then, for each Θi we would need to compute L(Θ1 , . . . , Θi + ϵ, . . . , ΘM ). If
we have a million parameters, we would need to do a million computations.
• We also need to compare with L(Θ1 , . . . , ΘM ), so that’s one million and one
computations. All of that just to perform one update to the parameters of the
network, with one training example.
• We would like to compute all the partial derivatives in a faster way.

27
Backpropagation

• Algorithm to compute the gradient of L with respect to all the parameters in a

clever way, using the derivative chain rule.
• Plan of attack: do a forward pass with a training example, storing the values at
each hidden layer. Then, compute the derivatives iteratively in reverse. Start by
calculating the derivatives of the parameters in the last layer, pass the results to
the previous layer, then the one before that, and continue until reaching the first
layer.
∂L ∂L
• Goal: compute and for each layer l.
∂wijl ∂bjl

28
Derivative Chain Rule Recap

• Simple chain rule: let z = f (y), y = g(x),

∂z ∂z ∂y
= ×
∂x ∂y ∂x
• Example: z = ey , y = 2x

∂z ∂z ∂y
= × = ey × 2 = 2e2x
∂x ∂y ∂x

0
Figure taken from:
[Link]
29
Derivative Chain Rule Recap

• Multiple path chain rule: let z = f (y1 , y2 ), y1 = g1 (x), y2 = g2 (x)

∂z ∂z ∂y1 ∂z ∂y2
= × + ×
∂x ∂y1 ∂x ∂y2 ∂x
• Example: z = ey1 ×y2 , y1 = 2x, y2 = x 2

∂z 3
= (ey1 ×y2 × y2 ) × 2 + (ey1 ×y2 × y1 ) × 2x = e2x × 6x 2
∂x

0
Figure taken from:
[Link]
30
Derivative Chain Rule Recap

The general version of the multiple path chain rule would be:

∂z X ∂z
n
∂y
= × i
∂x ∂yi ∂x
i=1

31
0
Figure taken from:
Backpropagation: notation

• M = number of layers in the � �

network L �ŷ, �y

• dl = dimension of layer l
• h⃗l = z l−1
⃗ W l + b⃗l
�ŷ1 �ŷ2 �ŷ3 �y13 �y23 �y33

• Equivalently,
Pdl−1 l l−1 Softmax Gold output
hjl = i=1 wij zi + bjl

• z⃗l = g(h⃗l ). Equivalently, �h3

1
�h3
2
�h3
3

zjl = g(hjl ). 3
W11 3
W12
3
W23
3
W13 3
W22

• The first layer is just the training �z12

3
W21
�z22
sample: z⃗0 = ⃗ x. g 2
g 2

• The final layer of the network is just �h2

1
�h2
2

z M = [z1M , . . . , zdM ]. 2 2 2
W22
M W11 W12
2
W21
• Just for now, assume that zjM �z11 �z21
depends only on hjM . For instance, g1 g1
g on the final layer could be a �h1
1
�h1
2

sigmoid, but not softmax (just for 1

W11 1
W12 1
1
W22 W32
now). 1
W21 1
W31
�x1 �x2 �x3
∂L
• Finally, a useful value: δlj = 32
∂hjL
Backpropagation: equations

We have four equations that describe the backpropagation algorithm:

∂L ′ M
δM
j = g (hj ) (BP1)
∂zjM

dl+1
X
l+1 ′ l
δlj = δl+1
k wjk g (hj ) (BP2)
k=1

∂L
= δlj (BP3)
∂bjl

∂L
= δlj zil−1 (BP4)
∂wijl

0
All the following derivation of backpropagation is adapted from
[Nielsen, 2015], Chapter 2.

33
Backpropagation: equation 1

Goal: find an expression for δM

j (i.e., in the last layer).

∂L
δM
j =
∂hjM

XdM
∂L ∂zkM
=
k =1
∂zkM ∂hjM
M
∂L ∂zj ∂zkM
= (since zkM only depends on hkM , then = 0 for k ̸= j)
∂zj ∂hjM
M ∂hjM
∂L ′ M
= g (hj ) (since zjM = g(hjM ))
∂zjM

Then:
∂L ′ M
δM
j = g (hj ) (BP1)
∂zjM

34
Backpropagation: equation 2

Goal: find an expression for δlj (i.e., for a hidden layer)

∂L
δlj =
∂hjl
dl+1
X ∂L ∂hkl+1
= (since the output of the layer l can affect all the outputs in layer (l + 1)
k=1
∂hkl+1 ∂hjl
dl+1
X ∂hkl+1
= δl+1
k (by definition of δl+1
k )
k=1
∂hjl

35
Backpropagation: equation 2
∂hkl+1
Let’s focus on .
∂hjl
We know that:

X
dl
hkl+1 = wikl+1 zil + bkl
i=1

X
dl
= wikl+1 g(hil ) + bkl (by definition of zil )
i=1

∂hkl+1
We note that hjl only appears multiplied by wjk . Then, when we compute all the
∂hjl
terms with i ̸= j will be 0.

∂hkl+1
= wjkl+1 g ′ (hjl )
∂hjl

36
Backpropagation: equation 2

Putting everything together:

dl+1
X ∂hkl+1
δlj = δl+1
k
k=1
∂hjl
(BP2)
dl+1
X
l+1 ′ l
= δl+1
k wjk g (hj )
k=1

⃗l = δl+1
Alternatively, it can be written in vector form as δ ⃗ (W l+1 )T ⊙ g ′ (h⃗l ), where ⊙

is the Hadamard or element-wise product.

37
Backpropagation: equation 3
∂L
Goal: find an expression for (i.e., the derivative of L with respect to the bias of
∂bjl
neuron j in layer l)

∂L Xdl
∂L ∂hkl
=
∂bjl k=1
∂hkl ∂bjl
l
∂L ∂hj
= (since bjl only affects hjl )
∂hjl ∂bjl
∂hjl
= δlj (by definition of δlj )
∂bjl
X
= δlj × 1 (since hjl = wi j l zil−1 + bjl )

Then:

∂L
= δlj (BP3)
∂bjl

38
Backpropagation: equation 4
∂L
Goal: find an expression for (i.e., the derivative of L with respect to the weight that
∂wijl
connects input i-th to neuron j-th in layer l)

∂L X dl
∂L ∂hkl
=
∂wijl k =1
∂hkl ∂wijl
l
∂L ∂hj
= (since wijl only affects hjl )
∂hjl ∂wijl
∂hjl
= δlj (by definition of δlj )
∂wijl
X
= δlj zil−1 (since hjl = wkjl zkl−1 + bjl )

Then:

∂L
= δlj zil−1 (BP4)
∂wijl

39
Backpropagation: algorithm

Let ⃗
x be a training input example, and ⃗
y its corresponding value.

1. Input: Set z⃗0 = ⃗

x
2. Feedforward: For each l = 1, 2, . . . , M compute h⃗l = z l−1 W l + b⃗l and
z⃗l = g(h⃗l ), and store those values.
3. Output error: Compute the vector ∇δM , where each component is given by
δM ∂L ′ M
j = M g (hj ).
∂zj

4. Backpropagate the error: for each layer l = M − 1, M − 2, . . . , 1, compute

⃗l = (δl+1
δ ⃗ (W l+1 )T ) ⊙ g ′ (h⃗l )
∂L
5. Update parameters: update the weights with the rule wijl ← wijl − ∂wij and the
∂L
biases with bjl ← bjl − ∂bj . From equations (BP3) and (BP4), we know that
∂L
∂wij = δlj zjl−1 and ∂L
∂bj = δlj

Now we can update the weights with just one forward and one backward pass :)

40
Backpropagation: softmax?

• When deriving the previous equations, me assume that zjM = g(hjM ); i.e., the
activation for the final layer depends only on the value for that neuron.
• That assumption doesn’t hold for the softmax function, often used in multi-class
classification problems
zM
e j
• softmax(zjM ) = P zM .
ke k
• It’s easy to derive the equation BP1 in that case, without our previous
assumption (see problem Backpropagation with softmax and the log-likelihood
cost, in [Nielsen, 2015], Chapter 3).
• But it’s not that easy to extend this demonstration to other architectures (residual
connections, shared weights, etc).
• We need a better abstraction.

41
The Computation Graph
Abstraction
The Computation Graph Abstraction

• One can compute the gradients of the various parameters of a network by hand
and implement them in code.
• This procedure is cumbersome and error prone.
• For most purposes, it is preferable to use automatic tools for gradient
computation [Bengio, 2012].
• A computation graph is a representation of an arbitrary mathematical
computation (e.g., a neural network) as a graph.
• This abstraction will allow us computing the gradients from any kind of neural
network architecture using the backpropagation algorithm.
• Previous formulation was restricted to feedforward networks.

42
The Computation Graph Abstraction

• A computation graph is a directed acyclic graph (DAG).

• Nodes correspond to mathematical operations or (bound) variables.
• Edges correspond to the flow of intermediary values between the nodes.
• The graph structure defines the order of the computation in terms of the
dependencies between the different components.
• The graph is a DAG and not a tree, as the result of one operation can be the
input of several continuations.

43
The Computation Graph Abstraction

• Consider for example a graph for the computation of (a ∗ b + 1) ∗ (a ∗ b + 2):

• The computation of a ∗ b is shared.

• Since a neural network is essentially a mathematical expression, it can be
represented as a computation graph.

44
The Computation Graph Abstraction

0
Figure taken from: [Goldberg, 2017] 45
The Computation Graph Abstraction

• The figure above shows the computation graph for an MLP with one hidden-layer
and a softmax output transformation.
• Oval nodes represent mathematical operations or functions, and shaded
rectangle nodes represent parameters (bound variables).
• Network inputs are treated as constants, and drawn without a surrounding node.
• Input and parameter nodes have no incoming arcs, and output nodes have no
outgoing arcs.
• The output of each node is a matrix, the dimensionality of which is indicated
above the node.

46
The Computation Graph Abstraction

• This graph is incomplete: without specifying the inputs, we cannot compute an

output.
• Figure 5.1b shows a complete graph for an MLP that takes three words as
inputs, and predicts the distribution over part-of-speech tags for the third word.
• This graph can be used for prediction, but not for training, as the output is a
vector (not a scalar) and the graph does not take into account the correct answer
or the loss term.
• Finally, the graph in Figure 5.1c shows the computation graph for a specific
training example, in which the inputs are the (embeddings of ) the words “the,”
“black,” “dog,” and the expected output is “NOUN” (whose index is 5).
• The pick node implements an indexing operation, receiving a vector and an index
(in this case, 5) and returning the corresponding entry in the vector.

47
Forward Computation

• The forward pass computes the outputs of the nodes in the graph.
• Since each node’s output depends only on itself and on its incoming edges, it is
trivial to compute the outputs of all nodes.
• This is done by traversing the nodes in a topological order and computing the
output of each node given the already computed outputs of its predecessors.
• More formally, in a graph of N nodes, we associate each node with an index i
according to their topological ordering.
• Let fi be the function computed by node i (e.g., multiplication, addition , etc.).

48
Forward Computation

• Let π(i) be the parent nodes of node i , and π−1 (i) = {j|i ∈ π(j)} the children
nodes of node i (these are the arguments of fi ).
• Denote by v (i) the output of node i , that is, the application of fi to the output
values of its arguments π−1 (i).
• For variable and input nodes, fi is a constant function and π−1 (i) is empty.
• The computation-graph forward pass computes the values v (i) for all i ∈ [1, N].

Algorithm 1: Computation graph forward pass

for i = 1 to N do
Let a1 , . . . , am = π−1 (i)
v (i) ← fi (v (a1 ), . . . , v (am ))
end

49
Backward Computation (Backprop)

• The backward pass begins by designating a node N with scalar (1 × 1) output

as a loss-node, and running forward computation up to that node.
• The backward computation computes the gradients of the parameters with
respect to that node’s value.
• Denote by d(i) the quantity ∂N∂i .
• The backpropagation algorithm is used to compute the values d(i) for all nodes
i.
• The backward pass fills a table of values d(1), . . . , d(N) as shown in the
following algorithm.

Algorithm 2: Computation graph backward pass (backprop-

agation)
∂N
d(N) ← 1; // ∂N =1
;
for i = N-1 to 1 do
P ∂fj ∂N P ∂N ∂j
d(i) = j∈π(i) d(j) · ∂i ; // ∂i = j∈π(i) ∂j · ∂i
;
end

50
Backward Computation (Backprop)

• The backpropagation algorithm is essentially following the chain-rule of

differentiation.
∂fj
• The quantity ∂i is the partial derivative of fj (π−1 (j)) w.r.t the argument
i ∈ π−1 (j).
• This value depends on the function fj and the values v (a1 ), . . . , v (am ) (where
a1 , . . . , am = π−1 (j)) of its arguments, which were computed in the forward
pass.
• Thus, in order to define a new kind of node, one needs to define two methods:
one for calculating the forward value v (i) based on the node’s inputs, and the
∂fj
another for calculating ∂i for each x ∈ π−1 (i).

51
Summary of the Computation Graph Abstraction

• Notice that the above formulation of backpropagation is equivalent to one given

earlier in the class.
• Te computation graph abstraction allows us to:
1. Easily construct arbitrary networks.
2. Evaluate their predictions for given inputs (forward pass)
3. Compute gradients for their parameters with respect to arbitrary scalar
losses (backward pass or backpropagation).
• A nice property of the computation graph abstraction is that it allows computing
the gradients for arbitrary networks (e.g., networks with skip-connections, shared
weights, special loss functions, etc.)

0
A comprehensive tutorial on the backpropagation algorithm over the
computational graph abstraction can be found here:
[Link]

52
Derivatives of “non-mathematical” functions

∂fj
• Defining ∂i for mathematical functions such is as log or + is straightforward.
• It may be challenging to think about the derivative of operations as as pick(⃗
x , 5)
that selects the fifth element of a vector.
• The answer is to think in terms of the contribution to the computation.
• After picking the i-th element of a vector, only that element participates in the
remainder of the computation.
x , 5) is a vector ⃗
• Thus, the gradient of pick(⃗ v with the dimensionality of ⃗
x where
⃗
v[5] = 1 and ⃗
v[i̸=5] = 0.
• Similarly, for the function max(0, x) the value of the gradient is 1 for x > 0 and 0
otherwise.

53
More on neural networks
Regularization and Dropout

• Multi-layer networks can be large and have many parameters, making them
especially prone to overfitting.
• Model regularization is just as important in deep neural networks as it is in linear
models, and perhaps even more so.
• The regularizers discussed for linear models, namely L2 , L1 , and the elastic-net,
are also relevant for neural networks.
• Another effective technique for preventing neural networks from overfitting the
training data is dropout training [Srivastava et al., 2014].

54
Regularization and Dropout

• The dropout method is designed to prevent the network from learning to rely on
specific weights.
• It works by randomly dropping (setting to 0) half of the neurons in the network (or
in a specific layer) in each training example in the stochastic-gradient training.

0
Figure taken from: [Link]
uploads/[Link]

55
Deep Learning Frameworks

Several software packages implement the computation-graph

model. All these packages support all the essential
components (node types) for defining a wide range of neural
network architectures.
• TensorFlow ([Link] an open source software
library for numerical computation using data-flow graphs originally developed by
the Google Brain Team.
• Keras: High-level neural network API that runs on top of Tensorflow as well as
other backends ([Link]
• PyTorch: open source machine learning library for Python, based on Torch,
developed by Facebook’s artificial-intelligence research group. It supports
dynamic graph construction, a different computation graph is created from
scratch for each training sample. ([Link]

56
Questions?

Thanks for your Attention!

57
References i

Bengio, Y. (2012).
Practical recommendations for gradient-based training
of deep architectures.
In Neural networks: Tricks of the trade, pages 437–478.
Springer.
Cybenko, G. (1989).
Approximation by superpositions of a sigmoidal
function.
Mathematics of control, signals and systems, 2(4):303–314.

58
References ii

Glorot, X., Bordes, A., and Bengio, Y. (2011).

Deep sparse rectifier neural networks.
In Proceedings of the fourteenth international conference
on artificial intelligence and statistics, pages 315–323.
Goldberg, Y. (2017).
Neural network methods for natural language
processing.
Synthesis Lectures on Human Language Technologies,
10(1):1–309.

59
References iii

Hornik, K., Stinchcombe, M., and White, H. (1989).

Multilayer feedforward networks are universal
approximators.
Neural networks, 2(5):359–366.
Nielsen, M. A. (2015).
Neural Networks and Deep Learning.
Determination Press.

60
References iv

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and

Salakhutdinov, R. (2014).
Dropout: a simple way to prevent neural networks from
overfitting.
The journal of machine learning research,
15(1):1929–1958.

61