Neural Networks

(1) The use of modern neural nets is often called deep learning, because modern networks are
often deep (have many layers).

(2) Neural unit is taking a weighted sum of its inputs, with one additional term in the sum called
a bias term.

(3) Three popular non-linear functions f below (the sigmoid, the tanh, and the rectified linear
unit or ReLU)

(4) The sigmoid has a number of advantages; it maps the output into the range (0; 1), which is
useful in squashing outliers toward 0 or 1.

(5) A function that is very similar but almost always better is the tanh function; tanh is a variant
of the sigmoid that ranges from -1 to +1:

(6) The simplest activation function, and perhaps the most commonly used, is the rectified
linear unit, also called the ReLU. It’s just the same as z ReLU when z is positive, and 0 otherwise:

(7) These activation functions have different properties that make them useful for different
language applications or network architectures. For example, the tanh function has the nice
properties of being smoothly differentiable and mapping outlier values toward the mean. The
rectifier function, on the other hand, has nice properties that result from it being very close to
linear. In the sigmoid or tanh functions, very high values of z result in values of y that are
saturated, i.e., extremely close to 1, and have saturated derivatives very close to 0.

(8) Zero derivatives cause problems for learning, because as, we’ll train networks by propagating
an error signal back-wards, multiplying gradients (partial derivatives) from each layer of the
network; gradients that are almost 0 cause the error signal to get smaller and smaller until it is
too small to be used for training, a problem called the vanishing gradient problem.

(9) How to compute XOR using two layers of ReLU-based units. The input being processed by
two layers of neural units. The middle layer (called h) has two units, and the output layer (called
y) has one unit. A set of weights and biases are shown that allows the network to correctly
compute the XOR function.

(10) A feedforward network is a multilayer feedforward network in which the units are
connected with no cycles; the outputs from units in each layer are passed to units in the next
higher layer, and no outputs are passed back to lower layers.

(11) The core of the neural network is the hidden layer h formed of hidden units hi, each of
which is a neural unit, taking a weighted sum of its inputs and then applying a non-linearity.

Page 1 of 10
(12) In the standard architecture, each layer is fully-connected, meaning that each unit in each
layer takes as input the outputs fully-connected from all the units in the previous layer, and
there is a link between every pair of units from two adjacent layers. Thus, each hidden unit
sums over all the input units.

(13) The advantage of using a single matrix W for the weights of the entire layer is that now
the hidden layer computation for a feedforward network can be done very efficiently with
simple matrix operations. In fact, the computation only has three steps: multiplying the weight
matrix by the input vector x, adding the bias vector b, and applying the activation function g.

(14) The algorithm for computing the forward step in an n-layer feedforward network, given
the input vector a [0] is thus simply:

(15) The need for non-linear activation functions: One of the reasons we use non-linear
activation functions for each layer in a neural network is that if we did not, the resulting
network is exactly equivalent to a single-layer network. Imagine the first two layers of such a
network of purely linear layers:

This generalizes to any number of layers. So, without non-linear activation functions, a
multilayer network is just a notational variant of a single layer network with a different set of
weights, and we lose all the representational power of multilayer networks.

(16) Replacing the bias unit:

Page 2 of 10
(17) Training Neural nets

First, we’ll need a loss function that models the distance between the system output and the
gold output, and it’s common to use the loss function used for logistic regression, the cross-
entropy loss. Second, to find the parameters that minimize this loss function, we’ll use the
gradient descent optimization algorithm. Third, gradient descent requires knowing the gradient
of the loss function, the vector that contains the partial derivative of the loss function with
respect to each of the parameters.

(18)

(19) The cross-entropy loss is simply the negative log of the output probability corresponding to
the correct class, and we therefore also call this the negative log likelihood loss:

Page 3 of 10
(20) How do we compute the gradient of this loss function? Computing the gradient requires
the partial derivative of the loss function with respect to each parameter. For a network with
one weight layer and sigmoid output (which is what logistic regression is), we could simply use
the derivative of the loss that we used for logistic regression in Equation.

(21) A computation graph is a representation of the process of computing a mathematical

expression, in which the computation is broken down into separate operations, each of which is
modeled as a node in a graph.

Consider computing the function L(a; b; c) = c(a + 2b).

d=2*b
e=a+d
L=c*e
The simplest use of computation graphs is to compute the value of the function with some
given inputs. In the forward pass of a computation graph, we apply each operation left to right,
passing the outputs of each computation as the input to the next node.

Page 4 of 10
(22) computing the derivative of the loss function L with respect to z. By the chain rule:

(23) Various forms of regularization are used to prevent overfitting. One of the most important
is dropout: randomly dropping some units and their connections from the network during
training. At each iteration of training, we repeatedly choose a probability p and for each unit we
replace its output with zero with probability p.

(24) Tuning of hyperparameters is also important. The parameters of a neural network are the
weights W and biases b; those are learned by gradient descent. The hyperparameters are things
that are chosen by the algorithm designer; optimal values are tuned on a devset rather than by
gradient descent learning on the training set. Hyperparameters include the learning rate h , the
mini-batch size, the model architecture (the number of layers, the number of hidden nodes per
layer, the choice of activation functions), how to regularize, and so on.

(25) Neural language models have many advantages over the n-gram language models.
Compared to n-gram models, neural language models can handle much longer histories, can
generalize better over contexts of similar words, and are more accurate at word-prediction. On
the other hand, neural net language models are much more complex, are slower and need
more energy to train, and are less interpretable than n-gram models, so for some smaller tasks
an n-gram language model is still the right tool.

Page 5 of 10
(26) Forward inference or decoding is the task, given an input, of running a forward pass on the
network to produce a probability distribution over possible outputs, in this case next words.

(27) A one-hot vector is a vector that has one element equal to 1—in the dimension
corresponding to that word’s index in the vocabulary— while all the other elements are set to
zero.

(28) Forward inference in the neural language model

(29) The equations for a neural language model with a window size of 3, given one-hot input
vectors for each input context word, are:

Note that we formed the embedding layer e by concatenating the 3 embeddings for the three
context vectors; we’ll often use semicolons to mean concatenation of vectors.

(30) In self-training for language modeling, we take a corpus of text as training material and at
each time step t ask the model to predict the next word. At first it will do poorly at this task, but
since in each case we know the correct answer, we can easily train it to be better at predicting
the correct next word. We call such a model self-supervised because we don’t have to add any
special gold labels to the data; the natural sequence of words is its own supervision!

(31) Freezing means we use word2vec or some other pretraining algorithm to compute the
initial embedding matrix E, and then hold it constant while we only modify W, U, and b, i.e., we
don’t update E during language model training.

Page 6 of 10
 Logistic Regression
(1) An algorithm that is admirably suited for discovering the link between features or clues
and some particular outcome: logistic regression. Logistic regression is one of the most
important analytic tools in the social and natural sciences. In natural language
processing, logistic regression is the base-line supervised machine learning algorithm for
classification, and also has a very close relationship with neural networks.
(2) Generative and Discriminative Classifiers: The most important difference be-tween
naive Bayes and logistic regression is that logistic regression is a discriminative classifier
while naive Bayes is a generative classifier. These are two very different frameworks for
how to build a machine learning model. A generative model would have the goal of
understanding what dogs look like and what cats look like. Given a test image, the
system then asks whether it’s the cat model or the dog model that better fits the image,
and chooses that as its label. A discriminative model, by contrast, is only trying to learn
to distinguish the classes. If that one feature neatly separates the classes, the model is
satisfied.
(3) A generative model like naive Bayes makes use of this likelihood term, which generative
model expresses how to generate the features of a document if we knew it was of class
c.
(4) By contrast a discriminative model in this text categorization scenario attempts to
directly compute P(c|d).
(5) A machine learning system for classification then has four components: 1. A feature
representation of the input. For each input observation x (i), this will be a vector of
features [x1, x2, …,xn]. 2. A classification function that computes y ^, the estimated class,
via p(y|x). 3. An objective function that we want to optimize for learning, usually
involving minimizing a loss function corresponding to error on training examples. 4. An
algorithm for optimizing the objective function. We introduce the stochastic gradient
descent algorithm.
(6) How do we make a decision about which class to apply to a test instance x? For a given
x, we say yes if the probability P(y = 1|x) is more than .5, and no otherwise. We call .5
the decision boundary:

(7) Recent NLP systems avoid hand-designed features and instead focus on representation
learning: ways to learn features automatically in an unsupervised way from the input.
(8) Scaling input features: When different input features have extremely different ranges of
values, it’s common to rescale them so they have comparable ranges. We standardize
input values by centering them to result in a zero mean and a standard standardize
Page 7 of 10
 deviation of one (this transformation is sometimes called the z-score). That is, if i is the
mean of the values of feature xi across the m observations in the input dataset, and i is
the standard deviation of the values of features xi across the input dataset, we can
replace each feature xi by a new feature x’i computed as follows:

(9) Logistic regression has a number of advantages over naive Bayes. Naive Bayes has
overly strong conditional independence assumptions. Consider two features which are
strongly correlated; in fact, imagine that we just add the same feature f1 twice. Naive
Bayes will treat both copies of f1 as if they were separate, multiplying them both in,
overestimating the evidence. By contrast, logistic regression is much more robust to
correlated features; if two features f1 and f2 are perfectly correlated, regression will
simply assign part of the weight to w1 and part to w2. Thus when there are many
correlated features, logistic regression will assign a more accurate probability than naive
Bayes. So logistic regression generally works better on larger documents or datasets and
is a common default.
(10) Despite the less accurate probabilities, naive Bayes still often makes the correct
classification decision. Furthermore, naive Bayes can work extremely well (some-times
even better than logistic regression) on very small datasets or short documents.
Furthermore, naive Bayes is easy to implement and very fast to train. So it’s still a
reasonable approach to use in some situations.
(11) The multinomial logistic classifier uses a generalization of the sigmoid, called the
softmax function, to compute p(yk = 1|x). The softmax function takes a vector z = [z 1, z2,
…, zK] of K arbitrary values and maps them to a probability distribution, with each value
in the range [0,1], and all the values summing to 1. Like the sigmoid, it is an exponential
function. For a vector z of dimensionality K, the softmax is defined as:

(12) Fig. 5.3 shows the difference between binary and multinomial logistic
regression by illustrating the weight vector versus weight matrix in the computation of
the output class probabilities.
(13) How are the parameters of the model, the weights w and bias b, learned?
Logistic regression is an instance of supervised classification in which we know the
correct label y (either 0 or 1) for each observation x. What the system produces via Eq.

Page 8 of 10
 5.5 is ˆ y, the system’s estimate of the true y. We want to learn parameters (meaning w
and b) that make ˆ y for each training observation as close as possible to the true y. This
requires two components. The first is a metric for how close the current label (ˆ y) is to
the true gold label y. The distance between the system output and the gold output, and
we call this distance the loss function or the cost function. The second thing we need is
an optimization algorithm for iteratively updating the weights so as to minimize this loss
function. The standard algorithm for this is gradient descent.
(14) This is called conditional maximum likelihood estimation: we choose the
parameters w, b that maximize the log probability of the true y labels in the training data
given the observations x. The resulting loss function is the negative log likelihood loss,
generally called the cross-entropy loss.
(15) How shall we find the minimum of this (or any) loss function? Gradient descent
is a method that finds a minimum of a function by figuring out in which direction the
function’s slope is rising the most steeply, and moving in the opposite direction.
(16) A convex function has at most one minimum; there are no local minima to get
stuck in, so gradient descent starting from any point is guaranteed to find the minimum.
(17) Stochastic gradient descent is an online algorithm that minimizes the loss
function by computing its gradient after each training example, and nudging q in the
right direction. Stochastic gradient descent is called stochastic because it chooses a
single random example at a time.
(18) The learning rate  is a hyperparameter that must be adjusted. If it’s too high,
hyperparameter the learner will take steps that are too large, overshooting the
minimum of the loss function. If it’s too low, the learner will take steps that are too
small, and take too long to get to the minimum. It is common to start with a higher
learning rate and then slowly decrease it, so that it is a function of the iteration k of
training; the notation k can be used to mean the value of the learning rate at iteration
k.
(19) For example in batch training we compute the gradient over the entire dataset.
(20) A compromise is mini-batch training: we train on a group of m examples that is
less than the whole dataset.
(21) Mini-batch training also has the advantage of computational efficiency. The
mini-batches can easily be vectorized, choosing the size of the mini-batch based on the
computational resources.
(22) There is a problem with learning weights that make the model perfectly match
the training data. If a feature is perfectly predictive of the outcome because it happens
to only occur in one class, it will be assigned a very high weight. The weights for features
will attempt to perfectly fit details of the training set, in fact too perfectly, modeling
noisy factors that just accidentally correlate with the class. This problem is called

Page 9 of 10
 overfitting. A good model should be able to generalize well from the training overfitting
generalize data to the unseen test set, but a model that overfits will have poor
generalization.
(23) L2 regularization is a quadratic function of the weight values, named because it
uses the (square of the) L2 norm of the weight values. The L2 norm, ||||2, is the same
as the Euclidean distance of the vector  from the origin. If  consists of n weights, then:

(24) L1 regularization is a linear function of the weight values, named after the L1
norm ||W||1, the sum of the absolute values of the weights, or Manhattan distance:

Page 10 of 10