NEURAL NETWORK
lhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhh
h
Convex Loss Functions
Convex loss functions only have one minimum (global).
Often (but not always!) easy to optimize (no iterative
algorithms required).
Non-Convex Loss Functions
Non-convex loss functions are harder to handle since they
have different local and global minima.
Difficult to optimize (in the majority of the cases, iterative
methods needed).
Pretty common in machine learning, especially in
neuronal computation.
One Subtlety
A non-convex loss function often has a non-closed-form (I cannot compute it directly) solution, i.e.,
iterative methods are needed, there is no analytical solution.
A convex loss function has sometimes a closed-form solution (linear regression), and sometimes not
(logistic regression).
While a loss function itself can be convex with respect to its own parameters (e.g., MSE - Mean
Square Error loss), this can (and typically will) change when applied to a neural network with
respect to its parameters, bacause a neaural network has a lot of parameters.
The coefficients (ai) of the function can directly be calculated from the data set.
For quadratic functions like the MSE in combination with linear regression, the minimum can be
computed analytically, i.e., there are equations we can plug in our coefficients to obtain the
coordinates of the minimum.
Lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
llllllllllllllllllllllll
A simple linear model is not enough, we need some thing else.
Binary Cross Entropy for Logistic Regression (it computes the difference between two
probability distibutions)
vvvvvvvvvvvvvvvvvvvvvvvv
Again, BCE can be regarded as a function that depends on the data set and the model parameters d,
[Link]
llllllllllllllllllllllllllll
Since we have a sigmoid, now we will have a no closed-solutionfor the minimum that we can
compute from the dataset. It is no more possible to compute it directly, it is no more possible to use
the MSE approach.
We must use iterative methods such as gradient descent to solve this problem.
Kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk
Recap: Gradient Descent
Given: a function f(x)
Task: find x that maximizes (or minimizes) f(x)
Idea: start at some value x0, and take a small step ω in the direction in which the function decreases
strongest
cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
Gradient Descent for Logistic Regression
For iterative optimization, we need the gradient of the loss function.
This gradient is another function of d and k which tells us the slope of the loss function at a
particular position.
At each position (d, k), the gradient gives the direction of steepest ascent.
Zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
Gradient Descent exploits exactly this principle: It starts at a random position and repeatedly takes
small steps in the direction opposing the gradient.
C
A Simple Logistic Regression Example
Given binary labeled 1D data ((x1, y1),...(xn, yn)), where x belongs to R1 and y belongs to {0, 1}
(i.e., two classes 0 and 1).
A matching logistic regression model would look like o = g(x; {d, k}) = ε(z) = ε(d(intercept) +
k(slope) · x), where z is the “raw” model output and o the final model output (a probability) for
some given input x and the model parameters d and k.
As preparation for the following steps, let’s substitute the model parameters as follows: d ↑ b, k ↑ w,
so our model now looks like g(x; {b, w}) = ε(b + w · x).
As preparation for the following steps, let’s substitute the model parameters as follows: d ↑ b, k ↑ w,
so our model now looks like g(x; {b, w}) = ε(b + w · x). Also, for brevity, we can optionally write
just g(x), i.e., we do not explicitly include the parameters for a more compact notation.
We will call b the bias and w the weight.
Visualization and Output Calculation
b
The above is a simplified visualization of the data flow through our model: o = ε(z) = ε(b + w · x).
The output o can then be used for two things:
• Calculating the loss during gradient descent to repeatedly update our model parameters (bias
b and weight w) to decrease the loss.
• Determining the class prediction yˆ (recall: we have to apply some decision rule, e.g., via a
threshold).
Multi-Dimensional Example (a lot of features)
Given binary labeled M-dimensional data ((x1, y1),...(xn, yn)), where x ↓ RM and y ↓ {0, 1} (i.e.,
two classes 0 and 1).
A matching logistic regression model would look like o = g(x; {b, w}) = ε(z) = ε(b + w · x), where
z is the “raw” model output and o the final model output (a probability) for some given input x and
the model parameters b and w.
Note that the parameter w is now a vector, and w · x is the scalar/dot product, i.e.,
bb
Visualization and Output Calculation
k
The above is a simplified visualization of the data flow through our model with an M = 4-
dimensional input size. See the next slide for how the final model output is calculated.
Multi-Class Example
Given K-multi-class labeled 1D data ((x1, y1),...(xn, yn)), where x belongs to R1 and y belongs to
{1,...,K}.
Recall that the softmax function is suitable for multi-class classification, so we replace our logistic
function ε(z) with the softmax function ε(z)i, where i means the i-th class and z = (z1,...,zK)→ is
the vector of “raw” model outputs.
Recap: Softmax
Generalization of the sigmoid function.
Suitable for multi-class classification.
For K classes with y belongs to {1,...,K} the probability of x belonging to class i is:
where z = (z1,...,zK) is the vector of “raw” model outputs, i.e., there is an output for each class (see
example later).
While not necessary, as a regular sigmoid function would suffice, this also works for binary
classification, i.e., K = 2.
Multi-Class Example
Given K-multi-class labeled 1D data ((x1, y1),...(xn, yn)), where x belongs to R1 and y belongs to
{1,...,K}.
Recall that the softmax function is suitable for multi-class classification, so we replace our logistic
function ε(z) with the softmax function ε(z)i, where i means the i-th class and z = (z1,...,zK) is the
vector of “raw” model outputs.
A matching logistic regression model would look like o = (o1,...,oK)→ = g(x; {b, w})=(ε(z)1,...,
ε(z)K), where o is the final vector of K class probabilities for some given input x and the model
parameters b and w.
Note that the parameters b and w are now vectors.
Visualization
b
The above is a simplified visualization of the data flow through our model with 1D input size and K
= 3 classes. See the next slide for how the final model output is calculated.
Multi-Dimensional and Multi-Class Example
Given K-multi-class labeled M-dimensional data ((x1, y1),...(xn, yn)), where x belongs to R^M and
y belongs to {1,...,K}.
A matching logistic regression model would look like o = (o1,...,oK) = g(x; {b,W})=(ε(z)1,...,
ε(z)K), where o is the final vector of K class probabilities for some given input x and the model
parameters b and W.
Note that the parameter b is now a vector and the parameter W a matrix.
Visualization
Every input is connected to evry output.
V
The above is a simplified visualization of the data flow through our model with M = 4-dimensional
input size and K = 3 classes. See the next slide for how the final model output is calculated.
V
cccccccccccccccccccccccccccc
cccc
ccccccc
c
c
c
Logistic Regression → Neural Networks
What if we made our model g more complex/powerful?
We could just reuse the idea of our bias-weight calculation by adding an intermediate/hidden layer h
with output h = h(x; {b,W}) = b + W x and final output o = g(h(x)):
Neural Networks
We can add even more such layers, and the number of nodes (a.k.a. units or neurons) can also vary.
E.g., 3 layers with a final output of o = g(h^3(h^2(h^1(x)))) → this is a nested function. It’s not to
the power. It’s just notation for the hidden layer:
v
Stacking Linear Layers
There is a caveat (advice) when naively stacking such linear layers.
There is a problem when we just stack layers of linear transformations on top of each other, because
if we perform multiple linear transformations in a row, they can be collapsed into a single linear
transformation.
This means that simply stacking such layers does not increase the complexity/power of our model
(it is just a single linear transformation again).
Non-linearity
To get rid of the linear transformation problem, we thus add a non-linear function f after each layer.
Before: h(x) = b + W x After: h(x) = f(b + W x)
Looks familiar? Indeed it does, this is the same idea we had when going from linear regression to
logistic regression with f = ε.
The logistic/sigmoid function is actually already such a function that we can use to introduce non-
linearity.
These functions are also called activation functions.
Common Activation Functions
There are various activation functions, each with their own benefits and disadvantages.
Here are some often used activation functions:
vvv
PyTorch
PyTorch is an open source deep learning platform for Python that is very well suited for
experimental approach and prototyping as well as production:
• Easy to build big computational graphs (a representation of how data focus to our network).
• Automatic computation of gradients for learning.
• Smoothly switch between CPU and GPU.
• High execution efficiency, written in C and CUDA (CUDA is the interface that allows us to
communicate and run computation on GPUs.
Computational Graphs
A computational graph is a way to represent a math function in the language of graph theory.
Graph theory: nodes are connected by edges, and everything in the graph is either a node or an
edge.
Nodes are either input values or functions for combining values.
Edges receive their weights as the data flows through the graph.
Consider the relatively simple expression: f(x, y, z)=(x + y) · z
Tensors, Modules
PyTorch has three levels of abstraction:
• Tensors (collection of values) represent tensors of any order (e.g., a scalar is a 0th order
tensor), technically equivalent to numpy arrays, with some additional features like the ability
to put it on the GPU. A Pytorch sensor has also the ability to incorporate information from
computational graphs.
• Computational graphs: A PyTorch tensor represents a node in a computational graph. If x is
a tensor that has [Link]↓grad=True, then [Link] is another tensor holding the gradient of
x with respect to some scalar value.
• Modules: e.g., modules for neural networks (layers); allow for composition (a module can
be composed of other modules).
Autograd (automatic grading computation)
In computational graphs, the provenance of tensors is tracked, i.e., they know how they are
constructed.
The computation of the gradient comes therefore for free: automatic computation of the gradient is
called “autograd”.
No need to compute gradients ourselves!
Logistic Regression in PyTorch
Define the model, e.g, for M = 25-dimensional input and K = 10 classes, and define the loss:m
m
m
Adding Another Layer
Neural network model with one hidden layer and 20 nodes in PyTorch:
g
Adding Non-linearity
Neural network model with one hidden layer and 20 nodes in PyTorch including appropriate non-
linearity: